01:58:29 down below 100m 06:01:19 yeah! :) 12:38:47 morning. 14:02:51 If anyone needs a sign to go /that much harder/ on URLs, this is it: https://usercontent.irccloud-cdn.com/file/uMT1LHgg/image.png 14:03:26 Can we do it? Absolutely! How long will it take us? Probably longer than it should! Will kiska sneak in and queue 1.34B URLs on us? He's already threatened me once! 14:15:06 we've got like 10B urls stashed too, so no worries about hogging all the items for yourself :D 14:15:39 imer: is that a challenge 14:15:42 yes 14:15:48 i could do it you know 14:16:01 i know the spell 14:16:28 I forgot which tracker has the stash... 14:16:43 https://tracker.archiveteam.org/urls-stash-blogger/ is a big one 14:17:07 is the sources project still running 14:17:20 urls-sources i think it is called 14:17:37 There is almost 800M here https://tracker.archiveteam.org/urls-stash-reddit/ 14:22:07 And there is a pretty big stash here https://tracker.archiveteam.org/telegram-groups-temp/ 14:22:28 Maybe I should grafana the stashes too 14:54:41 What does it mean that items are ‘stashed’? 14:55:36 wickerz: it's like a squirrel storing its nuts for the winter 15:04:42 What makes these nuts special? Why not keep them with the other nuts in to:do? 15:04:50 Cause we can't keep up 15:05:06 And need to flush the todo for new items being put in 15:05:14 So stash them til we can process those later 15:05:15 Ahhh 15:05:43 for context, i have five dedicated servers running solely the urls project right now, with a combination of i9-9900k and amd epyc cpu's, and while it's making a large dent in the todo, we've got lots more to chew through :( 15:06:00 Some few billions! :D 15:06:02 I'm phasing out the i9-9900k's in favor of the epyc servers 15:08:16 kiska: not to mention all the mp4's 15:08:17 lol 15:08:32 Yeah I need to find that stash... 15:08:58 arkiver said the other day it was something like only 6m or so 15:09:01 i think 21:26:04 So, because I find this absolutely hilarious as much as I do concerning, I wanted to share with you the reason why my OVH workers in BHS cannot work URLs. To make a long story short - The code uses Quad9 resolvers as they have an "unsecured" option (without any kind of bad site filtering etc.). When you launch the code, it runs a function called CheckIP which communicates with a specific URL (on.quad9.net) to check if the resolvers are working. 21:26:32 For whatever reason, on Quad9's Montreal PoP, the IP 9.9.9.10 CANNOT resolve "on.quad9.net" 21:26:53 (rather, no quad9 IPs can resolve that subdomain from the Montreal PoP) 21:28:09 If you have a VPS/server in OVH BHS, you can test this for yourself by running `dig on.quad9.net A @9.9.9.10` or whatever quad9 IP your heart fancies. The end result should be ";; communications error to 9.9.9.10#53: timed out" 21:39:01 That sounds like a quad9 issue to fix 21:39:07 Yup 21:39:19 Debugging this one was fun. 21:39:27 I'm surprised Quad9 didn't notice this themselves though. 21:39:39 The 403 when you use DoH is a nice touch. 21:43:28 Indeed 21:44:29 In any case, I've looped back with the guy that originally handled my ticket at Quad9, whom confirmed the issue on their end with the Montreal PoP, and they will investigate and get back to me sometime tomorrow 21:44:34 Which I eagerly look forward to lol 21:44:47 Cc arkiver ^ 21:46:52 incidentally 21:47:01 there's no way to bypass the CheckIP function is there? 21:47:12 If we know that the resolvers should be working (which they do)? 21:49:01 There isn't, and there shouldn't be. 21:49:15 The resolver clearly is not working correctly. 21:49:28 Maybe it's just the on.quad9.net domain, maybe not, we don't really know. 22:51:27 fair. 23:46:15 Once this gets fixed, I'll definitely get my money's worth from fourone before the term is up. That server is scheduled for cancellation at the end of the month