-
nyany
down below 100m
-
arkiver
yeah! :)
-
nyany
morning.
-
nyany
If anyone needs a sign to go /that much harder/ on URLs, this is it:
usercontent.irccloud-cdn.com/file/uMT1LHgg/image.png
-
nyany
Can we do it? Absolutely! How long will it take us? Probably longer than it should! Will kiska sneak in and queue 1.34B URLs on us? He's already threatened me once!
-
imer
we've got like 10B urls stashed too, so no worries about hogging all the items for yourself :D
-
nyany
imer: is that a challenge
-
imer
yes
-
nyany
i could do it you know
-
nyany
i know the spell
-
kiska
I forgot which tracker has the stash...
-
imer
-
nyany
is the sources project still running
-
nyany
urls-sources i think it is called
-
kiska
-
kiska
-
kiska
Maybe I should grafana the stashes too
-
wickerz
What does it mean that items are ‘stashed’?
-
nyany
wickerz: it's like a squirrel storing its nuts for the winter
-
wickerz
What makes these nuts special? Why not keep them with the other nuts in to:do?
-
kiska
Cause we can't keep up
-
kiska
And need to flush the todo for new items being put in
-
kiska
So stash them til we can process those later
-
wickerz
Ahhh
-
nyany
for context, i have five dedicated servers running solely the urls project right now, with a combination of i9-9900k and amd epyc cpu's, and while it's making a large dent in the todo, we've got lots more to chew through :(
-
kiska
Some few billions! :D
-
nyany
I'm phasing out the i9-9900k's in favor of the epyc servers
-
nyany
kiska: not to mention all the mp4's
-
nyany
lol
-
kiska
Yeah I need to find that stash...
-
nyany
arkiver said the other day it was something like only 6m or so
-
nyany
i think
-
nyany
So, because I find this absolutely hilarious as much as I do concerning, I wanted to share with you the reason why my OVH workers in BHS cannot work URLs. To make a long story short - The code uses Quad9 resolvers as they have an "unsecured" option (without any kind of bad site filtering etc.). When you launch the code, it runs a function called CheckIP which communicates with a specific URL (on.quad9.net) to check if the resolvers are working.
-
nyany
For whatever reason, on Quad9's Montreal PoP, the IP 9.9.9.10 CANNOT resolve "on.quad9.net"
-
nyany
(rather, no quad9 IPs can resolve that subdomain from the Montreal PoP)
-
nyany
If you have a VPS/server in OVH BHS, you can test this for yourself by running `dig on.quad9.net A @9.9.9.10` or whatever quad9 IP your heart fancies. The end result should be ";; communications error to 9.9.9.10#53: timed out"
-
kiska
That sounds like a quad9 issue to fix
-
JAA
Yup
-
JAA
Debugging this one was fun.
-
JAA
I'm surprised Quad9 didn't notice this themselves though.
-
JAA
The 403 when you use DoH is a nice touch.
-
nyany
Indeed
-
nyany
In any case, I've looped back with the guy that originally handled my ticket at Quad9, whom confirmed the issue on their end with the Montreal PoP, and they will investigate and get back to me sometime tomorrow
-
nyany
Which I eagerly look forward to lol
-
JAA
Cc arkiver ^
-
nyany
incidentally
-
nyany
there's no way to bypass the CheckIP function is there?
-
nyany
If we know that the resolvers should be working (which they do)?
-
JAA
There isn't, and there shouldn't be.
-
JAA
The resolver clearly is not working correctly.
-
JAA
Maybe it's just the on.quad9.net domain, maybe not, we don't really know.
-
nyany
fair.
-
nyany
Once this gets fixed, I'll definitely get my money's worth from fourone before the term is up. That server is scheduled for cancellation at the end of the month