-
AKHmm why is our completed items so low
-
imerJAA/arkiver: ebedzin.pl might need nuking, very long urls, server is rejecting most requests transfer.archivete.am/14vAIk/2023-10-30_16-22-17.txt
-
imerAK: also seeing a bunch of timeouts for domains on my end
-
AKAhh that might be it, I really need to setup better logging+aggregation of logs for my containers
-
imer9=0 unshieldedcables...................….......5.3.2higher-frequencyoutages......... yup.
-
AKimer: are you using anything to collect logs from the containers? A long time ago I had logstash and elasticsearch but it didn't end well haha
-
imernope, just docker logs -f and looking at stuff scrolling by
-
imerI have this atrocity: `((for i in {1..40}; do sudo docker logs -n0 -f at-urls$i & done); wait ) | rg '[0-9]+=[0-9]+ https?://([^/]+)/' -a -or '$1' | logtop`
-
imer(opens a live log for all containers and greps out the domain, then feeds into logtop so you see stats per domain)
-
AKMight try setting up opensearch again, I had stuff like this which could show me all the percentages by domain so I was able to tell if we were being banned or not
-
AK
-
arkiverchecking
-
arkiveroof
-
imerthat doesn't sound good
-
arkiverit's being filtered
-
imerthat does sound good
-
imer:)
-
arkiver:P
-
kiskaimer: nah that sounds horrible :D
-
imertransfer.archivete.am/JoDwa/0domains-sorted.log here's a sampling of domains with the "=0" code, there's some patterns that stood out to me:
-
imerobviously bogus tlds like .buzzcdn and .bestcdn, the [numbers].domain.tld counting those: transfer.archivete.am/FKIGV/2023-10-30_18-23-04.txt and some obviously bogus ones (domain finder too broad now?)
-
imernot sure if any of those are direct cause for concern, just curious there's seemingly lots of bad domains currently
-
arkiverhaving some dinner and then giving this some more attention, will probably move some stuff around too
-
arkiverand we might stop queuing all URLs found on pages of 'special interest'
-
arkiversince it may be a bit too much for now, we can retry that later next month
-
arkivermoving backfeed to secondary
-
AKI'm working on setting up some logging so I can track the status code 0 domains too
-
fireonlivewhat's for dinner, arkiver :3
-
arkiverfireonlive: salad with salmon :P
-
fireonlive=]