16:18:51 Hmm why is our completed items so low 16:23:26 JAA/arkiver: ebedzin.pl might need nuking, very long urls, server is rejecting most requests https://transfer.archivete.am/14vAIk/2023-10-30_16-22-17.txt 16:23:26 AK: also seeing a bunch of timeouts for domains on my end 16:23:49 Ahh that might be it, I really need to setup better logging+aggregation of logs for my containers 16:26:32 9=0 https://unshieldedcables...................................5.3.2higher-frequencyoutages......... yup. 16:27:53 imer: are you using anything to collect logs from the containers? A long time ago I had logstash and elasticsearch but it didn't end well haha 16:28:18 nope, just docker logs -f and looking at stuff scrolling by 16:29:24 I have this atrocity: `((for i in {1..40}; do sudo docker logs -n0 -f at-urls$i & done); wait ) | rg '[0-9]+=[0-9]+ https?://([^/]+)/' -a -or '$1' | logtop` 16:36:38 (opens a live log for all containers and greps out the domain, then feeds into logtop so you see stats per domain) 16:41:24 Might try setting up opensearch again, I had stuff like this which could show me all the percentages by domain so I was able to tell if we were being banned or not 16:41:25 https://share.aktheknight.co.uk/riJe0/laguyoto34.png/raw 17:12:12 checking 17:12:29 oof 18:13:30 that doesn't sound good 18:13:59 it's being filtered 18:14:06 that does sound good 18:14:13 :) 18:14:17 :P 18:15:23 imer: nah that sounds horrible :D 18:28:56 https://transfer.archivete.am/JoDwa/0domains-sorted.log here's a sampling of domains with the "=0" code, there's some patterns that stood out to me: 18:28:56 obviously bogus tlds like .buzzcdn and .bestcdn, the [numbers].domain.tld counting those: https://transfer.archivete.am/FKIGV/2023-10-30_18-23-04.txt and some obviously bogus ones (domain finder too broad now?) 18:28:56 not sure if any of those are direct cause for concern, just curious there's seemingly lots of bad domains currently 18:30:18 having some dinner and then giving this some more attention, will probably move some stuff around too 18:30:30 and we might stop queuing all URLs found on pages of 'special interest' 18:30:40 since it may be a bit too much for now, we can retry that later next month 18:32:54 moving backfeed to secondary 19:32:08 I'm working on setting up some logging so I can track the status code 0 domains too 19:56:31 what's for dinner, arkiver :3 21:29:22 fireonlive: salad with salmon :P 21:29:58 =]