03:55:41 <pabs> this person regularly deletes older blog posts, would it be appropriate to use this project to save their new posts? http://blog.sesse.net/
03:56:01 <pabs> curren't I'm just doing AB !ao but it would be nice to automate it
08:49:38 <AK> arkiver will be able to correct me, but I guess a monthly queue of the homepage with a depth of 1 should mean we grab each new article
10:23:12 <DLoader> Got abuse from theanarchistlibrary.org
10:23:19 <DLoader> We suspect this may be from a botnet, as we have been hit by a number of requests from this the IP above as well as several others. It is a resource exhaustion attack against the webapp running at theanarchistlibrary.org resolvable to 216.252.162.140.
10:25:36 <AK> Seeing quite a few status code 0 from a couple of domains: https://share.aktheknight.co.uk/riJe0/NApuloJa91.png/raw
10:28:07 <DLoader> itunese... doesn't exist anymore, but it's still seen quite a lot. digital-forum.it seems to have banned Hetzner, esug.org is just some 302 redirect always
10:28:13 <DLoader> *itunesu
10:28:49 <DLoader> arkiver maybe something to exclude ^
15:50:03 <nstrom|m> digital-forum.it seems fine on all my non hetz workers
15:51:31 <nstrom|m> I feel like we're looping on silky-europe.com due to session ids in url query strings but can't find any actual evidence of that. just seeing a lot of it in my logs
15:51:34 <nstrom|m> example url https://www.silky-europe.com/arborist/fixed-saw?SID=8179f580143d7c2b60b8c9cd3d8e1448
15:52:48 <nstrom|m> (also .nl version, e.g. https://www.silky-europe.nl/boomverzorger/vouwzagen?SID=08e9309aa89bf082ea7245d89d8635b5   )
16:06:21 <arkiver> yeah
17:14:55 <arkiver> getting my brush
17:22:13 <arkiver> putting the trash in our little /dev/null corner
17:33:03 <arkiver> pabs: AK is correct. you can make a PR on urls-sources
17:33:15 <arkiver> or else i'll add it later
17:40:10 <arkiver> so this seems to be a bunch of spam sites coming from China aimed at Thailand
17:43:27 <arkiver> we have indeed been getting quite a bit of anarchistlibary
17:44:08 <arkiver> partially filtering it
17:45:43 <arkiver> nstrom|m: indeed SID was not one we checked for, only sid
17:45:54 <arkiver> fixing the forum URLs too
17:48:36 <arkiver> cleaning done
17:48:43 <arkiver> forcing new version
17:49:16 <arkiver> new minimum version is set
17:49:25 <arkiver> moving todo:backfeed to todo:secondary
18:26:56 <nstrom|m> danke
18:32:20 <AK> Thanks
18:52:44 <imer> ooh, we're actually making progress now. many thanks :)
20:00:20 <arkiver> still seem to be having a problem
20:00:40 <arkiver> different one though
20:19:47 <arkiver> but where do those bad URLs come from...
20:28:40 <imer> arkiver: which ones specifically? can dig through my logs if you want
20:32:44 <arkiver> these odd .de site (without ending /)
20:32:49 <arkiver> trailing /, i should say
20:41:45 <nstrom|m> one of them actually loaded once for me but otherwise nearly all seem to be broken
20:42:17 <arkiver> yeah
20:42:20 <arkiver> the problem is where they come from
20:42:24 <arkiver> i don't see them being queued
20:42:47 <arkiver> if you see them being queued somewhere, please give my a bit of log (including that URL, and the parent URL mentioned a little earlier)
20:43:27 <nstrom|m> roger that
20:45:50 <nstrom|m> 2023-11-08T20:35:12.825451117Z Queuing for parent URL https://bpgqt.petrography.de/app-ads.txt.
20:45:51 <nstrom|m> 2023-11-08T20:35:12.825602179Z Queuing URL https://6pi8.stoutly.de.
20:46:05 <arkiver> nice nice
20:46:09 <arkiver> very nice thanks nstrom|m
20:46:17 <nstrom|m> 2023-11-08T20:37:24.035748601Z Queuing for parent URL https://cbxrt.sociably.de/app-ads.txt.
20:46:17 <nstrom|m> 2023-11-08T20:37:24.036022264Z Queuing URL https://78k7y.selflimited.de.
20:46:17 <nstrom|m> 2023-11-08T20:37:24.036036848Z Queuing for parent URL https://68q3z.theorize.de/ads.txt.
20:46:17 <nstrom|m> 2023-11-08T20:37:24.036042207Z Queuing URL https://9udw2.peripatetic.de.
20:46:24 <nstrom|m> hope that helps
20:46:27 <arkiver> greatly!
20:46:30 <arkiver> i see the problem now
20:47:26 <arkiver> project paused a bit
20:53:05 <arkiver> interesting way of doing spam
20:53:52 <arkiver> i'll disable this queuing for now
20:55:56 <arkiver> they could do something similar with trust.txt but leaving logic for that in for now
20:57:25 <arkiver> an update is in
20:57:46 <arkiver> restarted
21:01:29 <arkiver> looking pretty good
21:01:31 <arkiver> thanks nstrom|m :)
21:04:46 <arkiver> it's not blowing up :)
21:07:08 <arkiver> or maybe it is
21:09:03 <DLoader> https://a1oel.spidery.de/.well-known/security.txt, https://2n4w8.epithelial.de/.well-known/nodeinfo, https://2h9lu.predestine.de/.well-known/dnt-policy.txt
21:09:06 <DLoader> saw these
21:09:18 <DLoader> maybe it's all or most of the special urls
21:09:18 <arkiver> yeah but not sure if it's coming through those
21:10:35 <JAA> Queuing for parent URL https://44rw5.discriminate.de/.well-known/ai-plugin.json.
21:10:35 <JAA> Queuing URL custom:random=202311&url=https%3a%2f%2f4i792%2emeagre%2ede%2f%2ewell%2dknown%2fsecurity%2etxt.
21:11:12 <JAA> And then quite a list of them.
21:11:28 <arkiver> blegh :/
21:11:40 <JAA> AK's logs are helpful again. :-)
21:12:43 <arkiver> actually those are 301s
21:13:13 <arkiver> right
21:13:17 <arkiver> well that's an annoying loop
21:13:30 <DLoader> # curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0)100101 Firefox/119.0" https://5otqf.reprieve.de
21:13:30 <DLoader> FUCK YOU!
21:13:31 <DLoader> :D
21:20:07 <arkiver> updates again
21:20:23 <arkiver> DLoader: that real?
21:20:34 * arkiver check
21:21:09 <arkiver> no :P
21:24:14 <DLoader> I still get that, but not on my Hetzner servers
21:24:26 <DLoader> so dunno what they doing
21:36:46 <DLoader> Queuing for parent URL https://9b953.resemblance.de/.well-known/gpc.json.
21:36:46 <DLoader> Queuing URL custom:random=202311&url=https%3a%2f%2f1mgu%2epainstaking%2ede%2fsitemap%2exml.
21:36:54 <DLoader> still seeing this arkiver
21:59:47 <arkiver> hmm
22:05:09 <arkiver> DLoader: are you sure that second line belonged with that first line?
22:06:14 <JAA> arkiver: Still seeing similar things in AK's logs.
22:06:40 <JAA> And I think those are set to 1 concurrency to avoid log mixing etc.
22:07:30 <arkiver> yeah i see them too now
22:07:37 <arkiver> (couldn't check earlier)
22:08:21 <DLoader> just uploaded some more :D https://transfer.archivete.am/inline/94ijC/dedede
22:08:32 <arkiver> i see the problem
22:11:28 <arkiver> another fix pushed
22:11:32 <arkiver> this should really fix it now
22:11:41 <arkiver> will leave the queue as is, should go down fast
22:31:55 <arkiver> alright not looking bad
22:32:08 <arkiver> will move :backfeed to :secondary
22:39:16 <arkiver> looks good!
23:28:11 <pabs> arkiver: posted to https://github.com/ArchiveTeam/urls-sources/pull/27
23:28:59 <pabs> arkiver: and I have one other PR open: https://github.com/ArchiveTeam/urls-sources/pull/22