00:29:32 <arkiver> this will take a while it seems
00:29:37 <arkiver> will fix tomorrow
11:34:44 <datechnoman> arkiver - Seem to be getting millions of hits on spam domains again. They all appear to have the same kind of structure so hopefully it will make it a little easier to clear them out. Here are some examples from my workers. More than happy to provide more examples if required:
11:34:45 <datechnoman> https://transfer.datechnoman.com/get/0slhsGUD1o/spam_domains_2023_03_22.txt
11:35:21 <datechnoman> Looks like the /news/ is working and picked up the spam domains lol
11:43:02 <datechnoman> Silly loops popping out everywhere :o
11:43:12 <datechnoman> Hand me my rifle....
13:57:11 <arkiver> yeaj
13:57:18 <arkiver> those are the same ones I was talking about previously
13:57:34 <arkiver> they are similar to something that was blocked a week ago or so, then they had /en/
13:57:47 <arkiver> now /news/ and /pages/. looking into blocking them with the new method
14:12:42 <arkiver> blegh queue has grown to 45 million already
14:12:51 <arkiver> luckily these loops are rare.
14:12:57 <arkiver> fix coming up, will take some time
14:31:18 <AK> What's the source of todo:secondary?
14:33:10 <arkiver> whatever I put into it
14:33:25 <arkiver> usually when todo:backfeed becomes very large, I push it to todo:secondary to create room
14:33:43 <arkiver> because what we want overall is for todo:backfeed to stay near 0. this means we're 'healthy'
14:34:31 <arkiver> but if we have a big queue there after a pause or other problem, it get's in the way checking if we're 'healthy', so I push the excess todo:backfeed as backlog to todo:secondary from where it'll be slowly consumed
14:34:38 <AK> Ahh okay, so we still want to get through secondary, but it won't keep growing like backfeed does
14:34:46 <AK> (Except when you put more into it of course)
14:35:25 <AK> all the todo are going down, does the tracker pull a bit from everything when handing out tasks?
14:36:18 <arkiver> order of importance: todo, todo:backfeed, todo:secondary, todo:redo
14:36:39 <arkiver> it'll start with todo, then whatever 'room' is left it, it'll get form the next queue, etc.
14:37:02 <arkiver> also filtered items. if a requested item gets filtered out, it'll request an item from the next queue, etc.
14:37:39 <AK> Makes sense
14:38:11 <AK> https://share.aktheknight.co.uk/riJe0/VUyAGIFI21.png/raw All the cpu
14:38:36 <arkiver> yeah :/
14:38:42 <arkiver> i hope to improve that soon
14:39:37 <AK> I think I also probably overload this box, at the moment it runs 20 concurrent per thread, might be a case of me needing to slow down a bit
14:41:22 <arkiver> it could contribute to it yeah
21:24:00 <datechnoman> Thanks for the update. Wasnt sure if you were aware but sounds like your already on that one :)
23:18:17 <datechnoman> !a https://transfer.archivete.am/R4754/goo-gl.2023-01-01-04-17-02.txt
23:47:18 <datechnoman> Looks like the import script broke again
23:51:39 <arkiver> right
23:51:46 <arkiver> power outage last night... :/
23:51:55 <arkiver> should let that start automatically
23:52:47 <datechnoman> I didnt do it!
23:53:58 <arkiver> guilty until proven innocent for me :P
23:55:52 <datechnoman> Haha look, your just the go to person
23:56:04 <datechnoman> Both for getting things fixed and also to be blamed :P
23:59:44 <datechnoman> When items are uploaded to the todo queue through the bot are they added to the start of that queue or the end?