00:00:48 <JAA> <1% airbest.com now
00:18:39 <datechnoman> Nice. Great to see it clearing out :)
00:18:51 <datechnoman> Prob getting spam from #mataringa also
08:32:39 <datechnoman> JAA - Requesting/thoughts on filtering out "afy.ru". Literally processing thousands of these rules a minute and personally I dont think its worth archiving/using resources on? (Its a Russian real estate, land, agency and consulation site). We have been running back on these for weeks from what I can see in my logs
08:34:13 <datechnoman> My workers alone seem to be processing 40k links every 5 mins. Stacks of CSS and Images and stuff
08:38:05 <datechnoman> Thinking of being more efficient with what we target/process
08:45:30 <JAA> datechnoman: Interesting, it doesn't look bad on the tracker side at all. But I guess 3% of 76M still add up.
08:48:17 <JAA> It looks like we ran into the site before here, almost two years ago.
08:48:44 <JAA> Also a log mention from last year.
08:51:58 <JAA> We've been hitting them on and off for months, it seems.
08:52:59 <JAA> Fairly consistently since mid-Nov.
08:54:16 <JAA> arkiver: ^ What do you think?
08:59:34 <arkiver> if they're mostly in claims now, i'd say leave them
08:59:41 <arkiver> they'll be drained slowly
08:59:58 <arkiver> they may have been queued to some problematic reason in the past and ended up mostly in claims
09:00:20 <arkiver> given that todo:backfeed is not exploding, we're not dealing with a loop, so no need to serious action i think
09:00:27 <arkiver> or well
09:00:43 <arkiver> we _may_ be dealing with a loop, but not a loop that expands itself
09:00:47 <arkiver> so we'll see
09:00:53 <arkiver> datechnoman: JAA ^
09:01:17 <JAA> Yeah, at the moment, it's stable at 3% of claims.
09:02:19 <JAA> todo:backfeed is getting drained immediately, so a bit hard to see what's going through it.
09:03:20 <JAA> On repeated sampling, I occasionally see a couple images or tracking pixels on it, but nothing else.
10:06:34 <datechnoman> ok no worries. Thanks for having a look atleast. Sorry to take up your time
10:17:06 <imer> don’t know if we did end up filtering out the realtytexas site but that was similar (with the site dying under load)
10:17:25 <imer> can’t hurt to ask :)
10:21:20 <JAA> I don't remember either, but at least this site remains responsive under load.
10:23:35 <datechnoman> It handles the load just fine. Wasnt sure with the huge stashes we had, if we were going to optimise a little :)
10:28:13 <imer> very happy to see we're running empty now as well :) ~1 week for out to run down
10:30:47 <arkiver> datechnoman: no need for sorry at all :)
10:31:04 <arkiver> ideas are always welcome!
10:37:37 <DLoader> Have seen afy.ru returning 503 back in the day, but it didn't seem like an issue so didn't even mention it
10:37:51 <DLoader> this on the other hand is interesting, takes quite a lot of CPU https://transfer.archivete.am/inline/1aL9L/fonts-api.wp.com.txt
10:38:06 <DLoader> dunno if we want to do anything to that though, not even sure where that comes from