-
JAA
<1% airbest.com now
-
datechnoman
Nice. Great to see it clearing out :)
-
datechnoman
Prob getting spam from #mataringa also
-
datechnoman
JAA - Requesting/thoughts on filtering out "afy.ru". Literally processing thousands of these rules a minute and personally I dont think its worth archiving/using resources on? (Its a Russian real estate, land, agency and consulation site). We have been running back on these for weeks from what I can see in my logs
-
datechnoman
My workers alone seem to be processing 40k links every 5 mins. Stacks of CSS and Images and stuff
-
datechnoman
Thinking of being more efficient with what we target/process
-
JAA
datechnoman: Interesting, it doesn't look bad on the tracker side at all. But I guess 3% of 76M still add up.
-
JAA
It looks like we ran into the site before here, almost two years ago.
-
JAA
Also a log mention from last year.
-
JAA
We've been hitting them on and off for months, it seems.
-
JAA
Fairly consistently since mid-Nov.
-
JAA
arkiver: ^ What do you think?
-
arkiver
if they're mostly in claims now, i'd say leave them
-
arkiver
they'll be drained slowly
-
arkiver
they may have been queued to some problematic reason in the past and ended up mostly in claims
-
arkiver
given that todo:backfeed is not exploding, we're not dealing with a loop, so no need to serious action i think
-
arkiver
or well
-
arkiver
we _may_ be dealing with a loop, but not a loop that expands itself
-
arkiver
so we'll see
-
arkiver
datechnoman: JAA ^
-
JAA
Yeah, at the moment, it's stable at 3% of claims.
-
JAA
todo:backfeed is getting drained immediately, so a bit hard to see what's going through it.
-
JAA
On repeated sampling, I occasionally see a couple images or tracking pixels on it, but nothing else.
-
datechnoman
ok no worries. Thanks for having a look atleast. Sorry to take up your time
-
imer
don’t know if we did end up filtering out the realtytexas site but that was similar (with the site dying under load)
-
imer
can’t hurt to ask :)
-
JAA
I don't remember either, but at least this site remains responsive under load.
-
datechnoman
It handles the load just fine. Wasnt sure with the huge stashes we had, if we were going to optimise a little :)
-
imer
very happy to see we're running empty now as well :) ~1 week for out to run down
-
arkiver
datechnoman: no need for sorry at all :)
-
arkiver
ideas are always welcome!
-
DLoader
Have seen afy.ru returning 503 back in the day, but it didn't seem like an issue so didn't even mention it
-
DLoader
this on the other hand is interesting, takes quite a lot of CPU
transfer.archivete.am/inline/1aL9L/fonts-api.wp.com.txt
-
DLoader
dunno if we want to do anything to that though, not even sure where that comes from