-
imer
JAA: can we nuke *.xuite.net? seeing 25/s for photo.xuite.net here, 9/s for api.xuite.net, both down
-
imer
also probably need a rate limit on live.staticflickr.com, seeing 75/s here and getting 429's "<html><body><h1>429 Too Many Requests</h1>You have sent too many requests in a given amount of time.</body></html>"
-
imer
if we want to archive those images ^, otherwise nuke too
-
imer
-
eggdrop
-
imer
-
eggdrop
-
JAA
imer (Cc arkiver): *.xuite.net is being filtered now.
-
imer
thanks
-
JAA
I'm going to limit
live.staticflickr.com/* to 1000 for now, let's see if that helps.
-
JAA
Or maybe we need to take a break for now and ramp it up again later?
-
imer
unfortunately no indication what the rate limit might be in headers :(
-
JAA
~40% of todo:backfeed is those currently.
-
imer
does rate limiting pull from the next queue if it skips an item?
-
JAA
IIRC no
-
imer
damn.
-
imer
oh, seeing 200s here!
-
JAA
It grabs N items from the queues, then applies limits and filters, then returns whatever remains.
-
JAA
Nice :-)
-
imer
does rate limiting calculate from when the limit is set or is it also retroactive?
-
imer
could try a bit higher if its only from when the limit is set
-
JAA
No idea how it works immediately after the addition, but it's irrelevant after a minute.
-
imer
although unsure if there's people running more per ip than me
-
JAA
2k
-
JAA
I wonder where all this Flickr stuff is coming from all of a sudden.
-
imer
cant spot anything in my logs, although might've rotated out by now
-
imer
JAA: still looking good here, try 4k?
-
JAA
4k
-
imer
still fine, not sure how much more micro management you want to do though :D
-
JAA
For some value of 'fine'.
-
JAA
~70% of todo:backfeed is now *//live.staticflickr.com/*
-
imer
that explains the irsr creeping down slowly
-
JAA
I guess claims are still close to the limit anyway for now, so it doesn't matter too much, but yeah.
-
arkiver
let me see
-
arkiver
is staticflickr actually a problem?
-
arkiver
ah 429s
-
arkiver
where did it come from
-
arkiver
well todo:backfeed seems to be going down anyway
-
imer
could probably turn up the rate limit some
-
JAA
imer: 8k
-
arkiver
JAA: when these URLs are done, please don't forget to remove the limit
-
JAA
arkiver: I'm specifically limiting *//live.staticflickr.com/*, and we'd run into the 429 wall again on surges in the future, so maybe it wouldn't hurt to keep it?
-
JAA
Or more precisely, it's a patternlimit on ^https?://live%.staticflickr%.com/ .