15:58:46 JAA: can we nuke *.xuite.net? seeing 25/s for photo.xuite.net here, 9/s for api.xuite.net, both down 15:59:32 also probably need a rate limit on live.staticflickr.com, seeing 75/s here and getting 429's "

429 Too Many Requests

You have sent too many requests in a given amount of time." 15:59:52 if we want to archive those images ^, otherwise nuke too 16:00:22 for example: https://transfer.archivete.am/2h3eJ/2024-01-29_16-00-07.txt 16:00:23 inline (for browser viewing): https://transfer.archivete.am/inline/2h3eJ/2024-01-29_16-00-07.txt 16:02:02 seem to be coming from actual flickr pages: https://transfer.archivete.am/NwiER/2024-01-29_16-01-47.txt 16:02:02 inline (for browser viewing): https://transfer.archivete.am/inline/NwiER/2024-01-29_16-01-47.txt 16:38:52 imer (Cc arkiver): *.xuite.net is being filtered now. 16:39:36 thanks 16:39:46 I'm going to limit https://live.staticflickr.com/* to 1000 for now, let's see if that helps. 16:40:46 Or maybe we need to take a break for now and ramp it up again later? 16:41:34 unfortunately no indication what the rate limit might be in headers :( 16:41:39 ~40% of todo:backfeed is those currently. 16:41:58 does rate limiting pull from the next queue if it skips an item? 16:42:10 IIRC no 16:42:15 damn. 16:42:33 oh, seeing 200s here! 16:42:37 It grabs N items from the queues, then applies limits and filters, then returns whatever remains. 16:42:56 Nice :-) 16:43:25 does rate limiting calculate from when the limit is set or is it also retroactive? 16:44:07 could try a bit higher if its only from when the limit is set 16:44:08 No idea how it works immediately after the addition, but it's irrelevant after a minute. 16:44:23 although unsure if there's people running more per ip than me 16:45:53 2k 16:46:39 I wonder where all this Flickr stuff is coming from all of a sudden. 16:48:46 cant spot anything in my logs, although might've rotated out by now 16:52:12 JAA: still looking good here, try 4k? 16:52:25 4k 17:00:48 still fine, not sure how much more micro management you want to do though :D 17:01:58 For some value of 'fine'. 17:02:10 ~70% of todo:backfeed is now *//live.staticflickr.com/* 17:02:32 that explains the irsr creeping down slowly 17:02:44 I guess claims are still close to the limit anyway for now, so it doesn't matter too much, but yeah. 17:29:19 let me see 19:02:11 is staticflickr actually a problem? 19:02:32 ah 429s 19:08:29 where did it come from 19:33:43 well todo:backfeed seems to be going down anyway 20:58:38 could probably turn up the rate limit some 21:34:29 imer: 8k 22:28:45 JAA: when these URLs are done, please don't forget to remove the limit 22:30:13 arkiver: I'm specifically limiting *//live.staticflickr.com/*, and we'd run into the 429 wall again on surges in the future, so maybe it wouldn't hurt to keep it? 22:31:50 Or more precisely, it's a patternlimit on ^https?://live%.staticflickr%.com/ .