18:18:28 Roblox servers are getting kinda wacky recently 18:18:35 just a notice 18:28:12 Hey JAA! Just reaching out since some users are worried that you might not have enough time to pull down the site content. Are you guys in good shape to get the content all pulled before the 31st and the plug is pulled? Referring to TechnologyGuide sites, that is. 18:28:54 We don't know yet. It appears that they have quite ridiculous rate limits. Their robots.txt advises a 1 second delay between requests, but even that's not sufficient. 18:33:42 Oh boy. I wish I would have reached out sooner. I sure hope they don't close any sooner than the 31st given this. 18:36:17 Are these blocks or capacity issues? 18:39:52 Blocks with 403s 18:41:37 That probably explains why when I tried to scrape the site (not at all like you guys do; just used a simple Node.JS script for scraping Xenforo forums) I would get blocked out. 18:44:06 Would something distributed be warranted in this case? 18:46:14 NotebookReview: 650k threads, 9.35M posts 18:46:27 TabletPCReview: 57k threads, 521k posts 18:46:39 Brighthand: 274k threads, 1.88M posts 18:46:47 DigitalCameraReview: 13k threads, 58k posts 18:47:49 And that doesn't yet include the websites, only the forums. 18:52:35 So we're looking at well over 2M requests (when including deleted thread IDs and pagination). The acceptable delay between requests is more than 3.75 seconds; that was the last attempt with AB and got banned. 19:03:07 arkiver: How do you feel about doing the above as a backfeed recursive crawl? A la #Y 19:07:54 deadline is 31st? 19:07:59 OrIdow6: may be good 19:10:15 Yes, 'will close Jan 31, 2022'. 19:10:32 actually 19:10:52 OrIdow6: JAA: i can probably rewrite urls-grab today to get something like that going 19:11:11 then we can use it for those sites and others like it in the future 19:11:17 eventually that project can turn into #Y 19:11:27 will ping back later 19:11:59 urls-grab has most of the required code already, a lot can be filtered out, and then we have what we need 19:16:56 arkiver: Alright 19:18:21 If it turns out you don't have time I can probably rig up something inflexible 19:51:46 Federated Networks Association shutting down February 1st 19:51:59 More info - https://old.reddit.com/r/Archiveteam/comments/s05wk0/federated_networks_association_ry_dissolving/ 19:59:04 Shutdowns on 1 Feb and 1 Mar 19:59:28 Megame: Have you added to Deathwatch? If not I can 20:02:19 Please add it OrIdow6 20:16:38 OrIdow6 edited Deathwatch (+735, /* 2022 */ Add Feneas stuff): https://wiki.archiveteam.org/?diff=48202&oldid=48197 20:19:39 OrIdow6 edited Deathwatch (+59, "On or soon after"): https://wiki.archiveteam.org/?diff=48203&oldid=48202 22:23:08 TheTechRobo edited Exploring by the Seat of your Pants (+111, I'm going to yt-dlp it): https://wiki.archiveteam.org/?diff=48204&oldid=47636 22:25:08 Arkiver uploaded File:Fandom-icon.png: https://wiki.archiveteam.org/?title=File%3AFandom-icon.png 22:25:09 Arkiver uploaded File:Fandom-logo.png: https://wiki.archiveteam.org/?title=File%3AFandom-logo.png