-
IDKRoblox servers are getting kinda wacky recently
-
IDKjust a notice
-
HifihedgehogHey JAA! Just reaching out since some users are worried that you might not have enough time to pull down the site content. Are you guys in good shape to get the content all pulled before the 31st and the plug is pulled? Referring to TechnologyGuide sites, that is.
-
JAAWe don't know yet. It appears that they have quite ridiculous rate limits. Their robots.txt advises a 1 second delay between requests, but even that's not sufficient.
-
HifihedgehogOh boy. I wish I would have reached out sooner. I sure hope they don't close any sooner than the 31st given this.
-
OrIdow6Are these blocks or capacity issues?
-
JAABlocks with 403s
-
HifihedgehogThat probably explains why when I tried to scrape the site (not at all like you guys do; just used a simple Node.JS script for scraping Xenforo forums) I would get blocked out.
-
OrIdow6Would something distributed be warranted in this case?
-
JAANotebookReview: 650k threads, 9.35M posts
-
JAATabletPCReview: 57k threads, 521k posts
-
JAABrighthand: 274k threads, 1.88M posts
-
JAADigitalCameraReview: 13k threads, 58k posts
-
JAAAnd that doesn't yet include the websites, only the forums.
-
JAASo we're looking at well over 2M requests (when including deleted thread IDs and pagination). The acceptable delay between requests is more than 3.75 seconds; that was the last attempt with AB and got banned.
-
OrIdow6arkiver: How do you feel about doing the above as a backfeed recursive crawl? A la #Y
-
arkiverdeadline is 31st?
-
arkiverOrIdow6: may be good
-
JAAYes, 'will close Jan 31, 2022'.
-
arkiveractually
-
arkiverOrIdow6: JAA: i can probably rewrite urls-grab today to get something like that going
-
arkiverthen we can use it for those sites and others like it in the future
-
arkivereventually that project can turn into #Y
-
arkiverwill ping back later
-
arkiverurls-grab has most of the required code already, a lot can be filtered out, and then we have what we need
-
OrIdow6arkiver: Alright
-
OrIdow6If it turns out you don't have time I can probably rig up something inflexible
-
MegameFederated Networks Association shutting down February 1st
-
Megame
-
OrIdow6Shutdowns on 1 Feb and 1 Mar
-
OrIdow6Megame: Have you added to Deathwatch? If not I can
-
MegamePlease add it OrIdow6
-
h2ibotOrIdow6 edited Deathwatch (+735, /* 2022 */ Add Feneas stuff): wiki.archiveteam.org/?diff=48202&oldid=48197
-
h2ibotOrIdow6 edited Deathwatch (+59, "On or soon after"): wiki.archiveteam.org/?diff=48203&oldid=48202
-
h2ibotTheTechRobo edited Exploring by the Seat of your Pants (+111, I'm going to yt-dlp it): wiki.archiveteam.org/?diff=48204&oldid=47636
-
h2ibotArkiver uploaded File:Fandom-icon.png: wiki.archiveteam.org/?title=File%3AFandom-icon.png
-
h2ibotArkiver uploaded File:Fandom-logo.png: wiki.archiveteam.org/?title=File%3AFandom-logo.png