-
TheTechRobokelly, if you're reading logs: i believe the item is archiveteam_urls_20201102030531_9c6f61f7/urls_20201102030531_9c6f61f7.megawarc.warc.gz
-
TheTechRobokeep in mind that while the chances are low, page requisites might be in a different megawarc
-
TheTechRobo(fwiw, i got that megawarc from headers; more specifically x-archive-src)
-
JAAPage requisites go through the tracker again, don't they? So there's actually a decent chance they might be in a different WARC.
-
TheTechRoboJAA: I thought just outlinks go through the tracker again
-
TheTechRobocould be wrong though
-
JAAWe wouldn't dedupe requisites if they didn't.
-
TheTechRoboright
-
TheTechRoboso uh yeah you'll probably have to go through the page requisites manually :/
-
TheTechRoboor at least check the warc and make sure everything's there, but that's easy to mess up
-
upintheairsheepI have since collected some more open directory links from the Internet.
-
upintheairsheep
-
upintheairsheep
-
upintheairsheep
-
upintheairsheepOld iOS hacking related open directories to AB
-
upintheairsheep185.8.7.44 an IP address filled with pictures of showers and other plumbing-related stuff, and I have tended to notice that IP address websites are WAY more volatile than their named counterparts
-
upintheairsheeps1.bitdl.ir - More iOS apps in this link
-
upintheairsheepte.censoft.com/download - additional iOS software
-
upintheairsheepmarcoalima.com - More iOS software
-
upintheairsheepdoc.downloadha.com - More iOS software, however it has ~200 GB of non-iOS zip files, maybe set an exclusion for those
-
upintheairsheepdl.ievo.top - Various softwares
-
upintheairsheepfiles.zmodo.com - Downloads OD of some sort of TV company?
-
upintheairsheepcarboncostume.com/wordpress/wp-content/uploads - Pictures of haloween costumes
-
upintheairsheepgwern.net/images - Memes and IT-related images, including AI ones
-
upintheairsheeppierrepapierciseaux.net/.skynet/img - images for Windows 93 SkyNethttps://pierrepapierciseaux.net/.skynet/midi/ - ditto for MIDIhttps://pierrepapierciseaux.net/.skynet/js/ - ditto for JS
-
upintheairsheepsmallake.kr/wp-content/uploads - Various academic papers
-
upintheairsheepstar-www.st-and.ac.uk/~kdh1/ada - University course about data
-
upintheairsheepcatsarecute.xyz - Random small OD site, however the strict rate limits may make the site harder for crawlers to archive
-
upintheairsheepftp.dexp.club - Another medium firmware site
-
upintheairsheeppastebin.com/dKSmBM7x - The rest of the OD links I have gathered online, make sure to remove the dashes on two of them
-
upintheairsheep142.11.229.92 - IP address site, has gone down a few days ago
-
TheTechRoboWhy is this in the URLs project?
-
TheTechRobo*URLs irc channel
-
TheTechRobobetter for archivebot
-
datechnomanUnfortunately we cannot archive these links here as we could potentially take them off-line with a DDOS. As techrobo has said the archivebot channel is much better suited :)
-
datechnomanIt will conduct a much slower (rate limited) crawl of the sites