00:54:54 Looks like I managed to find one target at some point during the night but my warrior cries for the sweet release of targets with an item thats been trying to upload for nearly 2 days 00:55:37 Where’s the old wiki? The old wiki can still be accessed on archive.org. 00:55:40 uh ok 00:55:57 https://nixos.wiki/wiki/Main_Page < was that archivebotted 00:56:14 nixos.org/wiki* 00:56:18 for the old one 00:56:28 in 2016: https://archive.fart.website/archivebot/viewer/job/20160829185202bmuqr 00:57:02 is there any way to check the capacity of our temp storage? 00:57:03 ah! awesome :) 00:57:25 Looks like that matches the timestamp they link to 00:57:35 so, should be complete 00:59:01 ^_^ 00:59:12 that's one way to pass it off! 00:59:18 glad it was saved though 01:09:48 flashfire42: there is capacity, but very few servers with storage now, so similar situation as IA where data can't be offloaded fast enough :| 01:10:38 so is temp storage starting to be offloaded to IA now? 01:10:53 not currently, no 01:11:12 offloading as in data on targets -> temp storage 01:11:40 and targets as in target :D 01:12:18 that + some bugs in the new tech for temp storage which don't help either 01:12:43 GitHubElapsed: 01d 22h 43m 52s 01:12:47 thats all 01:21:21 Also a question about how duplicate warrior items are handled? do they just get deleted or are they both archived. Say I take a super long time uploading a 10GB video from downthetube and someone else gets it due to TTL and uploads it before me does my upload get discarded when it finally arrives? 01:51:09 -+rss- Insiders reveal major problems at lab-grown meat startup: https://www.wired.com/story/upside-foods-lab-grown-chicken/ https://news.ycombinator.com/item?id=37540067 01:52:04 if they're menitoned in there - i can't quite tell right now... might be worth a archivebot or something 01:52:25 ah https://upsidefoods.com/ i think 01:53:33 one of these "claims to be yahoo videos but it's actually friendster" tarballs I'm indexing has 6 million files and counting 01:58:43 can we collect a list for later moving stuff around perhaps? https://pad.notkiska.pw/p/YahooVideos%26Friendster 03:02:19 I'll try to find if there are any files in the friendster archive that are actually yahoo video I guess 03:36:45 aaand they're almost all compressed rip 05:43:42 Howdy, I'm looking for archives of a defunct widget-hosting website called Widgetbox, referred to here: https://wiki.archiveteam.org/index.php/Widgetbox 05:43:51 I can't seem to find any on the Internet Archive, does anyone know where they might be? 05:50:17 Hmm, most archiveteam projects are on web.archive.org and the backing data is on archive.org. I'll do a quick check 05:52:12 Looking at https://web.archive.org/web/collections/20140915000000*/http://blog.widgetbox.com/ it seems like there's stuff in https://archive.org/details/arkivercrawls - specifically that page is somewhere in https://archive.org/details/arkiver20140612-2 it seems (based on the x-archive-src header) 05:52:24 I don't see any kind of organization to it, but arkiver probably knows more :) 05:52:41 (you might need to a wait a while for time zones to line up though) 05:54:03 specifically that page is on WEB-20140221120837710-00000-4480~Titan~8443.warc.gz, which just looks like a random file in that group of files 06:05:53 Thanks, I'll take a look in there. I emailed Arkiver back on the 6th and haven't heard back, that's cool if he's in here sometimes. 06:09:28 jacobgkau: the situation is basically thusly https://mkx9delh5a.execute-api.ca-central-1.amazonaws.com/uploads/c7743b41c33e6600/arkiver.png 06:12:26 project10: Lol. I can see how that would happen with the massive amount of data being saved and not having the chance to organize it a ton. 06:12:50 I'm looking for some specific widgets that used to be on WidgetBox's widgetserver domain, looks like I have some digging to do. 07:05:04 Sanqui job has already been running for a bit 07:05:30 Knowing its one of my jobs at some point it probably ran wildly off course 07:05:38 oh, I see it going 07:06:00 well, I started my job with no-offsite, and it's on a different pipeline, so might as well keep both running 07:06:09 Agreed 08:53:25 Looks like the Nixos wiki could do with an archive https://github.com/nix-community/wiki/issues/46 10:13:43 VoynichCr uploaded File:Glottolog 2023.jpg: https://wiki.archiveteam.org/?title=File%3AGlottolog%202023.jpg 10:14:43 VoynichCr edited Glottolog (+332): https://wiki.archiveteam.org/?diff=50787&oldid=44990 10:20:44 VoynichCr edited Svalbard Global Seed Vault (+9): https://wiki.archiveteam.org/?diff=50788&oldid=50778 10:21:44 VoynichCr edited Svalbard Global Seed Vault (+10): https://wiki.archiveteam.org/?diff=50789&oldid=50788 10:25:45 VoynichCr edited Template:Rescued (+270, categories by year): https://wiki.archiveteam.org/?diff=50790&oldid=24057 10:26:45 VoynichCr created Category:Archived in 2019 (+29, Created page with "[[Category:Archives by year]]"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202019 10:32:46 VoynichCr created Template:Category archives by year (+251, Created page with "This category…): https://wiki.archiveteam.org/?title=Template%3ACategory%20archives%20by%20year 10:32:47 VoynichCr created Category:Archives by year (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchives%20by%20year 10:34:46 VoynichCr edited Template:Category archives by year (+405): https://wiki.archiveteam.org/?diff=50794&oldid=50792 10:35:47 VoynichCr edited Template:Category archives by year (-24): https://wiki.archiveteam.org/?diff=50795&oldid=50794 10:35:48 VoynichCr edited Category:Archived in 2019 (+0): https://wiki.archiveteam.org/?diff=50796&oldid=50791 10:36:47 VoynichCr edited Template:Category archives by year (+6): https://wiki.archiveteam.org/?diff=50797&oldid=50795 10:36:48 VoynichCr created Category:Archived in 2018 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202018 10:36:49 VoynichCr created Category:Archived in 2020 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202020 10:39:47 VoynichCr edited Long Now Foundation (+20): https://wiki.archiveteam.org/?diff=50800&oldid=47956 10:41:48 VoynichCr edited Long Now Foundation (-1): https://wiki.archiveteam.org/?diff=50801&oldid=50800 10:42:48 VoynichCr edited Template:Rescued (+102): https://wiki.archiveteam.org/?diff=50802&oldid=50790 10:43:48 VoynichCr edited Voyager Golden Record (+19, {{saved|date=2018}}): https://wiki.archiveteam.org/?diff=50803&oldid=45225 10:45:48 VoynichCr edited International Internet Preservation Consortium (+19): https://wiki.archiveteam.org/?diff=50804&oldid=47936 10:48:48 VoynichCr edited Arctic World Archive (+41, | archiving_status = {{saved|date=2020}}): https://wiki.archiveteam.org/?diff=50805&oldid=47796 10:50:49 VoynichCr edited Arch Mission Foundation (+8): https://wiki.archiveteam.org/?diff=50806&oldid=47795 10:55:50 VoynichCr edited EcuRed (+26): https://wiki.archiveteam.org/?diff=50807&oldid=47509 10:56:50 VoynichCr created Category:Archived in 2021 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202021 10:56:51 VoynichCr created Category:Archived in 2022 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202022 10:56:52 VoynichCr created Category:Archived in 2023 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202023 10:57:50 VoynichCr edited Template:Category archives by year (+215): https://wiki.archiveteam.org/?diff=50811&oldid=50797 11:00:51 VoynichCr created Category:Archived in 2017 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202017 11:00:52 VoynichCr created Category:Archived in 2016 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202016 11:00:53 VoynichCr created Category:Archived in 2015 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202015 11:00:54 VoynichCr created Category:Archived in 2014 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202014 11:00:55 VoynichCr created Category:Archived in 2013 (+29, Created page with "{{Category archives by year}}"): https://wiki.archiveteam.org/?title=Category%3AArchived%20in%202013 11:05:51 VoynichCr edited Memory of Mankind (+19, {{saved|date=2020}}): https://wiki.archiveteam.org/?diff=50817&oldid=48112 11:06:52 VoynichCr edited Rosetta Project (+19, {{saved|date=2020}}): https://wiki.archiveteam.org/?diff=50818&oldid=45220 11:08:52 VoynichCr edited Endangered Languages Project (+38): https://wiki.archiveteam.org/?diff=50819&oldid=45116 11:08:53 VoynichCr edited Endangered Languages Project (+226): https://wiki.archiveteam.org/?diff=50820&oldid=50819 11:10:53 VoynichCr edited Endangered Languages Project (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50821&oldid=50820 11:13:53 VoynichCr edited Rosetta Project (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50822&oldid=50818 11:13:54 VoynichCr edited Memory of Mankind (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50823&oldid=50817 11:13:55 VoynichCr edited Arch Mission Foundation (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50824&oldid=50806 11:13:56 VoynichCr edited EcuRed (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50825&oldid=50807 11:13:57 VoynichCr edited Arctic World Archive (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50826&oldid=50805 11:13:58 VoynichCr edited International Internet Preservation Consortium (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50827&oldid=50804 11:13:59 VoynichCr edited Voyager Golden Record (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50828&oldid=50803 11:14:53 VoynichCr edited Long Now Foundation (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50829&oldid=50801 11:14:54 VoynichCr edited Svalbard Global Seed Vault (+30, | archiving_type = ArchiveBot): https://wiki.archiveteam.org/?diff=50830&oldid=50789 11:22:16 the fck is going on 💀 11:38:26 I dont know. HOLD ME Exorcism 11:38:41 Also All I know is targets are taking ages to be found by my warrior 13:16:31 flashfire42, Exorcism: What issues are you running into? 13:16:47 rewby: #gitgud 14:18:27 Exorcism edited ZOWA (+107): https://wiki.archiveteam.org/?diff=50831&oldid=50707 14:45:11 https://theintercept.com/2023/09/17/new-york-times-website-internet-archive/ 14:49:31 HadeanEon created Deaths in 2023 (+2735, BOT - Updating page: {{saved}} (0),…): https://wiki.archiveteam.org/?title=Deaths%20in%202023 14:50:32 HadeanEon created Deaths in 2023/list (+360, BOT - Updating list): https://wiki.archiveteam.org/?title=Deaths%20in%202023/list 14:55:39 is HadeanEon back? :o 14:57:33 VoynichCr edited Template:Deathwatch (+292, 2023): https://wiki.archiveteam.org/?diff=50835&oldid=38848 15:05:34 HadeanEon edited Deaths in 2023 (+112414, BOT - Updating page: {{saved}} (0),…): https://wiki.archiveteam.org/?diff=50836&oldid=50833 15:05:35 HadeanEon edited Deaths in 2023/list (+19850, BOT - Updating list): https://wiki.archiveteam.org/?diff=50837&oldid=50834 15:11:08 o: 15:22:37 HadeanEon edited List of current heads of state and government (+9434, BOT - Updating page): https://wiki.archiveteam.org/?diff=50838&oldid=44351 15:22:38 HadeanEon edited List of current heads of state and government/websites-list (+388, BOT - Updating list): https://wiki.archiveteam.org/?diff=50839&oldid=44340 15:22:39 HadeanEon edited List of current heads of state and government/facebook-list (+1482, BOT - Updating list): https://wiki.archiveteam.org/?diff=50840&oldid=44324 15:22:40 HadeanEon edited List of current heads of state and government/instagram-list (+2888, BOT - Updating list): https://wiki.archiveteam.org/?diff=50841&oldid=44200 15:22:41 HadeanEon edited List of current heads of state and government/twitter-list (+1412, BOT - Updating list): https://wiki.archiveteam.org/?diff=50842&oldid=44352 15:35:06 Any plans for a channel for Taiwan, now that the situation with China is not looking that great. 15:50:41 Exorcism edited ZOWA (+31, /* Domains */): https://wiki.archiveteam.org/?diff=50843&oldid=50831 16:42:01 Hello! 16:42:01 I wrote some changes to the https://wiki.archiveteam.org/index.php/Coub article where I clarified about the current status why it survived. I added related reference source, and at the change notice, I added the explanation about the reason why Medium blog was abandoned after 2018. 17:42:25 Wohlstand: thanks! :) someone with moderation abilities will look though the queue soon 18:56:20 Exorcism edited ZOWA (+21): https://wiki.archiveteam.org/?diff=50844&oldid=50843 18:59:20 VoynichCr edited Glottolog (+68): https://wiki.archiveteam.org/?diff=50845&oldid=50787 19:01:20 VoynichCr edited Glottolog (+16, /* Archive */): https://wiki.archiveteam.org/?diff=50846&oldid=50845 19:16:22 VoynichCr created Wikistats (+195, Created page with "'''WikiStats''' is a site…): https://wiki.archiveteam.org/?title=Wikistats 19:20:23 VoynichCr edited Wikistats (+335): https://wiki.archiveteam.org/?diff=50848&oldid=50847 19:21:23 VoynichCr edited Wikistats (+31): https://wiki.archiveteam.org/?diff=50849&oldid=50848 19:21:24 VoynichCr edited Wikistats (-2): https://wiki.archiveteam.org/?diff=50850&oldid=50849 19:24:24 VoynichCr uploaded File:WikiStats 2023.png: https://wiki.archiveteam.org/?title=File%3AWikiStats%202023.png 19:24:25 VoynichCr edited Wikistats (+0): https://wiki.archiveteam.org/?diff=50852&oldid=50850 19:27:25 VoynichCr edited Template:Wikis (+31): https://wiki.archiveteam.org/?diff=50853&oldid=50536 19:46:28 VoynichCr edited Template:Rescued (+723, multiple dates with ','): https://wiki.archiveteam.org/?diff=50854&oldid=50802 19:52:29 VoynichCr edited Memory of Mankind (+15): https://wiki.archiveteam.org/?diff=50855&oldid=50823 19:53:46 #telegrab now seems to be running just as fast (in Mibps) as #shreddit was yesterday, probably clawing back any gains we get from pausing #shreddit 19:54:29 VoynichCr edited Template:Rescued (+41): https://wiki.archiveteam.org/?diff=50856&oldid=50854 19:54:30 VoynichCr edited Memory of Mankind (+131): https://wiki.archiveteam.org/?diff=50857&oldid=50855 19:54:31 VoynichCr edited Memory of Mankind (-131): https://wiki.archiveteam.org/?diff=50858&oldid=50857 19:58:30 Arcorann edited Miraheze (+147): https://wiki.archiveteam.org/?diff=50859&oldid=49991 19:58:31 Gabrinori edited FTP/List (+38, Add DataSUS (Brazil's NHS data division) public…): https://wiki.archiveteam.org/?diff=50861&oldid=46521 19:58:32 Wohlstand edited Coub (+262, The Coub has been transferred to a new team and…): https://wiki.archiveteam.org/?diff=50862&oldid=48730 19:58:33 VoynichCr moved Wikistats to WikiStats: https://wiki.archiveteam.org/?title=WikiStats 19:58:34 Nulldata edited Deathwatch (+164, /* Pining for the Fjords (Dying) */ added…): https://wiki.archiveteam.org/?diff=50865&oldid=50744 20:03:31 VoynichCr edited Arch Mission Foundation (+5): https://wiki.archiveteam.org/?diff=50866&oldid=50824 20:11:32 VoynichCr edited Template:Rescued (+28, and more): https://wiki.archiveteam.org/?diff=50867&oldid=50856 20:34:39 I think ARCHIVETEAM-YV-003100001-003199999/YV-003100001-003199999.tar is *truncated* too x_x 20:36:05 nicolas17: ? 20:36:43 rewby: we have been looking at the yahoo-videos stuff archived years ago, they were uploaded to IA as .tar or .tar.bz2 files 20:37:03 Ah 20:37:08 Interesting 20:37:18 Was worried I'd fucked something up 20:37:19 I was indexing them (first via curl | tar tv > list, later with something smarter) 20:37:26 it turns out there's 4 tarballs that actually contain data from friendster, not yahoo videos 20:37:49 I may have filed things under the wrong collection in the past before once 20:38:01 now I found one of those is also truncated 20:38:18 rewby: this is weirder because the filename has YV- 20:38:29 It's not werid 20:38:30 *weird 20:38:50 What likely happened is that if this was a dpos project, it was sent to the wrong rsync port 20:39:15 Although that file format is very old so I don't know 100% what tech was in use back then 20:39:25 if it was friendster-1234.tar item uploaded to the wrong collection it would make more sense to me 20:39:27 I wouldn't put it past a manual filing oopsie of something 20:39:55 nicolas17: This can easily still happen. If I fuck up the port number in the tracker stuff will end up in the wrong projects with the wrong names 20:40:11 also it's a 76GB .tar 20:40:36 Yeah, it's from before my time so I dunno how they used to do it. But I can see various ways this can happen 20:40:54 all of which appears to be friendster, not mixed files from friendster and yv 20:40:55 but yeah 20:41:02 Hm 20:41:03 Weird 20:41:12 another concern is where are the videos 003100001-003199999 20:41:13 I do know that back in the day targetry was fully manual 20:41:27 With a lot of things just ending up in directories that would occasionally be found years later 20:41:54 It's highly automated these days, but a decent chunk of that was my doing 20:41:56 the tar metadata has jscott/jscott as username/group so yeah it seems manual stuff was involved :P 20:42:28 The megawarc factory itself is from before my time 20:42:37 But even that is still sensitive to manual messups 20:42:47 So I automated a lot around it 20:43:44 my smol VPS is taking ages to "zstd -19" this file list with 3 million lines 22:51:27 https://archive.org/details/archiveteam-friendster-index 22:52:16 So there was apparently a tar containing file lists for all (presumably) of the friendster archives 22:53:15 I checked them and there are no flvs/mp4s and only one mention of yahoo video that is literally a blog post about it so I don't think any yv data was uploaded to the friendster collection 22:53:58 But there are some archives that were uploaded in later years that probably weren't in these lists 23:01:33 Tokyo Lab to close in November: https://old.reddit.com/r/Archiveteam/comments/16kfjb3/ "Sad news : japanese Tokyo Lab company, which is archiving many old animes since 1955, will close in November, and they'll have to destroy all the masters unclaimed by right holders."