01:59:24 Atom package pages are done except for a handful of seemingly broken packages (over 200 requests all failed). No images to be retrieved there since they're all already broken. 01:59:40 This site has clearly been under- or unmaintained for a while. 05:45:44 JustAnotherArchivist edited Deathwatch (+222, /* 2023 */ Add Revue): https://wiki.archiveteam.org/?diff=49242&oldid=49223 05:52:57 Revue has an API which might help with discovery, but it requires auth, and registration is no longer possible. 05:53:46 Docs for it are here: https://www.getrevue.co/api 06:16:21 Amazing, they've removed everything on the support site except the shutdown notice 06:16:56 Not exactly, looks like the articles are still there, they're just not listed anywhere anymore. 06:17:10 But yeah... 06:18:17 Oh 06:34:13 May have found an enumeration method 06:37:14 But I'm not sure where it's used in the live site 06:47:10 You can get rid of the slug in the issue URLs with https://www.getrevue.co/profile/00/archive/384543 06:47:39 Still requires the username though. 06:47:55 Ironically they made the announcement that Revue is shutting down less than a day after Jack Dorsey started using the service 06:50:49 Heh 06:51:05 Although apparently the news about it getting shut down have been known for a bit already: https://www.platformer.news/p/inside-twitters-product-roadmap-under?post_id=82372028 07:10:58 JustAnotherArchivist edited The WARC Ecosystem (+2, /* Information */ Fix link): https://wiki.archiveteam.org/?diff=49243&oldid=46405 07:16:59 JustAnotherArchivist edited Template:Url (+80, Add category to pages with broken URLs (usually…): https://wiki.archiveteam.org/?diff=49244&oldid=45771 07:19:59 JustAnotherArchivist edited Reddit (+4, Fix broken URLs): https://wiki.archiveteam.org/?diff=49245&oldid=48498 07:20:00 JustAnotherArchivist edited Sketch (+2, Fix broken URL): https://wiki.archiveteam.org/?diff=49246&oldid=47562 07:20:59 JustAnotherArchivist edited List of lost online videos/list (+2, Fix broken URL): https://wiki.archiveteam.org/?diff=49247&oldid=48234 07:26:45 Revue has a link shortener at rev.vu, but that seems to be used for outlinks from newsletters. 07:27:11 Also, not all things are under the getrevue.co domain: https://newsletter.mgsiegler.com/ 08:26:55 Hi 08:32:45 I was wondering about whether FOS uses rsync over ssh or normal rsync? 08:33:21 Why? 08:35:52 JAA: only a question... 08:36:44 Well, FOS isn't even used anymore here. So, neither? 08:38:44 from my experience, it uses verification without password... 08:39:18 Are you uploading data there? 08:41:03 for example, I tried yesterday to pick any nickname from the pipeline list (only to try), and it uploaded my test file (sorry) to user's rsync files even without asking for a password... 08:41:32 JAA: no, I only uploaded one file to test 08:41:36 So you did. 08:41:39 Stop it. 08:41:59 I told you this weeks ago when you tried to mess with ArchiveBot 'just to test'. 08:42:08 Stop testing shit against our prod systems. 08:42:28 SketchCow: ^ Someone uploaded random shit to FOS. 08:42:29 sorry, the name of file is (sorry) 08:46:09 JAA: Why not use SSH for authentication? I mean, as you can see, anyone can upload shit to your servers... 08:46:47 SketchCow: I'll send you an email with details instead. 08:55:27 JAA: the nickname of pipeline is (jap-kakapo-e), and sorry again. 08:56:29 lol 08:56:36 At least come up with your own names. :-) 08:56:54 That part doesn't matter, you can't connect to the AB control node anyway (fortunately). 09:04:00 you mean that FOS is using a single rsync secret file with nicknames for authentication, and not every pipeline has a separate user account on FOS with its own rsync..? 09:06:17 FOS dates back to the dark ages, and its configuration is ... suboptimal. As I said, we don't use it anymore. 09:06:49 We should've just deactivated the target when we moved away from it, but that never happened for $reasons. 09:14:19 That's been fixed now. 09:14:34 And now stop messing with our systems. 09:16:30 surprise pentest 10:16:44 good to know... but, I am still can't understand how are you running a rsync server without password or even ssh? 10:39:34 rsync has a daemon mode that can just accept connections unauthenticated 10:40:09 this is how archiveteam warriors submit their jobs to the targets 10:47:17 monika: but why, when I try to connect to FOS without a username, it returns "Connection refused"? 10:47:43 i don't know, maybe you shouldn't be poking around FOS??? 15:48:07 https://aegirproject.org/ is likely going to EOL and completely shut down, unsure what will happen with the assets 15:48:12 https://tenyears.aegirproject.org/ 15:49:35 The repo and tenyears sites could be moved to gitlab pages to keep them 15:49:36 online as an archive. 15:49:54 https://debian.aegirproject.org/ is the debian repository 15:50:14 the rest is likely part of the normal drupal.org infrastructure and unlikely threatened 20:16:51 Retrofan: It refuses the connection now because the issue has been fixed by taking rsync down. Which we should've done a long while ago. 20:18:54 anarcat: https://aegirproject.org/ returns an Apache page for me now (with an invalid certificate, too). :-| 20:19:13 that's a great start 20:19:55 Ah, broken server config, HTTP redirects to the www subdomain, which works. 20:20:54 good catch 20:25:02 found clara.io on the deathwatch list, says it will shutdown at 2022-12-31 and it also seems to be not yet saved. has there been progress on it? 20:27:30 anarcat: Everything I could find is in AB now. 20:28:13 amazing, thanks 21:03:46 jacksonchen666: A bit on it from me, hope to have something written by Monday 21:37:13 I'm trying to get all the metadata of the games on the wii shop channel. I'm seeing a section of the wiki page that talks about that but I can't get either of them to work. Everything downloaded with the first method is a 404 and I have no idea how I'm supposed to do the second one. Any tips? 21:39:24 Well yeah, those URLs are dead now since the site was shut down. The data is linked in the box on the wiki page, but I don't know anything about the details. 21:41:13 I'm pretty sure the metadata still exists somewhere. You can still access the website through the method explained in the wiki page, and you can still download games you already own. 21:44:54 CompArison: I thought the Wii Shop is still down 21:46:23 Nevermind, it went back up awhile ago. Missed that. 21:47:17 Yeah. Maybe it has something to do with the certificate it had me install? the command it had me paste doesn't have anything like that so maybe that's why its only seeing 404s 21:48:09 Larsenv: ^ 21:48:46 i think i lost internet a bit ago so now i got a 9 for some reason 21:49:11 Do you mean the Commands section on https://wiki.archiveteam.org/index.php/Wii_Shop_Channel ? 21:49:21 yeah 22:04:56 JAA: anyways... I recommend using ssh for authentication on whatever rsync server you are using for uploading warcs. 22:06:01 Retrofan: If you understood how we use rsync targets, you'd know why that doesn't work. 22:07:00 you mean using SSH for authentication? 22:07:16 But yes, for AB specifically, this has been known for a while. One of the reasons why we no longer use FOS, albeit only a minor one. 22:07:21 Yes 22:09:51 so... you see that it is impossible to use rsync over ssh to transfer finished warcs? 22:11:43 If you control both ends, you absolutely can. 22:12:42 that will be a good point for my archive system 22:26:28 The Atom packages API was shut down sometime between 15:30 and 16:20 today, redirecting to broken github.blog URLs instead. By now, it presents an invalid certificate and an HTTP 400. Great engineering, much wow. 22:27:32 More accurately, I saw the first such redirects at 15:30 and the last successful requests at 16:24. 22:31:04 The trigram bruteforce found 266k packages or so. 10k I had already retrieved before and were skipped. No clue currently how many of the others were retrieved successfully. The vast, vast majority of them were spam anyway though. As you can tell from the size, too: the oldest 500 pages of packages produced 35 GiB of WARCs, this run only 800 MiB. 22:46:06 It looks like running a public archiving system is so hard... got over 100k attempts to login to root in the last hour, and my main archivebot VM had been hacked a few days ago, and the hacker added his ssh key to authorized_keys. fortunately, I realised that early and deleted his entry and secured the VM's root ssh login, before he did anything. 22:58:41 CompArison, JAA, TheTechRobo, I've done that before 22:58:58 fun fact, you can grab metadata even using the dsi shop 22:59:03 you can rquery stuff with the switch endpoint 22:59:15 https://archive.org/details/WiiShopChannelBackup 22:59:28 https://archive.org/details/NintendoDSiShopBackup 22:59:37 I think they're in here? 22:59:49 the hard drive containing my work stopped functioning 22:59:51 it crashed...