-
JAA
Atom package pages are done except for a handful of seemingly broken packages (over 200 requests all failed). No images to be retrieved there since they're all already broken.
-
JAA
This site has clearly been under- or unmaintained for a while.
-
h2ibot
JustAnotherArchivist edited Deathwatch (+222, /* 2023 */ Add Revue):
wiki.archiveteam.org/?diff=49242&oldid=49223
-
JAA
Revue has an API which might help with discovery, but it requires auth, and registration is no longer possible.
-
JAA
Docs for it are here:
getrevue.co/api
-
OrIdow6
Amazing, they've removed everything on the support site except the shutdown notice
-
JAA
Not exactly, looks like the articles are still there, they're just not listed anywhere anymore.
-
JAA
But yeah...
-
OrIdow6
Oh
-
OrIdow6
May have found an enumeration method
-
OrIdow6
But I'm not sure where it's used in the live site
-
JAA
You can get rid of the slug in the issue URLs with
getrevue.co/profile/00/archive/384543
-
JAA
Still requires the username though.
-
tech234a
Ironically they made the announcement that Revue is shutting down less than a day after Jack Dorsey started using the service
-
JAA
Heh
-
JAA
Although apparently the news about it getting shut down have been known for a bit already:
platformer.news/p/inside-twitters-p…duct-roadmap-under?post_id=82372028
-
h2ibot
JustAnotherArchivist edited The WARC Ecosystem (+2, /* Information */ Fix link):
wiki.archiveteam.org/?diff=49243&oldid=46405
-
h2ibot
JustAnotherArchivist edited Template:Url (+80, Add category to pages with broken URLs (usually…):
wiki.archiveteam.org/?diff=49244&oldid=45771
-
h2ibot
JustAnotherArchivist edited Reddit (+4, Fix broken URLs):
wiki.archiveteam.org/?diff=49245&oldid=48498
-
h2ibot
JustAnotherArchivist edited Sketch (+2, Fix broken URL):
wiki.archiveteam.org/?diff=49246&oldid=47562
-
h2ibot
JustAnotherArchivist edited List of lost online videos/list (+2, Fix broken URL):
wiki.archiveteam.org/?diff=49247&oldid=48234
-
JAA
Revue has a link shortener at rev.vu, but that seems to be used for outlinks from newsletters.
-
JAA
Also, not all things are under the getrevue.co domain:
newsletter.mgsiegler.com
-
Retrofan
Hi
-
Retrofan
I was wondering about whether FOS uses rsync over ssh or normal rsync?
-
JAA
Why?
-
Retrofan
JAA: only a question...
-
JAA
Well, FOS isn't even used anymore here. So, neither?
-
Retrofan
from my experience, it uses verification without password...
-
JAA
Are you uploading data there?
-
Retrofan
for example, I tried yesterday to pick any nickname from the pipeline list (only to try), and it uploaded my test file (sorry) to user's rsync files even without asking for a password...
-
Retrofan
JAA: no, I only uploaded one file to test
-
JAA
So you did.
-
JAA
Stop it.
-
JAA
I told you this weeks ago when you tried to mess with ArchiveBot 'just to test'.
-
JAA
Stop testing shit against our prod systems.
-
JAA
SketchCow: ^ Someone uploaded random shit to FOS.
-
Retrofan
sorry, the name of file is (sorry)
-
Retrofan
JAA: Why not use SSH for authentication? I mean, as you can see, anyone can upload shit to your servers...
-
JAA
SketchCow: I'll send you an email with details instead.
-
Retrofan
JAA: the nickname of pipeline is (jap-kakapo-e), and sorry again.
-
JAA
lol
-
JAA
At least come up with your own names. :-)
-
JAA
That part doesn't matter, you can't connect to the AB control node anyway (fortunately).
-
Retrofan
you mean that FOS is using a single rsync secret file with nicknames for authentication, and not every pipeline has a separate user account on FOS with its own rsync..?
-
JAA
FOS dates back to the dark ages, and its configuration is ... suboptimal. As I said, we don't use it anymore.
-
JAA
We should've just deactivated the target when we moved away from it, but that never happened for $reasons.
-
JAA
That's been fixed now.
-
JAA
And now stop messing with our systems.
-
monika
surprise pentest
-
Retrofan
good to know... but, I am still can't understand how are you running a rsync server without password or even ssh?
-
monika
rsync has a daemon mode that can just accept connections unauthenticated
-
monika
this is how archiveteam warriors submit their jobs to the targets
-
Retrofan
monika: but why, when I try to connect to FOS without a username, it returns "Connection refused"?
-
monika
i don't know, maybe you shouldn't be poking around FOS???
-
anarcat
aegirproject.org is likely going to EOL and completely shut down, unsure what will happen with the assets
-
anarcat
-
anarcat
The repo and tenyears sites could be moved to gitlab pages to keep them
-
anarcat
online as an archive.
-
anarcat
debian.aegirproject.org is the debian repository
-
anarcat
the rest is likely part of the normal drupal.org infrastructure and unlikely threatened
-
JAA
Retrofan: It refuses the connection now because the issue has been fixed by taking rsync down. Which we should've done a long while ago.
-
JAA
anarcat:
aegirproject.org returns an Apache page for me now (with an invalid certificate, too). :-|
-
anarcat
that's a great start
-
JAA
Ah, broken server config, HTTP redirects to the www subdomain, which works.
-
anarcat
good catch
-
jacksonchen666
found clara.io on the deathwatch list, says it will shutdown at 2022-12-31 and it also seems to be not yet saved. has there been progress on it?
-
JAA
anarcat: Everything I could find is in AB now.
-
anarcat
amazing, thanks
-
OrIdow6
jacksonchen666: A bit on it from me, hope to have something written by Monday
-
CompArison
I'm trying to get all the metadata of the games on the wii shop channel. I'm seeing a section of the wiki page that talks about that but I can't get either of them to work. Everything downloaded with the first method is a 404 and I have no idea how I'm supposed to do the second one. Any tips?
-
JAA
Well yeah, those URLs are dead now since the site was shut down. The data is linked in the box on the wiki page, but I don't know anything about the details.
-
CompArison
I'm pretty sure the metadata still exists somewhere. You can still access the website through the method explained in the wiki page, and you can still download games you already own.
-
TheTechRobo
CompArison: I thought the Wii Shop is still down
-
TheTechRobo
Nevermind, it went back up awhile ago. Missed that.
-
CompArison9
Yeah. Maybe it has something to do with the certificate it had me install? the command it had me paste doesn't have anything like that so maybe that's why its only seeing 404s
-
JAA
Larsenv: ^
-
CompArison9
i think i lost internet a bit ago so now i got a 9 for some reason
-
TheTechRobo
-
CompArison9
yeah
-
Retrofan
JAA: anyways... I recommend using ssh for authentication on whatever rsync server you are using for uploading warcs.
-
JAA
Retrofan: If you understood how we use rsync targets, you'd know why that doesn't work.
-
Retrofan
you mean using SSH for authentication?
-
JAA
But yes, for AB specifically, this has been known for a while. One of the reasons why we no longer use FOS, albeit only a minor one.
-
JAA
Yes
-
Retrofan
so... you see that it is impossible to use rsync over ssh to transfer finished warcs?
-
JAA
If you control both ends, you absolutely can.
-
Retrofan
that will be a good point for my archive system
-
JAA
The Atom packages API was shut down sometime between 15:30 and 16:20 today, redirecting to broken github.blog URLs instead. By now, it presents an invalid certificate and an HTTP 400. Great engineering, much wow.
-
JAA
More accurately, I saw the first such redirects at 15:30 and the last successful requests at 16:24.
-
JAA
The trigram bruteforce found 266k packages or so. 10k I had already retrieved before and were skipped. No clue currently how many of the others were retrieved successfully. The vast, vast majority of them were spam anyway though. As you can tell from the size, too: the oldest 500 pages of packages produced 35 GiB of WARCs, this run only 800 MiB.
-
Retrofan
It looks like running a public archiving system is so hard... got over 100k attempts to login to root in the last hour, and my main archivebot VM had been hacked a few days ago, and the hacker added his ssh key to authorized_keys. fortunately, I realised that early and deleted his entry and secured the VM's root ssh login, before he did anything.
-
Larsenv
CompArison, JAA, TheTechRobo, I've done that before
-
Larsenv
fun fact, you can grab metadata even using the dsi shop
-
Larsenv
you can rquery stuff with the switch endpoint
-
Larsenv
-
Larsenv
-
Larsenv
I think they're in here?
-
Larsenv
the hard drive containing my work stopped functioning
-
Larsenv
it crashed...