-
avoozl
euhm.. is there a simple command line tool to decompress these .warc.zst files with embedded dictionaries? I know I can read it in code, but I miss a simple cli tool to go through the content
-
derleth
hello.
-
Sanqui
hi
-
derleth
How do I propose a site to get saved?
-
Sanqui
you propose
-
Sanqui
go ahead and propose
-
derleth
OK, just a second.
-
derleth
-
derleth
Hoax-Slayer, a debunking website, is going away at the end of the month.
-
Sanqui
looks like a blog basically
-
Sanqui
we can handle it with archivebot
-
derleth
It's also this site,
hoax-slayer.com
-
derleth
I think that's where a lot of the debunkings still are.
-
Sanqui
ah ok, we can do both domains
-
derleth
Great!
-
Sanqui
it's in progress -- if you're interested you can watch it at
dashboard.at.ninjawedding.org/3 -- if not stand back and relax, it'll make its way into the wayback machine in a few days
-
derleth
Wonderful. That was a lot easier than i thought. Hooray for scripts.
-
derleth
Thank you.
-
Sanqui
thanks for bringing it to our attention!
-
yarrow
what's the difference between bs and ot?
-
OrIdow6
See topic
-
Sanqui
#archiveteam i announcements, -bs is on-topic discussion, -ot is off-topic discussion
-
Sanqui
I agree the nomenclature isn't exactly straightforward
-
lunik1
ddg sure does a lot of work to get search results that still aren't very good
-
LeighR
what's the correct channel for discussing grab-site?
-
EggplantN
probably here or -dev LeighR
-
LeighR
ok
-
Jake
-
billy549
mentioned it in #noanswers but it should also be mentioned here
-
arkiver
Jake: what a mess
-
Jake
yup. quite the mess.
-
thuban
JAA: thanks, that sure explains why turning it up didn't seem to make it go faster :)
-
thuban
fortunately the data's not gone yet and i think it'll finish today, or even this morning
-
jodizzle
serx: (from #archiveteam) I threw
meristation.as.com/zonaforo into AB to get something at least, but I agree that AB will not be able to get everything by the deadline.
-
jodizzle
This would need a different solution to get it all
-
JAA
Oh boy.
-
jodizzle
yep
-
JAA
Hold my beer.
-
LeighR
You do have a bunch of people who would love to be going fuller-throttle on Y!A...
-
HCross
if Yahoo would let us
-
HCross
JAA: please no
-
HCross
please no qwarc on y!a
-
LeighR
I mean, people who are probably willing to redirect that energy to whatever JAA comes up with
-
LeighR
for meristation
-
JAA
HCross: I'm looking into MeriStation, not Y!A.
-
HCross
ahh
-
LeighR
I am willing to look closely at logs and tell what I see
-
JAA
Have it working but seems very slow. Like 5+ second response time per page. I'll look a bit closer later.
-
Jake
-
nuroten
hi, I would like to suggest a podcast to consider for archival please, if this is the right channel for suggestions. it's called Hong Kong Connection, a weekly video documentary series by Radio Television Hong Kong (RTHK), a government news outlet.
podcast.rthk.hk/podcast/item.php?pid=280&lang=en-US its archival extends back to 2010, but
-
nuroten
there was an announcement today that the agency is moving to remove videos over 1 year old from their site, to align with their Youtube presence
-
nuroten
that podcast is one of several programmes on the site, but it is arguably one of the more iconic series with historical value, as it has footage of political protests from 2019, etc.
-
nuroten
-
nuroten
the show has also been subject to episodes being cut recently due to its coverage of issues perceived as sensitive by new leadership at the broadcaster
-
nuroten
-
nuroten
thanks for considering
-
thuban
sounds doable. i'm thinking download all video + upload to ia as one item for each language. opinions? do we want anything from the web presence? (there seems to be very little content other than the videos themselves--they don't, e.g., have individual descriptions.)
-
LeighR
thuban: the xml feeds seem to provide more details for each video than the main site itself
-
G4te_Keep3r
-
G4te_Keep3r
click description
-
thuban
there only seems to be a single description for the entire podcast
-
nuroten
thanks very much! yeah, the main content are the videos themselves, the site is mainly for format context (i.e. it once lived on this page and it had a dropdown menu to select year)
-
G4te_Keep3r
it was loading blank on 1 video and that on other, had only looked at 2 till your reply. Yea that makes it easier to just need video and title
-
LeighR
I'd say that right now, any old news programs in HK are in danger of disappearing
-
thuban
LeighR: thanks, for some reason ff's rss viewer doesn't seem to want to print the details
-
thuban
that said, a lot of them seem to be cut off; if there's an authoritative source they're being drawn from i'm not sure where it is
-
LeighR
in the English version, some of the details are stuck inside [CDATA] custom tags
-
nuroten
LeighR: indeed, especially given the new leadership and the recent report about the agency needing "reform"
-
G4te_Keep3r
wait, clicking "this episode" right under video does give diff descriptions like this short one. Are the descripts in the feed?
-
G4te_Keep3r
θΏε
-
G4te_Keep3r
After the Lockdown
-
G4te_Keep3r
2021-04-18
-
G4te_Keep3r
The government imposed lockdown in four streets in the Jordan district in late January and conducted mandatory Covid-19 tests on the residents. This episode records this unprecedented operation.
-
thuban
ooh
-
LeighR
yes - under itunes:subtitle and itunes:summary, stuck inside [CDATA ] tags
-
thuban
those are the cut-off descriptions in the feed
-
LeighR
thuban: you're right
-
G4te_Keep3r
so need web for full ones or they are all cut off?
-
thuban
need web
-
thuban
they seem to be in the source rather than xhr'd from an api, but that's ok
-
thuban
i can pull those
-
LeighR
is there anything I can do to help? stuff like trying to grab all of Y!A is fun, but this is actually important.
-
nuroten
the back episodes of Letter to Hong Kong might be worth saving:
podcast.rthk.hk/podcast/item.php?pid=162&lang=en-US it's an audio podcast of short audio clips, public figures (politicians from different parties, current affairs critics, academics, etc.) reading a letter addressed to the people. for example, this one is from a former
-
nuroten
pan-democratic legislator who is now in prison commenting on the events leading up to his trial:
podcast.rthk.hk/podcast/item.php?pi…=171170&year=2020&list=1&lang=en-US
-
nuroten
(background: he was convicted on charges along the lines of obstruction of justice for attempting to reason with a group that rushed into a subway station and started attacking commuters. he got injured while trying to help people on the train. the incident was all over the local news with questions over the official narrative of the incident, and
-
nuroten
he was later arrested.))
-
OrIdow6
arkiver: So I had an idea for how to save Aimix-BBS ("shut down" a few days ago, still up), based on the point that it blocks seemingly everyone but typically takes a few hundred requests to do it
-
OrIdow6
I thought it might work to have items be single URLs, and then have the warriors queue on-targeted extracted URLs to backfeed (there should be a few million at most, not a big site) instead of running them themselves
-
OrIdow6
This would basically turn it into a big, distributed ArchiveBot, where you have to make a project update instead of give a command in IRC to add or remove an ignore
-
thuban
nuroten: ok, i'm currently pulling thumbnails + videos + descriptions and formatting metadata in a way IA will understand
-
thuban
downloading will take a while, but everything will be set to upload once it's finished!
-
nuroten
thuban: thanks so much! :)
-
thuban
no problem!
-
thuban
someone remind me to update this every six months or something
-
thuban
i'll take a look at _letter to hong kong_ later
-
nuroten
if I had to choose only one thing to be saved from the RTHK website, that would be it (HK Connection). it's one of the most recognised local shows.
-
thuban
i was going to suggest earlier that if they also delete old material from their youtube--
youtube.com/channel/UC6of7UYhctnYmqABjUqzuxw--that might be worth looking into as well, but it seems like a _lot_ of content. did our news project ever extend to video?
-
nuroten
thanks! anything you can save from either would be great, the announcement seems to apply to anything in their archive older than 1 year, but if I had to guess, current affairs shows would be on the main chopping board
-
arkiver
OrIdow6: sounds good with me
-
arkiver
but
-
arkiver
JAA: didnt we have some setup with a ton of IPs that could be used for those purposes?
-
arkiver
OrIdow6: how permanent are the band