the fuck

Made a table of Steam Workshops (the table starting out collapsed for obvious reasons)

nice

ye

>283 kilobytes

anon.cafe is shutting down on March 15

anon.cafe/meta/res/16466.html announcement

what is it?

It's an imageboard, part of a webring. Shutting down because of operating costs anon.cafe/meta/res/16467.html#16486

Oh sorry that's not the board owner, it's just speculation they don't know.

pokechu22: a jira jira.ecmwf.int

Pokechu22 edited Jira (+23, /* Not yet archived */ jira.ecmwf.int): wiki.archiveteam.org/?diff=51661&oldid=51655

thanks

I'm going to try to get something started on those soon

I'm pretty sure the database doesn't actually need to be saved to get attachments, as the same URL extraction issue that causes a bunch of junk relative URLs for attachments means that all attachments get logged... so that simplifies things a bit

JustAnotherArchivist edited Current Projects (+1, Fix date): wiki.archiveteam.org/?diff=51662&oldid=51658

FireonLive edited Current Projects (+16, move Blogger to long-term to reflect new…): wiki.archiveteam.org/?diff=51663&oldid=51662

Weaveworks is shutting down - linkedin.com/posts/richardsonalexis…e-activity-7160295096825860096-ZS67 news.ycombinator.com/item?id=39262650

Pokechu22 edited Jira (+50, /* Not yet archived */…): wiki.archiveteam.org/?diff=51664&oldid=51661

Pokechu22 edited Jira (+244, /* Strategy */ database isn't needed; link script): wiki.archiveteam.org/?diff=51665&oldid=51664

Pokechu22 edited Jira (+26, /* Not yet archived */ bugs.openjdk.org/): wiki.archiveteam.org/?diff=51666&oldid=51665

Exorcism edited Vbox7 (+64): wiki.archiveteam.org/?diff=51667&oldid=51648

Switchnode edited Deathwatch (+390, /* 2024 */ add world of tanks forums): wiki.archiveteam.org/?diff=51668&oldid=51649

Entartet edited Deathwatch (+231, Added thebillionscompanion.net.): wiki.archiveteam.org/?diff=51669&oldid=51668

Pokechu22 edited Games/Engines, Platforms and Hostings (+12, /* PC and Web */ [[Steam]]): wiki.archiveteam.org/?diff=51670&oldid=50184

Hmm, `(echo a; echo b; echo c) | zstdgrep -e 'a' -e 'b'` gives no output for me but `zstdgrep -e 'a'` does as does `zgrep -e 'a' -e 'b'` or `grep -e 'a' -e 'b'` - this also happened when I used zstdgrep on a .gz file. Is this a bug or have I misunderstood something about zstdgrep?

This is a bug.

zstdless has similar issues with option parsing: facebook/zstd #2880

Oof

Er, zstdless had*, although I haven't verified whether everything behaves correctly now.

I didn't even intend to type zstdgrep the first time, glad I noticed the missing output (I was verifying that extracting JIRA attachments from junk that gets logged in the meta-warc would work by comparing it with one where we extracted it from the DB)

Yeah, zstdgrep is fine for very simple cases, but if in doubt, it's better to use `zstdcat | grep ...` instead.

... ok, new problem, and this seems like it's not a grep one: from view-source:https://web.archive.org/web/20230929192111id_/https://bugs.mojang.com/browse/MC-180529 archivebot saw data-downloadurl="application/zip:Normal_Font_TT_v3.zip:https://bugs.mojang.com/secure/attachment/286387/Normal_Font_TT_v3.zip" and extracted

bugs.mojang.com/browse/application/…chment/286387/Normal_Font_TT_v3.zip but it *didn't* do anything with data-downloadurl="text/plain:hs_err_pid9900.log:https://bugs.mojang.com/secure/attachment/286386/hs_err_pid9900.log"

both bugs.mojang.com/secure/attachment/286386/hs_err_pid9900.log and bugs.mojang.com/secure/attachment/286387/Normal_Font_TT_v3.zip ended up in the database though

It doesn't seem to have extracted anything along the lines of browse/text.*\.log:

but it did accept bugs.mojang.com/browse/text/plain:c…cure/attachment/71965/crash.log.txt

hmm, it also didn't extract any .nbt or .dat files - does archivebot have a list of extensions it'll assume might be files when doing extraction from data attributes?

This would be on wpull, not AB.

And github.com/ArchiveTeam/wpull/blob/c…df5/wpull/scraper/util.py#L136-L217

That should pass `is_likely_link`.

Oh hmm, unless it's the `mimetype.guess` check.

`mimetype.guess_type` *

Yeah, it fails the `is_likely_link` check.

alright, I guess we do need the database after all :|

Yep, `mimetypes.guess_type` doesn't know about `.log`.

It wouldn't be in the DB either.

`mimetypes.guess_type('text/plain:hs_err_pid9900.log:https://bugs.mojang.com/secure/attachment/286386/hs_err_pid9900.log', strict=False)` → `(None, None)`

That wouldn't, but the correct URL (bugs.mojang.com/secure/attachment/286387/Normal_Font_TT_v3.zip or bugs.mojang.com/secure/attachment/286386/hs_err_pid9900.log) will be; they're just not saved due to the no-parent rule

Ah

this also means I need to find the database for hub.spigotmc.org which we ran a while back and saved the DB for, but I don't think I ever extracted outlinks from

I'll start !a < list jobs for several of the JIRA instances since we are running low on time, and then ping you for the DBs to be saved

fwiw on 3.11 `mimetypes.guess_type('text/plain:hs_err_pid9900.log:https://bugs.mojang.com/secure/attachment/286386/hs_err_pid9900.log', strict=False)` → `('text/plain', None)`

It probably still won't like .dat or .nbt though

indeed not

thuban: I'm still getting `(None, None)` on 3.11.

I think the `mimetypes` module does some discovery stuff in /usr/share or something like that.

So it can differ from system to system.

ah, so it does

Not /usr/share but same concept. :-)

you beat me to it, new github is awful v_v

It sure is, I do more and more stuff locally with a clone instead.

Especially since code search is loginwalled anyway.

anyway, perhaps the ab pipelines should be fitted with local mimetype files?

Perhaps wpull should ship its own list and init the `mimetypes` module with that.

ah! i didn't see that option. yes, that would simplify things

Apache's list doesn't even have .gz and .zst...

Looks like they're open to changes: apache/httpd #372

Pokechu22 edited Jira (+163, the database is still needed): wiki.archiveteam.org/?diff=51671&oldid=51666

Pokechu22 edited Jira (+0, update script): wiki.archiveteam.org/?diff=51672&oldid=51671