on that note, what about world of tanks?

Manu edited Mailman/2 (+50, /* calypso.tux.org/pipermail lost */): wiki.archiveteam.org/?diff=52173&oldid=52170

thuban: should I repost those messages here?

tourist: for the benefit of log-readers, sure

[reposting from #archiveteam]

Hi, just want to discuss before editing Deathwatch because it's a bit vague:

booru.org is a site which allowed people to host their own tag-based 'booru' imageboards. Some are basically archives themselves for fandoms or special interests.

There are about 3000 boorus hosted, eighty have over 10,000 images, ten have over 100,000 images and two exceptional boorus have 1.5 and 1.7 million images respectively.

I propose it be placed on deathwatch due to this post a couple of weeks ago from the site admin: forum.booru.org/viewtopic.php?t=14193

>The project is closed and winding down, resources for search functionality etc. at peak times will get tapped out.

Does it seem like this site would require a dedicated project or should it be added to the Deathwatch page as normal, with 'Unknown' date.

[/end repost]

tourist: in theory sites with troubling vital signs but no clear shutdown announcement should go on 'fire drill' rather than deathwatch, but that page is a bit of a mess and i've been meaning to clean it up for some time, so i think deathwatch is ok for now

and we do add sites to deathwatch even if they get their own wiki pages/dedicated projects (although my guess is that it won't be necessary in this case)

Alright, I'll add it to the list now. Thanks :)

tourist: you're welcome! do you know whether there's a way to get a list of all the boorus, and/or whether booru creation/activity has been disabled?

Booru creation is closed. Boorus are still active.

List of boorus can be found at booru.org/top but you can only grab up to 200 per page.

that's fine if it's a complete list; let's see

yep, looks like it

thanks!

should we do something for opensubtitles.org ? they have been restricting access greatly lately

Ryz: no updates, and no reply from them

perhaps at this point best is just gathering lists of abload.de URLs and pushing them through AB if it's not an extreme number of URLs

huh is opensubtitles.org completely behind login now?

what... looks like it, same for others?

:(

arkiver: no, not for me

thuban: do you have a example of a subtitle URL?

that is not behind a login for you

sure: opensubtitles.org/en/subtitleserve/sub/10010159 / opensubtitles.com/en/subtitles/legacy/10010159/download ('beta')

sends me to a login form

let me VPN this

thuban: hmm from a different location i get no login scen

screen*

i feel like opensubtitles.org is becoming more shitty fast though

arkiver: X-Forwarded-For trick work?

thuban: no i think

arkiver: yes, very much becoming more shitty fast

i think at one point (some months ago) it also tried to make me login but doesn't do it now

i think we'll launch a project for them

better archive it before its too late

now they apparently have a forced login for some IPs

---

WHAT

---

- --

what magic is this

sourcery

strip trailing '--', trim remainder of message

-- -

no pre-decrement

anyway

so something i came across

this one is apparently going away in June bedfordregiment.org.uk - clearly some simple site made by probably a single enthusiastic person

(i put it in AB)

they have a list of sources/links to similar sites at bedfordregiment.org.uk/links.html

near the bottom of the page they have

> A Northamptonshire family history site worth knowing, which carries a wide array of [...]

with a link to familyhistorynorthants.co.uk , which is a blog about gambling.

but looking the front page up in the wayback machine, one finds a beautiful simple little site rich with information... gone and taken over by some gambling/scam business recently

i wonder if we can find these sites easily somehow and get them all archived, it's sad to see how some of these end up. i bet many of these are maintained by old enthusiastic people, who may pass away in the coming years, after which their sites go down and tons of information gets lost

arkiver: i was just thinking along the same lines (i love sites like this and run them trough ab whenever i come across them)

thuban: yeah!

marginalia.nu is not a bad source for these; i think the index is available somewhere

we should get them all

perhaps we can also contact several of these sites and let them know that we can archive these types of sites

perhaps they could spread the word, and people behind these sites could submit lists of sites like these that they know about

there's maybe forums with enthusiasts around these kind of subjects?

thuban: did you ever contact marginalia? maybe we should contact them?

arkiver: downloads.marginalia.nu/exports ! i think 'domains' is what we would want?

or 'urls', depending on how we handle it

^ *through

thuban: love it, yeah! i don't know much about marginalia, do they only collect these type of little home made sites?

i don't know that much more, but there's a fair amount of writing about the project(s) and philosophy on the site

see also marginalia.nu/marginalia-search/about/#similar-projects

thuban: i love it

just looking at search.marginalia.nu

i need to #Y up and running really

so we can get all these domains

Manu edited Mailman/2 (+46, /* datacast.hu/mailman/listinfo saved */): wiki.archiveteam.org/?diff=52174&oldid=52173

bedfordregiment.org.uk is in the list!

familyhistorynorthants.co.uk too

yeah we need to get this archive, amazing!

arkiver: are you looking for a crawled index of individual pages, or seed URLs?

c3manu: any

this is where people can submit urls :)

..or what people submitted

lovely!

yeah we should get that too

feel free to extend the wiki page ;)

i also think webrings would be good for indices. in the indie corners of the internet those are getting popular again

just look at this: webring.xxiivv.com

in theory yes; in practice it might be difficult to identify webring to/from links (since they can be formatted arbitrarily)

ah, a central index :)

yeah, that's definitely not going to be fun ^^

perhaps it's more something for marginalia to find these sites through those ^ and list them online?

i will send marginalia.nu an email about this awesomeness

do we have a pipeline on AB that can handle a 180 GB file?

i want to throw downloads.marginalia.nu into it

^^ sounds good, i'm not sure whether the index is really curated or the search engine is doing the heavy lifting

i approve re awesomeness email :)

thuban: i guess they do some checks on the website front page to see if it is "old style" and include it only if it is

I feel like this is going to be a problem server8.kiska.pw/uploads/0bc070b5366d602c/image.png

Limited to 10 per day...

kiska: yeah it would be a very long term effort

arkiver: depends on how it's done

if it's ip based, sure we're metaphorically screwed

but if it's SESSION based... (JAA's favorite)

i.e. store session with 24h expiry as cookie object, thus enabling easy bypassing if one were to simply ignore cookies

arkiver: Space wise the new pipelines like firepipe should fit 100+ gig files easily

that_lurker: thanks! i put it on firepipe-f

Interesting

Never heard of marginalia before now

nyany: looks to be IP based

does look like www.opensubtitles.org supports ipv6 though

That 10/day limit only appears on their new "beta" site for me, I can still download as much as I want through the regular/older one

Yet another reason to archive it before they change that I guess

My friend has a file analysis site and would like all his current reports archived, I have a list of almost 400k links and was wondering if someone could start the archivebot archive of them please - transfer.archivete.am/ffsKw/neiki%20analytics%20links.txt

inline (for browser viewing): transfer.archivete.am/inline/ffsKw/neiki%20analytics%20links.txt

sites currently running cloudflare with the "essentially off" setting enabled, but I can get them to disable cloudflare completely if needed, but don't think it will be needed

JaffaCakes118: your friend's reports depend on javascript-initiated requests; archivebot will be useless

i suppose it might work if we generated the corresponding api url for every page

thuban the links are able to be archived perfectly through the save now page

is it not the same for archivebot?

save page now is not archivebot

no

save page now runs a browser, archivebot doesn't

ah ok

is there any way we can still archive it? My friend of course will be willing to make changes

but seems we'd just need api.neiki.dev/analyze/reports?sha256=...

yeah he said save the api instead

and it will return the data of it

well, alongside

I will get a list of links now for the api.neiki.dev

no need

oh ok