we're very much looking to be killing that site arkiver if you can have a look ^

imer: around now, checking...

ah right interesting one

people hiding their timestamp everywhere nowadays

nstrom|m: nyany: imer: that one is handled now too

few days left at this speed! :)

running very smooth

6.8 million mp4s in the stash!

arkiver and you said there werent many :P

We are smashing through!

datechnoman: yeah, there's many actually

That will keep us busy. I'm keen to get the pdfs loaded in slowly that I scrapped a few weeks back that we were talking about

Will be pushing a small commit on github for the medical research sites to be regularly scanned also :D

arkiver, is there still any interest in OSM links? i saw there was some discussion from pabs earlier this year, i have 2 million links exported from the latest planet.osm (transfer.archivete.am/AifZw/2024-04-29.zst) via transfer.archivete.am/inline/AXj68/2024-04-29.txt - they need to be filtered

inline (for browser viewing): transfer.archivete.am/inline/AifZw/2024-04-29.zst)

I threw the OSM links into the bot around the time of that discussion

ah i see, i was looking for them in urls-sources

btw, the OSM links have a lot of broken stuff in them. things like example.com/https://example.com

yeah

or file://...

or random strings

also their taginfo API is useful if you don't want the whole planet.osm download taginfo.openstreetmap.org/taginfo/apidoc#api_4_search_by_value

i figured it'd be nicer for them if i downloaded it and parsed it locally

also I think when I submitted here I filtered out things like facebook and youtube, guess those don't work well

If you look at grafana, disregard the 0 reading values... My machine's disk went into read only mode

Between 29 Apr 2024 1350 and 1500 UTC

good job kiska