-
h2ibot
datechnoman: Deduplicated and queued 24992054 items. (Wx5KNGEC)
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 24990754 items. (gqnV5nbc)
-
datechnoman
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 13724758 items. (EOZxQYNK)
-
h2ibot
datechnoman: Deduplicated and queued 24990755 items. (gqnV5nbc)
-
datechnoman
!status
-
h2ibot
datechnoman: Jobs running: 1, jobs waiting for a slot: 0.
-
h2ibot
datechnoman: Deduplicated and queued 13724759 items. (EOZxQYNK)
-
datechnoman
-
h2ibot
-
TheTechRobo
-
h2ibot
-
h2ibot
TheTechRobo: Deduplicating and queuing 5489 items. (rjtwSINt)
-
h2ibot
TheTechRobo: Deduplicated and queued 5489 items. (rjtwSINt)
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 24994727 items. (BVq3IfHU)
-
datechnoman
-
h2ibot
-
h2ibot
datechnoman: Deduplicated and queued 24994730 items. (BVq3IfHU)
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 24994891 items. (dBUwCpLn)
-
h2ibot
datechnoman: Deduplicated and queued 24994892 items. (dBUwCpLn)
-
datechnoman
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 14343597 items. (rMLCrn7k)
-
h2ibot
datechnoman: Deduplicated and queued 14343600 items. (rMLCrn7k)
-
pabs
nyuuzyou: can you add that to Deathwatch? also ensure you add a reference link
wiki.archiveteam.org/index.php/Deathwatch
-
knecht4
JAA: hey, just checking in if you found the time to add xrel to the list of scheduled captures.
-
knecht4
i'm not sure if the AT captures arrive in batches in the wayback machine
-
imer
transfer.archivete.am/PJutJ/flagella.crbs.ucsd.edu.log this one has been around for a while,
flagella.crbs.ucsd.edu/images?image…_parms[cellular_component]=polytene chromosome&advanced_search=advanced+search&per_page=10&page=3&per_page=10&page=3[...] - probably filter out more than one (per_)page entry?
-
eggdrop
-
JAA
knecht4: Not yet, no. If you could send a PR to
github.com/ArchiveTeam/urls-sources , that'd be great. You can model it after 60_tech_link_forums.txt, I think.
-
JAA
The first number is the request interval in seconds.
-
knecht4
I'll look into it. As a separate file or into 60_tech_link_forums.txt?
-
katia
-
knecht4
nice find. would it be beneficial to use this instead of the home page?
-
thuban
imo yes, would reduce noise
-
knecht4
ah yeah these also exist for board and reviews
-
knecht4
yeah nice, could grab all three and cut out the fluff
-
katia
this is just an iframe on the main page afaict
-
katia
so not sure it'd even get grabbed if you just gave the homepage to urls
-
knecht4
yeah and it still contains links to the actual comment pages
-
knecht4
redirects are followed? because the links are shortened
-
katia
yeah redirects should be followed with a depth of 1 i think, but i'm not sure
-
knecht4
alright, thank you! PR is incoming
-
JAA
Separate file, I think.
-
katia
the hacker news file was renamed to tech forums
-
JAA
Let's add it and check in a few days what ends up in the WBM. :-)
-
JAA
Yeah, but xREL isn't a tech forum.
-
knecht4
i was thinking to add a generic file named "warez_related" or whatever but now its just "60_xrel.txt"
-
katia
right :p
-
knecht4
i guess it could be renamed in the future if fitting stuff gets added.
-
JAA
Yep, 60_xrel.txt sounds good.
-
knecht4
-
JAA
arkiver: ^
-
imer
-
katia
does h2ibot take .zst?
-
JAA
No, but transfer can auto-decompress when you remove .zst.
-
katia
TIL!
-
JAA
Originally added for socialbot, but turned out to be useful for qubert, too.