-
datechnoman
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 5516002 items. (cYpn43AW)
-
datechnoman
arkiver could I hassle you to look at the bot? Doesn't looks like its queuing the jobs into the tracker. Noticed on my last 2 jobs that there was no increase in the todo queue and no "deduplicated and queued XXXXX items
-
datechnoman
The first job is a duplicate upload so it won't produce any new items (didn't notice at the time). The second job isn't and I'd expect a lot of new items from it. Threw in a bunch of new workers to process through it so don't worry about that :)
-
datechnoman
Uploaded to other channels/bots without issue so might be a character in one of the files causing issues or the bot not being happy :(
-
datechnoman
Also if you dont mind topping up the queue with some stash urls that would be great. Going away for the next 3 days and will leave the extra workers running so please feel free to keep them busy :)
-
arkiver
datechnoman: i will put some extra in tonight
-
arkiver
might move that out when i make updates to the code again
-
arkiver
which well get any outlinks (so to a different domain) from web pages on sites that we have identified as news sites
-
arkiver
... which could go a little deep as we would also queue outlinks from one news site to another, from where more outlinks are queued, etc.
-
arkiver
so that might explode and then i'll make changes to only get outlinks to non-news sites
-
arkiver
but it may also not explode, and then we'll get significantly better news site coverage and news site sources coverage
-
datechnoman
All good. I shall leave it in your capable hands :)
-
datechnoman
archiving news is very important and historical so well worth the compute/resources
-
datechnoman
It will most likely be a large expansion of urls being queued but we will cut down on them as we process through them like the monthly sitemaps
-
datechnoman
Sounds well worth it
-
arkiver
absolutely!
-
arkiver
regularly i come across some government pdf document on the Internet, and wonder "is it saved?". so i look it up in the wayback machine, and really in 8/10 cases it is saved, and the only capture is this project
-
datechnoman
Honestly doing some really great work here 👏
-
datechnoman
I did some really rough testing of the unique pdfs in the second job and found a bunch missing so hoepfully that helps! They aren't targeted for things like news and government. Just pdf outlinks from Twitter posts
-
h2ibot
Queuing bot shutting down.
-
h2ibot
Queuing bot started.
-
h2ibot
datechnoman: Restarting unfinished job isAJoDlg for '!a
transfer.archivete.am/gsq2b/unique_pdfs_output.txt'.
-
h2ibot
datechnoman: Restarting unfinished job cmKPUONH for '!a
transfer.archivete.am/HTbgw/filtered_.pdf_output.txt'.
-
h2ibot
datechnoman: Restarting unfinished job cYpn43AW for '!a
transfer.archivete.am/nIY9O/filtered_pdf_files_unique.txt'.
-
arkiver
indeed datechnoman ^
-
arkiver
it may be a characters yeah, i'm not immediately sure
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 5516002 items. (cYpn43AW)
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 9401326 items. (isAJoDlg)
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 9401326 items. (cmKPUONH)
-
JAA
Archiving news sources would be great!
-
JAA
(It'd work even better if news outlets actually linked their sources more frequently.)
-
JAA
We still haven't found that IS Telegram channel, have we?
-
Ryz
Hmm, came across
eurosport.com/football/premier-leag…opped-do-you_sto5859298/story.shtml - but it's geoblocked; does this project take account of prioritizing geoblocked content to other Warriors that may have access to it?
-
Ryz
Seeing that
eurosport.com is constantly processed here
-
Vokun
Woah, that's a very long URLs:
-
JAA
There is no prioritisation. Stuff just gets retried randomly by any worker. Hopefully one of them isn't getting geoblocked.
-
JAA
Eurosport should be geoblocked to ~Europe, and we have lots of workers there, so unless they block Hetzner, that particular case should mostly be fine, probably maybe.
-
datechnoman
Somehow I'm always finding the bugs in the system :P
-
datechnoman
-
h2ibot
-
h2ibot
datechnoman: Your job is waiting for a slot. (dSoiEwEl)