-
fireonlive
lol
-
h2ibot
Queuing bot shutting down.
-
h2ibot
Queuing bot started.
-
h2ibot
datechnoman: Restarting unfinished job isAJoDlg for '!a
transfer.archivete.am/gsq2b/unique_pdfs_output.txt'.
-
h2ibot
datechnoman: Restarting unfinished job cmKPUONH for '!a
transfer.archivete.am/HTbgw/filtered_.pdf_output.txt'.
-
h2ibot
datechnoman: Restarting unfinished job cYpn43AW for '!a
transfer.archivete.am/nIY9O/filtered_pdf_files_unique.txt'.
-
h2ibot
datechnoman: Restarting unfinished job YExqnYZo for '!a
transfer.archivete.am/y01za/filtered_pdf_files_unique.txt'.
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 494563 items. (YExqnYZo)
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 5516002 items. (cYpn43AW)
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 9401326 items. (cmKPUONH)
-
h2ibot
-
h2ibot
-
h2ibot
-
h2ibot
datechnoman: Deduplicating and queuing 9401326 items. (isAJoDlg)
-
arkiver
JAA: yeah something like this one is fine to filter, it's likely a loop very specific for this site alone
-
arkiver
filtering out and restarted!
-
thuban
arkiver: while we're talking about sources, did you ever put in those pubmed feeds? i know you were thinking about setting up a recrawl system for the delayed-open-access journals, but i see no reason that can't be added later
-
arkiver
thuban: no, those are not in at the moment
-
arkiver
we should first get through the current backlog
-
thuban
ah, k
-
datechnoman
Shouldnt take too long and we will be back to up-to-date :)
-
imer
we've been on like 3-7day eta for the past month it feels like haha
-
imer
arkiver: this one is still around
transfer.archivete.am/Jgl23/kep.adatbazisokonline.hu.log although low-ish volume
-
eggdrop
-
imer
still like 50/s extrapolated to the whole project
-
imer
up to 180/s now
-
nyany
I added more capacity from my side to assist with the destruction of that queue
-
arkiver
imer: it seems like legit images
-
arkiver
i will try to see where they come from
-
arkiver
downloading a log from AK to check
-
nyany
under 1 day to clear todo at current rate of speed
-
nyany
keep it up folks
-
nyany
never mind
-
imer
its just endless redirects though, not seeing a single 200 unless im missing something
-
xkey
ui, my runner passed 1.04TB in data scraped
-
xkey
btw, I asked the hoster; and so far no legal trouble with that machine
-
xkey
so sad
-
fireonlive
xkey: ill be sure to queue some illegal sites just for you
-
nyany
please don't lol
-
nyany
I'm in Hetzners' good graces, I don't need a reason to not be
-
AK
share.aktheknight.co.uk/riJe0/qAqOqIbi58.png/raw Question for arkiver, am I reading this right the theguardian.com is disabled because it's a dupe, but then we appear to be duplicated with a non root url of the domain? 🤔Or is it duplicated somewhere else?
-
fireonlive
nyany: ok i won't just for you :3
-
JAA
AK: Deduplication is across all files. It appears uncommented in 900_einpresswire_com.txt.
-
AK
🤦♂️Ahh amazing, thank you
-
nyany
We must've hit those MP4's again
-
nyany
-
Barto
nethogs mate :P
-
nyany
Barto: That's not even all my capacity lol
-
nyany
I have two servers that cannot work on URLs due to the previously discussed quad9 issue
-
Barto
yeah, ran #// at home, isp didnt like it
-
nyany
I could really piss my ISP off (I have a 3Gbps line) but I'd probably run into a threading issue before I saturate my line completely
-
nyany
and yes I've already tried
-
Barto
eh eh eh, i've almost fiber internet, except the damn last 10 meters are a pain in the butt because it's an old place and nobody knows what kind of zigzags was done for the current copper line.
-
Barto
and now way we're not going to punch holes in the neighbor's apartment to figure that out.
-
nyany
god bless last mile
-
nyany
FttN?
-
AK
My ISP had no issues with the traffic, they just don't like the abuse reports :(
-
Barto
it's already in the street, it's already in the house phone line entrance box, you've got to dispatch to the 4 apartments
-
nyany
My ISP doesn't know what an abuse report is tbh
-
nyany
They're not the brightest bunch of bulbs
-
Barto
AK: exactly, laws here (ianal) kind of ask them to 'act' even if it's bogus.
-
Barto
so yeah, block traffic and put a captcha in the router, and i dont know how far it can go, dont wanna know
-
xkey
fireonlive: lovely :}
-
AK
Partner and I both wfh, so any interruption to our internet would kinda be a problem unfortunately. Shame really as I was having fun downloading so much 😆
-
nyany
I put that to the test the other day actually
-
nyany
I spun up a few URLs workers on my best available laptop at home with an ethernet port while I was working
-
nyany
Spoiler: It did affect me slightly, but likely because I had too many threads
-
fireonlive
nyany: 3/3?
-
nyany
yea
-
fireonlive
:(
-
fireonlive
hate you
-
fireonlive
lol
-
nyany
as far as I know it's symmetrical
-
fireonlive
i got like
-
fireonlive
200/10? or something
-
nyany
ah the good old days
-
katia
me waits for 8/8
-
fireonlive
docsis pls
-
fireonlive
when can i get docsis 4 pls
-
fireonlive
sir