-
JAA
Yay :-)
-
arkiver
moving queues
-
imer
still seeing outdated project code on my end
-
arkiver
yeah, i paused it again while queues are being moved around
-
arkiver
i should do that in the proper way (which requires a few extra steps)
-
imer
ah, right
-
arkiver
imer: fixed :)
-
arkiver
now properly paused
-
imer
thanks, probably wasn't necessary though. moving the items shouldn't take too long, right?
-
JAA
<anakin_padme.png>
-
arkiver
no i think it's half way done or so
-
imer
10ish day eta at this rate
-
imer
should be speeding up a bit still as well
-
arkiver
imer: yeah, due to not getting URLs from special interest pages anymore
-
imer
mmh, guess if that's what needs to happen so we can keep up that's fine
-
imer
just a shame there's so much spam :(
-
arkiver
imer: that spam is now out of the way
-
imer
oh cool, nice work :)
-
arkiver
well until it pops up in a different shape
-
arkiver
we'll deal with it then again
-
imer
arkiver: there's still stuff left in the queue though to work through, right? that's just not going to queue more things?
-
imer
-
» pabs reminds arkiver about the FLOSS planets urls-sources PR :)
-
imer
we might want to filter some of the common domains, currently that looks like kalkulatorpolityczny.pl and unternehmen-mut.de (cc JAA)
-
JAA
imer: Those two should be gone now.
-
imer
thanks :)
-
imer
skalle66.de beachvolleyball2005.de multikodzik.pl aktualnewzory.pl tarmed.pl
-
JAA
Hmm, those are <2% each of the queue currently, compared to the 6-7% each on the previous two.
-
imer
just looking at what stands out to me currently (> 1/s)
-
imer
what to do is of course at your discretion, dont have the full picture
-
JAA
1% of the in-memory queue is still 200k URLs, so I'll yeet them.
-
JAA
Done
-
imer
nice, thanks
-
imer
rewby: seeing some -1's again
-
arkiver
moving secondary to redo, so we can put several million PDFs in secondary for archiving
-
AK
oooh buddy
-
arkiver
:)
-
imer
my poor cpu
-
AK
Think mine has melted
-
JAA
RIP
-
AK
My output has dropped, don't think I can manage as many concurrent when it's 90% pdfs
-
AK
Ahh yeah seeing mainly -1 now, think we've filled up
-
arkiver
AK: the PDFs are not going through now actually
-
arkiver
imer: ^
-
arkiver
moving items is 25% done only still
-
imer
yeah
-
arkiver
imer: the PDFs will go into secondary, so the regular URLs will still be going through backfeed, meaning the rate of PDFs is not 100%
-
arkiver
as in relative rate
-
arkiver
-
JAA
509 MB zstd-compressed
-
arkiver
yeah
-
arkiver
it's 1.5 GB decompressed
-
h2ibot
*chuckles* I'm in danger
-
arkiver
:P
-
arkiver
h2ibot: you'll be fine <3
-
JAA
19.7M PDF URLs
-
JAA
Nice :-)
-
h2ibot
-
h2ibot
arkiver: Deduplicating and queuing 19725857 items. (for '
transfer.archivete.am/MYKeH/pdfs.txt')
-
TheTechRobo
Bloom filter's not going to be forgiving you for this one
-
h2ibot
arkiver: Deduplicated and queued 19725857 items. (for '
transfer.archivete.am/MYKeH/pdfs.txt')
-
JAA
\o/
-
arkiver
it's moved to secondary, so it will slowly be eaten
-
nicolas17
chomp
-
imer
load average: 266.38
-
Barto
nice
-
imer
goes down as the targets are clogged :(
-
project10
-
vokunal|m
The problem with optane9 isn't that it's clogged, but it's eating while on the toilet
-
vokunal|m
He wasn't ready for that joke
-
nicolas17
:|
-
project10
:-)