-
TheTechRobo
arkiver: It's fine to queue here, right?
-
TheTechRobo
I've got ~500k URLs from #burnthetwitch
-
TheTechRobo
-
arkiver
TheTechRobo: woah 500k, we will not DDoS anything with that?
-
arkiver
if not, then yes
-
TheTechRobo
arkiver: They're mostly Twitch CDN
-
arkiver
well let's do it
-
fireonlive
:D
-
Pedrosso
so many
-
fireonlive
-
h2ibot
-
fireonlive
(it is !a right?)
-
h2ibot
fireonlive: Deduplicating and queuing 504120 items. (for '
transfer.archivete.am/14vHJu/burnthetwitch-urls')
-
fireonlive
ther we go
-
Pedrosso
look@em numbers go
-
fireonlive
pew pew
-
h2ibot
fireonlive: Deduplicated and queued 504120 items. (for '
transfer.archivete.am/14vHJu/burnthetwitch-urls')
-
imer
-
TheTechRobo
fireonlive: Thanks, I somehow forgot to queue the list. lol
-
fireonlive
np :3
-
Pedrosso
:3
-
TheTechRobo
Here's the remainder:
-
TheTechRobo
-
h2ibot
TheTechRobo: Deduplicating and queuing 78812 items. (for '
transfer.archivete.am/B4Tyi/burnthetwitch-urls')
-
h2ibot
TheTechRobo: Deduplicated and queued 78812 items. (for '
transfer.archivete.am/B4Tyi/burnthetwitch-urls')
-
TheTechRobo
(obligatory thank you to transfer for decompressing zstd on-the-fly)
-
fireonlive
transfer++
-
eggdrop
[karma] 'transfer' now has 1 karma!
-
JAA
:-)
-
TheTechRobo
Zstd is amazing for compression for temporary stuff
-
JAA
s/ for temporary stuff//
-
JAA
:-)
-
DigitalDragons
zstd is your friend
-
fireonlive
the only std you want!
-
TheTechRobo
JAA: Eh, I prefer lzip for long-term storage because it has nice data-recovery tools. (see
nongnu.org/lzip/lziprecover.html)
-
TheTechRobo
fireonlive: what about libstd?
-
fireonlive
i'll allow it!
-
TheTechRobo
also, lzip has an amazing logo, so there's that
-
TheTechRobo
-
JAA
TheTechRobo: Yeah, but it's also much slower. And I'd argue that if you need to recover corrupted files, your entire storage system is already flawed. :-P
-
TheTechRobo
JAA: Isn't AT's mission statement preserving shit "forever"? :P
-
TheTechRobo
Lziprecover provides another avenue to recovering data potentially decades into the future.
-
TheTechRobo
Check and mate. :P
-
JAA
Hmm, so we should use a compression algorithm that takes forever to compress the data in the first place! *taps forehead*
-
nicolas17
if you want the ability to recover corrupted files, use par2 to generate parity data
-
TheTechRobo
nicolas17: But that's work.
-
JAA
You'll never need to recover from corrupted compressed files if you never get to the compressed output.
-
TheTechRobo
JAA: hey don't be mean to lzip like that
-
TheTechRobo
What did lzip ever do to you?
-
JAA
I propose `sleep inf && cat`.
-
TheTechRobo
JAA: we fixed the storage problem! :P
-
JAA
Yay
-
fireonlive
zstd + par2 anyone?
-
fireonlive
anyone?
-
fireonlive
:3
-
nicolas17
isn't paq8 even slower? :D
-
TheTechRobo
Zpaq ftw
-
nicolas17
small files but at what cost
-
TheTechRobo
your power bill
-
TheTechRobo
fireonlive: What about lzip and par2?
-
TheTechRobo
(I'm not shutting up about this)
-
fireonlive
🤔
-
TheTechRobo
lzip is amazing and I will *make* you all realise that
-
nicolas17
xz is ubiquitous
-
fireonlive
TheTechRobo's lzip torture room
-
project10
I wonder what the best compression algo is with respect to a very compute-constrained future
-
» project10 is a party pooper :P
-
TheTechRobo
I love how the creator of BARF (
mattmahoney.net/dc/barf.html) explains how it works in great detail, showing its limitations too (how it stores information in both the decompressor and filename), and then at the bottom says "BARF2 (Oct. 5, 2010) iteratively compresses random data down to 0 bytes and restores it correctly without hiding data
-
TheTechRobo
in the decompresser or in the filename." and refuses to elaborate
-
JAA
X
-
JAA
Hehe
-
TheTechRobo
Ah, it saves the needed information to a separate file. Clever. lol
-
JAA
lol
-
nicolas17
looks like compression is basically treating the file as a a number and subtracting 1
-
nicolas17
then all you need is knowing how many times you have to run the decompression tool
-
JAA
It's described as 'How to cheat at compression benchmarks' elsewhere on the website, so yeah.
-
TheTechRobo
yeah
-
nicolas17
to compress "hi" to 0 bytes you may need to compress it 0x6869 times, then save that number so you know how many times to decompress it to get "hi" back :P
-
JAA
Ah, lol
-
TheTechRobo
-
fireonlive
.cpp? what year is it :D
-
pabs
interesting, these folks have a database of local news orgs in the USA
localnewsinitiative.northwestern.edu stateoflocalnews⊙ne
-
pabs
they describe it as "proprietary" though :(
-
fireonlive
-_-