-
TheTechRobo
-
arkiver
-
arkiver
TheTechRobo: ^
-
h2ibot
arkiver: Deduplicating and queuing 127527 items.
-
TheTechRobo
arkiver: thanks!
-
h2ibot
arkiver: Deduplicated and queued 127527 items.
-
TheTechRobo
arkiver: I assume this goes to todo:backfeed?
-
TheTechRobo
(Since the todo:backfeed is what just blew up)
-
arkiver
yeah
-
arkiver
all done already
-
arkiver
:)
-
TheTechRobo
this project is stupidly fast LOL
-
datechnoman
haha lol
-
datechnoman
was gonna say
-
datechnoman
annnnnnnnnd it gone :P
-
datechnoman
This project eats a million URL's each bite for breakfast
-
flashfire42
Have we considered doing all the URLteam urls through this?
-
datechnoman
makes sense considering we are finding the redirect but the link is unless if the site is no longer online
-
flashfire42
We are finding the redirects but as far as I know we arent actually archiving the found stuff
-
datechnoman
useless*
-
datechnoman
yeah exactly
-
systwi
Thanks for the info, arkiver.
-
systwi
And you didn't have to run that test list through, but thank you for doing so. :-)
-
systwi
Is data saved through here accessible in WARC format afterwards?
-
JAA
Yes, but it's mixed in with everything else going through this project.
-
vortico
I wonder if Hetzner support ever gets sick of "Abuse Messages"
-
vortico
I get 1 every 2-3 days
-
arkiver
vortico: at this point they may know about us and not care anymore :P
-
arkiver
guess everyone here gives the same reply every time
-
datechnoman
Haha don't think they do. 2 months ago I was averaging 10-20 a day requiring replys
-
vortico
geez that's a lot
-
arkiver
datechnoman: interesting, not anymore?
-
datechnoman
When we did the Russian and Ukrainian bulk
-
arkiver
ah right
-
vortico
Oh yeah Ukraine website make sense since we're hitting the same IP more frequently
-
datechnoman
Now its super quiet. Like 1 every 2-3 day requiring no replies. Nothing to worry about
-
datechnoman
I have copy and paste replies at the ready :P
-
vortico
Haha same
-
datechnoman
We're doing important work. Might not seem like it now but in the future when sites are missing
-
vortico
i'm just in it for the high-score
-
vortico
(just kidding, it's cool that we can participate in the WBM)
-
datechnoman
Haha the leader boards are just an insensitive
-
datechnoman
But if they get people donating more conpute that's a bonus
-
datechnoman
Compute**
-
vortico
I've wondered what the economics of storage is, at IA.
-
vortico
If a 16TB HDD costs $300 but each Hetzner server costs $20/yr, they could easily budget for what we're doing.
-
datechnoman
I've head 1TB of data costs like 1000 bucks to store and backup etc over its lifetime
-
vortico
Hmm, sounds about right. Maybe that's out of date, but maybe considering building/staff costs it's easily $1000
-
vortico
Some cool photos of the Petabox here. Would love lots of high-res photos of the new racks.
archive.org/web/petabox.php
-
vortico
Funny that they only had 200TB or so in 2004. I have 160TB in my basement!
-
systwi
vortico: You lost 40TB? :P
-
systwi
I think you meant 200GB.
-
JAA
systwi: They meant that IA had 200 TB of storage in 2004.
-
JAA
So they're about 20 years behind IA now.
-
JAA
The $1000/TB number is probably still reasonably accurate. Hardware prices did decrease but not too much over the last few years. I first heard that sort of figure in 2017 during the SoundButt project, by the way, where it was slightly higher ($1500 to $2000):
twitter.com/textfiles/status/885527796583284741
-
JAA
So something like $1000 to $1500 now seems reasonable to me.
-
systwi
Ohhh, gotcha, I didn't read it close enough.
-
systwi
I guess I interpreted it originally as, "funny that I only had 200TB or so in 2004. I have 160TB in my basement now!"
-
arkiver
-
arkiver
lennier1: ^ your original appstore list
-
h2ibot
arkiver: Deduplicating and queuing 13492858 items.
-
h2ibot
arkiver: Deduplicated and queued 13492858 items.
-
arkiver
hmm this may be larger in size than I expected
-
arkiver
took it out for now
-
lennier1
arkiver: How large? Was that getting the screenshots and artwork? If so, a lot of those will be duplicates since that file includes links to the same app listed in multiple regions. I also uploaded separate lists of screenshots and artwork.
-
arkiver
lennier1: i see, i put it backj
-
lennier1
Cool. Yeah, I found about 1.6M unique apps.