-
AKI remember days when this got up to 100m items queued
-
imerI dont want to know how much is currently stashed away.. gotta be an amount
-
arkiverhundreds of millions
-
project10archiveteam running a mini heritrix
-
arkiverwe're way bigger :P
-
project10oh, interesting. How many unique warrior IPs are there? :)
-
fireonlivejava đ±
-
project10
-
vokunal|mThis is the stash just for reddit outlinks I think. There might be more stashes, but this is the one I remember. tracker.archiveteam.org/urls-stash-reddit/#show-all
-
vokunal|mI was thinking it would have grown more since I last saw it, but I forgot shreddit was off for a while too
-
nstrom|mQueuing for parent URL ayalon-ins.co.il/radware_waf/?_even…a7be5f1407bb993342c012cefbdf2972895.
-
nstrom|m
-
nstrom|mlooks to be a loop
-
project10"radware_waf" đŠč
-
imerSo, a billion urls easy? IA will love us once we destroy their newly fixed storage situation haha
-
» arkiver is trying to prevent that
-
arkiverwe're often walking some delicate line between "too much" (IA side) and "not enough" (both sides)
-
imerdoing a good job of that as far as I can tell from the outside :)
-
arkiverthanks :)
-
arkiverlast weeks have been challenging
-
arkiverand we have the big youtube troubles before the IA problems
-
katiaare they breaking archiving along with the ad popup?
-
arkiverad popup?
-
arkiveroh youtube
-
arkiverno
-
katiaadblock
-
arkiverrules for what could be queued for youtube project were too relaxed, and we did *a lot* for a few days with me not restricting that in time
-
katiaah
-
vokunal|mdown-the-tube is probably the single most dangerous thing for ia. It's throughput can go off the roof if not held back lol.
-
vokunal|mwe could probably undue all their progress in a single day if the limits were removed
-
vokunal|mbetween the queue and what's stashed I imagine there's somewhere near 300TB waiting to be archived
-
arkivermoving a stash from yesterday back
-
arkivervokunal|m: there's a lot stashed and yes
-
AKOh good, something for my workers to get stuck into for a bit
-
arkiverAK: enjoy :)
-
AKNot sure how many you put in but we do seem to be getting through them haha
-
that_lurkerI have 2 vps:s and my home server running a few projects so always good when they are not idle. Though its good to wait for IA to become ready again for the cheer amount these projects produce
-
JAAarkiver: Ack