-
h2ibotdatechnoman: Skipped 4923 invalid URLs: transfer.archivete.am/7KQnH/urls_batch_0004_002.txt.bad-urls.txt (N2VG7BEl)
-
h2ibotdatechnoman: Skipped 68 very long URLs: transfer.archivete.am/ptp4U/urls_batch_0004_002.txt.skipped.txt (N2VG7BEl)
-
h2ibotdatechnoman: Deduplicating and queuing 20671741 items. (N2VG7BEl)
-
datechnoman
-
h2ibotdatechnoman: Registering 4zU9DwyS for '!a transfer.archivete.am/G3T8O/urls_batch_0005_000.txt'
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 3, jobs waiting for a slot: 0.
-
datechnoman
-
h2ibotdatechnoman: Registering jc0fKHEd for '!a transfer.archivete.am/Lg4v9/urls_batch_0005_001.txt'
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 4, jobs waiting for a slot: 0.
-
h2ibotdatechnoman: Skipped 5437 invalid URLs: transfer.archivete.am/af8zx/urls_batch_0004_001.txt.bad-urls.txt (Dhc3Bsi7)
-
h2ibotdatechnoman: Skipped 113 very long URLs: transfer.archivete.am/SGbMm/urls_batch_0004_001.txt.skipped.txt (Dhc3Bsi7)
-
h2ibotdatechnoman: Deduplicating and queuing 24994450 items. (Dhc3Bsi7)
-
h2ibotdatechnoman: Deduplicated and queued 20672145 items. (N2VG7BEl)
-
h2ibotdatechnoman: Skipped 2153 invalid URLs: transfer.archivete.am/cxjrw/urls_batch_0005_001.txt.bad-urls.txt (jc0fKHEd)
-
h2ibotdatechnoman: Skipped 17 very long URLs: transfer.archivete.am/p6251/urls_batch_0005_001.txt.skipped.txt (jc0fKHEd)
-
h2ibotdatechnoman: Deduplicating and queuing 6755677 items. (jc0fKHEd)
-
h2ibotdatechnoman: Deduplicated and queued 6755801 items. (jc0fKHEd)
-
h2ibotdatechnoman: Deduplicated and queued 24994754 items. (Dhc3Bsi7)
-
datechnoman
-
h2ibotdatechnoman: Registering ifzcshQK for '!a transfer.archivete.am/dJzNi/filtered_doi_org_sorted.txt'
-
datechnoman
-
h2ibotdatechnoman: Registering XYO7FPUo for '!a transfer.archivete.am/9SsRV/filtered_doi_org_unique.txt'
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 3, jobs waiting for a slot: 0.
-
h2ibotdatechnoman: Skipped 5692 invalid URLs: transfer.archivete.am/B1UNt/urls_batch_0005_000.txt.bad-urls.txt (4zU9DwyS)
-
h2ibotdatechnoman: Skipped 61 very long URLs: transfer.archivete.am/CwCUL/urls_batch_0005_000.txt.skipped.txt (4zU9DwyS)
-
h2ibotdatechnoman: Deduplicating and queuing 24994246 items. (4zU9DwyS)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 3, jobs waiting for a slot: 0.
-
h2ibotdatechnoman: Deduplicated and queued 24994605 items. (4zU9DwyS)
-
datechnoman
-
datechnoman
-
h2ibotdatechnoman: Registering UgCMGLH0 for '!a transfer.archivete.am/pC2EQ/doi_urls_batch_0000_000.txt'
-
h2ibotdatechnoman: Registering JM1RxPRc for '!a transfer.archivete.am/gW9UQ/doi_urls_batch_0000_001.txt'
-
h2ibotdatechnoman: Skipped 1732 invalid URLs: transfer.archivete.am/p4swq/filtered_doi_org_sorted.txt.bad-urls.txt (ifzcshQK)
-
h2ibotdatechnoman: Deduplicating and queuing 20627203 items. (ifzcshQK)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 4, jobs waiting for a slot: 0.
-
h2ibotdatechnoman: Skipped 1104 invalid URLs: transfer.archivete.am/w68Ib/filtered_doi_org_unique.txt.bad-urls.txt (XYO7FPUo)
-
h2ibotdatechnoman: Deduplicating and queuing 16459689 items. (XYO7FPUo)
-
h2ibotdatechnoman: Deduplicated and queued 20627203 items. (ifzcshQK)
-
h2ibotdatechnoman: Deduplicated and queued 16459689 items. (XYO7FPUo)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 2, jobs waiting for a slot: 0.
-
datechnoman
-
datechnoman
-
h2ibotdatechnoman: Registering eVxaiC2q for '!a transfer.archivete.am/wIHIO/doi_urls_batch_0000_002.txt'
-
h2ibotdatechnoman: Registering DpFOw6i3 for '!a transfer.archivete.am/YiSJj/doi_urls_batch_0000_003.txt'
-
Flashfire42datechnoman it doesnt look to be moving?
-
datechnomanFlashfire42 - what isnt moving? The project/tracker?
-
Flashfire42the dashboard of the tracker
-
datechnomanYeah. The project has been "down" for the past 7 days :(
-
datechnomanWe are waiting on rewby|backup to come back and work their magic on the target to get it back up and running
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 4, jobs waiting for a slot: 0.
-
datechnoman
-
h2ibotdatechnoman: Registering HyGvCt5q for '!a transfer.archivete.am/TXQBF/urls_batch_0000_000.txt'
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 5, jobs waiting for a slot: 0.
-
datechnomanarkiver - Any chance that the jobs running can be limited to 2-3 which would allow me to just dump all of the batches in and walk away while they process? I am unsure on the current job limit but I am worried if I dump a state in they will all try to extract at the same time and kill the bot :(
-
h2ibotdatechnoman: Skipped 29 invalid URLs: transfer.archivete.am/12BzjE/doi_urls_batch_0000_000.txt.bad-urls.txt (UgCMGLH0)
-
h2ibotdatechnoman: Deduplicating and queuing 24999971 items. (UgCMGLH0)
-
h2ibotdatechnoman: Skipped 24 invalid URLs: transfer.archivete.am/yRGFR/doi_urls_batch_0000_001.txt.bad-urls.txt (JM1RxPRc)
-
h2ibotdatechnoman: Deduplicating and queuing 24999976 items. (JM1RxPRc)
-
h2ibotdatechnoman: Skipped 8 invalid URLs: transfer.archivete.am/XTLq/doi_urls_batch_0000_003.txt.bad-urls.txt (DpFOw6i3)
-
h2ibotdatechnoman: Deduplicating and queuing 16843467 items. (DpFOw6i3)
-
h2ibotdatechnoman: Deduplicated and queued 24999971 items. (UgCMGLH0)
-
h2ibotdatechnoman: Deduplicated and queued 24999976 items. (JM1RxPRc)
-
h2ibotdatechnoman: Deduplicated and queued 16843467 items. (DpFOw6i3)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 2, jobs waiting for a slot: 0.
-
datechnoman
-
datechnoman
-
datechnoman
-
h2ibotdatechnoman: Registering DXMBOCFu for '!a transfer.archivete.am/SvjhZ/urls_batch_0000_001.txt'
-
h2ibotdatechnoman: Registering zQZsEcnG for '!a transfer.archivete.am/10naxp/urls_batch_0000_002.txt'
-
h2ibotdatechnoman: Registering hqUYey9D for '!a transfer.archivete.am/6eGtR/urls_batch_0001_000.txt'
-
h2ibotdatechnoman: Skipped 28 invalid URLs: transfer.archivete.am/cWH5Q/doi_urls_batch_0000_002.txt.bad-urls.txt (eVxaiC2q)
-
h2ibotdatechnoman: Deduplicating and queuing 24999972 items. (eVxaiC2q)
-
h2ibotdatechnoman: Something went wrong. (zQZsEcnG)
-
h2ibotdatechnoman: Deduplicated and queued 24999972 items. (eVxaiC2q)
-
h2ibotdatechnoman: Skipped 3843 invalid URLs: transfer.archivete.am/gR8HJ/urls_batch_0000_000.txt.bad-urls.txt (HyGvCt5q)
-
h2ibotdatechnoman: Skipped 181 very long URLs: transfer.archivete.am/5hvG4/urls_batch_0000_000.txt.skipped.txt (HyGvCt5q)
-
h2ibotdatechnoman: Deduplicating and queuing 24995975 items. (HyGvCt5q)
-
datechnoman
-
h2ibotdatechnoman: Registering tVXpvwov for '!a transfer.archivete.am/10naxp/urls_batch_0000_002.txt'
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 4, jobs waiting for a slot: 0.
-
h2ibotdatechnoman: Deduplicated and queued 24995977 items. (HyGvCt5q)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 3, jobs waiting for a slot: 0.
-
h2ibotdatechnoman: Skipped 3613 invalid URLs: transfer.archivete.am/GwtWF/urls_batch_0001_000.txt.bad-urls.txt (hqUYey9D)
-
h2ibotdatechnoman: Skipped 147 very long URLs: transfer.archivete.am/8yQ84/urls_batch_0001_000.txt.skipped.txt (hqUYey9D)
-
h2ibotdatechnoman: Deduplicating and queuing 24996239 items. (hqUYey9D)
-
h2ibotdatechnoman: Skipped 3697 invalid URLs: transfer.archivete.am/fcvzF/urls_batch_0000_001.txt.bad-urls.txt (DXMBOCFu)
-
h2ibotdatechnoman: Skipped 198 very long URLs: transfer.archivete.am/a5CIW/urls_batch_0000_001.txt.skipped.txt (DXMBOCFu)
-
h2ibotdatechnoman: Deduplicating and queuing 24996105 items. (DXMBOCFu)
-
h2ibotdatechnoman: Skipped 3077 invalid URLs: transfer.archivete.am/9DxCm/urls_batch_0000_002.txt.bad-urls.txt (tVXpvwov)
-
h2ibotdatechnoman: Skipped 194 very long URLs: transfer.archivete.am/DZq61/urls_batch_0000_002.txt.skipped.txt (tVXpvwov)
-
h2ibotdatechnoman: Deduplicating and queuing 17655467 items. (tVXpvwov)
-
h2ibotdatechnoman: Deduplicated and queued 24996241 items. (hqUYey9D)
-
h2ibotdatechnoman: Deduplicated and queued 24996110 items. (DXMBOCFu)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 1, jobs waiting for a slot: 0.
-
datechnoman
-
datechnoman
-
datechnoman
-
datechnoman
-
datechnoman
-
h2ibotdatechnoman: Registering Sx2vvZIW for '!a transfer.archivete.am/tWwDY/urls_batch_0001_001.txt'
-
h2ibotdatechnoman: Registering LH5O5J5N for '!a transfer.archivete.am/5sgWk/urls_batch_0001_002.txt'
-
h2ibotdatechnoman: Registering tWpMu5VX for '!a transfer.archivete.am/oOGvm/urls_batch_0002_001.txt'
-
h2ibotdatechnoman: Registering BQKwTjAY for '!a transfer.archivete.am/jIydO/urls_batch_0002_000.txt'
-
h2ibotdatechnoman: Registering wHXLosFP for '!a transfer.archivete.am/gzpA7/urls_batch_0002_002.txt'
-
h2ibotdatechnoman: Deduplicated and queued 17655470 items. (tVXpvwov)
-
arkiverdatechnoman: can we wait a little here with queuing more until everything is rolling again?
-
arkiverand yes i can reduce the maximum number of jobs
-
katiatime to deploy zramctl on the tracker
-
h2ibotdatechnoman: Skipped 2779 invalid URLs: transfer.archivete.am/11v07v/urls_batch_0001_002.txt.bad-urls.txt (LH5O5J5N)
-
h2ibotdatechnoman: Skipped 150 very long URLs: transfer.archivete.am/8BTt5/urls_batch_0001_002.txt.skipped.txt (LH5O5J5N)
-
h2ibotdatechnoman: Deduplicating and queuing 18129493 items. (LH5O5J5N)
-
h2ibotdatechnoman: Deduplicated and queued 18129495 items. (LH5O5J5N)
-
h2ibotdatechnoman: Skipped 3284 invalid URLs: transfer.archivete.am/ZK7mW/urls_batch_0002_002.txt.bad-urls.txt (wHXLosFP)
-
h2ibotdatechnoman: Skipped 148 very long URLs: transfer.archivete.am/B2o21/urls_batch_0002_002.txt.skipped.txt (wHXLosFP)
-
h2ibotdatechnoman: Deduplicating and queuing 18436424 items. (wHXLosFP)
-
h2ibotdatechnoman: Deduplicated and queued 18436426 items. (wHXLosFP)
-
h2ibotdatechnoman: Skipped 3999 invalid URLs: transfer.archivete.am/2oaj0/urls_batch_0001_001.txt.bad-urls.txt (Sx2vvZIW)
-
h2ibotdatechnoman: Skipped 198 very long URLs: transfer.archivete.am/6xwCr/urls_batch_0001_001.txt.skipped.txt (Sx2vvZIW)
-
h2ibotdatechnoman: Deduplicating and queuing 24995803 items. (Sx2vvZIW)
-
h2ibotdatechnoman: Skipped 4267 invalid URLs: transfer.archivete.am/158o3G/urls_batch_0002_001.txt.bad-urls.txt (tWpMu5VX)
-
h2ibotdatechnoman: Skipped 157 very long URLs: transfer.archivete.am/DsEnl/urls_batch_0002_001.txt.skipped.txt (tWpMu5VX)
-
h2ibotdatechnoman: Deduplicating and queuing 24995576 items. (tWpMu5VX)
-
h2ibotdatechnoman: Skipped 3583 invalid URLs: transfer.archivete.am/JzFFG/urls_batch_0002_000.txt.bad-urls.txt (BQKwTjAY)
-
h2ibotdatechnoman: Skipped 212 very long URLs: transfer.archivete.am/gsl10/urls_batch_0002_000.txt.skipped.txt (BQKwTjAY)
-
h2ibotdatechnoman: Deduplicating and queuing 24996204 items. (BQKwTjAY)
-
Exorcism|mhere it's mainly pdf files that are archived, right? :o
-
h2ibotdatechnoman: Deduplicated and queued 24995805 items. (Sx2vvZIW)
-
h2ibotdatechnoman: Deduplicated and queued 24995578 items. (tWpMu5VX)
-
h2ibotdatechnoman: Deduplicated and queued 24996208 items. (BQKwTjAY)
-
datechnomanarkiver - Holding off queuing until things get rolling again as requested :)
-
nstrom|mthe data, it flows
-
datechnomanI'm suddenly moist
-
datechnomanThank you sooooo much rewby
-
nstrom|mwonder how long it will take us to chew through 825 million
-
datechnomanWell most of them are edu urls and we archivd the outlinks so it will take a bit
-
datechnomanPlus it's nearly the start of a new month so bulk sitemaps