00:15:00 datechnoman: Skipped 4923 invalid URLs: https://transfer.archivete.am/7KQnH/urls_batch_0004_002.txt.bad-urls.txt (N2VG7BEl) 00:15:05 datechnoman: Skipped 68 very long URLs: https://transfer.archivete.am/ptp4U/urls_batch_0004_002.txt.skipped.txt (N2VG7BEl) 00:15:06 datechnoman: Deduplicating and queuing 20671741 items. (N2VG7BEl) 00:28:01 !a https://transfer.archivete.am/G3T8O/urls_batch_0005_000.txt 00:28:01 datechnoman: Registering 4zU9DwyS for '!a https://transfer.archivete.am/G3T8O/urls_batch_0005_000.txt' 00:28:03 !status 00:28:04 datechnoman: Jobs running: 3, jobs waiting for a slot: 0. 00:28:13 !a https://transfer.archivete.am/Lg4v9/urls_batch_0005_001.txt 00:28:14 datechnoman: Registering jc0fKHEd for '!a https://transfer.archivete.am/Lg4v9/urls_batch_0005_001.txt' 00:28:16 !status 00:28:16 datechnoman: Jobs running: 4, jobs waiting for a slot: 0. 00:29:45 datechnoman: Skipped 5437 invalid URLs: https://transfer.archivete.am/af8zx/urls_batch_0004_001.txt.bad-urls.txt (Dhc3Bsi7) 00:29:48 datechnoman: Skipped 113 very long URLs: https://transfer.archivete.am/SGbMm/urls_batch_0004_001.txt.skipped.txt (Dhc3Bsi7) 00:29:49 datechnoman: Deduplicating and queuing 24994450 items. (Dhc3Bsi7) 00:51:25 datechnoman: Deduplicated and queued 20672145 items. (N2VG7BEl) 00:55:22 datechnoman: Skipped 2153 invalid URLs: https://transfer.archivete.am/cxjrw/urls_batch_0005_001.txt.bad-urls.txt (jc0fKHEd) 00:55:23 datechnoman: Skipped 17 very long URLs: https://transfer.archivete.am/p6251/urls_batch_0005_001.txt.skipped.txt (jc0fKHEd) 00:55:24 datechnoman: Deduplicating and queuing 6755677 items. (jc0fKHEd) 01:06:37 datechnoman: Deduplicated and queued 6755801 items. (jc0fKHEd) 01:08:37 datechnoman: Deduplicated and queued 24994754 items. (Dhc3Bsi7) 01:11:20 !a https://transfer.archivete.am/dJzNi/filtered_doi_org_sorted.txt 01:11:22 datechnoman: Registering ifzcshQK for '!a https://transfer.archivete.am/dJzNi/filtered_doi_org_sorted.txt' 01:42:08 !a https://transfer.archivete.am/9SsRV/filtered_doi_org_unique.txt 01:42:09 datechnoman: Registering XYO7FPUo for '!a https://transfer.archivete.am/9SsRV/filtered_doi_org_unique.txt' 01:42:15 !status 01:42:16 datechnoman: Jobs running: 3, jobs waiting for a slot: 0. 01:49:55 datechnoman: Skipped 5692 invalid URLs: https://transfer.archivete.am/B1UNt/urls_batch_0005_000.txt.bad-urls.txt (4zU9DwyS) 01:50:00 datechnoman: Skipped 61 very long URLs: https://transfer.archivete.am/CwCUL/urls_batch_0005_000.txt.skipped.txt (4zU9DwyS) 01:50:01 datechnoman: Deduplicating and queuing 24994246 items. (4zU9DwyS) 02:23:03 !status 02:23:04 datechnoman: Jobs running: 3, jobs waiting for a slot: 0. 02:27:33 datechnoman: Deduplicated and queued 24994605 items. (4zU9DwyS) 02:33:36 !a https://transfer.archivete.am/pC2EQ/doi_urls_batch_0000_000.txt 02:33:36 !a https://transfer.archivete.am/gW9UQ/doi_urls_batch_0000_001.txt 02:33:36 datechnoman: Registering UgCMGLH0 for '!a https://transfer.archivete.am/pC2EQ/doi_urls_batch_0000_000.txt' 02:33:37 datechnoman: Registering JM1RxPRc for '!a https://transfer.archivete.am/gW9UQ/doi_urls_batch_0000_001.txt' 02:40:24 datechnoman: Skipped 1732 invalid URLs: https://transfer.archivete.am/p4swq/filtered_doi_org_sorted.txt.bad-urls.txt (ifzcshQK) 02:40:25 datechnoman: Deduplicating and queuing 20627203 items. (ifzcshQK) 02:41:04 !status 02:41:05 datechnoman: Jobs running: 4, jobs waiting for a slot: 0. 02:49:00 datechnoman: Skipped 1104 invalid URLs: https://transfer.archivete.am/w68Ib/filtered_doi_org_unique.txt.bad-urls.txt (XYO7FPUo) 02:49:01 datechnoman: Deduplicating and queuing 16459689 items. (XYO7FPUo) 03:03:19 datechnoman: Deduplicated and queued 20627203 items. (ifzcshQK) 03:08:16 datechnoman: Deduplicated and queued 16459689 items. (XYO7FPUo) 03:18:57 !status 03:18:57 datechnoman: Jobs running: 2, jobs waiting for a slot: 0. 03:19:17 !a https://transfer.archivete.am/wIHIO/doi_urls_batch_0000_002.txt 03:19:18 !a https://transfer.archivete.am/YiSJj/doi_urls_batch_0000_003.txt 03:19:18 datechnoman: Registering eVxaiC2q for '!a https://transfer.archivete.am/wIHIO/doi_urls_batch_0000_002.txt' 03:19:19 datechnoman: Registering DpFOw6i3 for '!a https://transfer.archivete.am/YiSJj/doi_urls_batch_0000_003.txt' 03:21:30 datechnoman it doesnt look to be moving? 03:22:01 Flashfire42 - what isnt moving? The project/tracker? 03:22:26 the dashboard of the tracker 03:22:40 Yeah. The project has been "down" for the past 7 days :( 03:23:01 We are waiting on rewby|backup to come back and work their magic on the target to get it back up and running 04:47:59 !status 04:48:00 datechnoman: Jobs running: 4, jobs waiting for a slot: 0. 04:48:25 !a https://transfer.archivete.am/TXQBF/urls_batch_0000_000.txt 04:48:25 datechnoman: Registering HyGvCt5q for '!a https://transfer.archivete.am/TXQBF/urls_batch_0000_000.txt' 04:48:40 !status 04:48:41 datechnoman: Jobs running: 5, jobs waiting for a slot: 0. 04:50:05 arkiver - Any chance that the jobs running can be limited to 2-3 which would allow me to just dump all of the batches in and walk away while they process? I am unsure on the current job limit but I am worried if I dump a state in they will all try to extract at the same time and kill the bot :( 07:06:08 datechnoman: Skipped 29 invalid URLs: https://transfer.archivete.am/12BzjE/doi_urls_batch_0000_000.txt.bad-urls.txt (UgCMGLH0) 07:06:09 datechnoman: Deduplicating and queuing 24999971 items. (UgCMGLH0) 07:12:39 datechnoman: Skipped 24 invalid URLs: https://transfer.archivete.am/yRGFR/doi_urls_batch_0000_001.txt.bad-urls.txt (JM1RxPRc) 07:12:40 datechnoman: Deduplicating and queuing 24999976 items. (JM1RxPRc) 07:42:27 datechnoman: Skipped 8 invalid URLs: https://transfer.archivete.am/XTLq/doi_urls_batch_0000_003.txt.bad-urls.txt (DpFOw6i3) 07:42:28 datechnoman: Deduplicating and queuing 16843467 items. (DpFOw6i3) 08:02:12 datechnoman: Deduplicated and queued 24999971 items. (UgCMGLH0) 08:12:48 datechnoman: Deduplicated and queued 24999976 items. (JM1RxPRc) 08:20:53 datechnoman: Deduplicated and queued 16843467 items. (DpFOw6i3) 08:43:25 !status 08:43:25 datechnoman: Jobs running: 2, jobs waiting for a slot: 0. 08:48:34 !a https://transfer.archivete.am/SvjhZ/urls_batch_0000_001.txt 08:48:34 !a https://transfer.archivete.am/10naxp/urls_batch_0000_002.txt 08:48:34 !a https://transfer.archivete.am/6eGtR/urls_batch_0001_000.txt 08:48:35 datechnoman: Registering DXMBOCFu for '!a https://transfer.archivete.am/SvjhZ/urls_batch_0000_001.txt' 08:48:36 datechnoman: Registering zQZsEcnG for '!a https://transfer.archivete.am/10naxp/urls_batch_0000_002.txt' 08:48:37 datechnoman: Registering hqUYey9D for '!a https://transfer.archivete.am/6eGtR/urls_batch_0001_000.txt' 09:15:48 datechnoman: Skipped 28 invalid URLs: https://transfer.archivete.am/cWH5Q/doi_urls_batch_0000_002.txt.bad-urls.txt (eVxaiC2q) 09:15:49 datechnoman: Deduplicating and queuing 24999972 items. (eVxaiC2q) 09:31:23 datechnoman: Something went wrong. (zQZsEcnG) 09:49:43 datechnoman: Deduplicated and queued 24999972 items. (eVxaiC2q) 09:58:34 datechnoman: Skipped 3843 invalid URLs: https://transfer.archivete.am/gR8HJ/urls_batch_0000_000.txt.bad-urls.txt (HyGvCt5q) 09:58:37 datechnoman: Skipped 181 very long URLs: https://transfer.archivete.am/5hvG4/urls_batch_0000_000.txt.skipped.txt (HyGvCt5q) 09:58:38 datechnoman: Deduplicating and queuing 24995975 items. (HyGvCt5q) 10:32:28 !a https://transfer.archivete.am/10naxp/urls_batch_0000_002.txt 10:32:29 datechnoman: Registering tVXpvwov for '!a https://transfer.archivete.am/10naxp/urls_batch_0000_002.txt' 10:32:42 !status 10:32:43 datechnoman: Jobs running: 4, jobs waiting for a slot: 0. 10:37:24 datechnoman: Deduplicated and queued 24995977 items. (HyGvCt5q) 12:13:02 !status 12:13:03 datechnoman: Jobs running: 3, jobs waiting for a slot: 0. 12:15:35 datechnoman: Skipped 3613 invalid URLs: https://transfer.archivete.am/GwtWF/urls_batch_0001_000.txt.bad-urls.txt (hqUYey9D) 12:15:43 datechnoman: Skipped 147 very long URLs: https://transfer.archivete.am/8yQ84/urls_batch_0001_000.txt.skipped.txt (hqUYey9D) 12:15:44 datechnoman: Deduplicating and queuing 24996239 items. (hqUYey9D) 12:25:06 datechnoman: Skipped 3697 invalid URLs: https://transfer.archivete.am/fcvzF/urls_batch_0000_001.txt.bad-urls.txt (DXMBOCFu) 12:25:10 datechnoman: Skipped 198 very long URLs: https://transfer.archivete.am/a5CIW/urls_batch_0000_001.txt.skipped.txt (DXMBOCFu) 12:25:11 datechnoman: Deduplicating and queuing 24996105 items. (DXMBOCFu) 12:54:28 datechnoman: Skipped 3077 invalid URLs: https://transfer.archivete.am/9DxCm/urls_batch_0000_002.txt.bad-urls.txt (tVXpvwov) 12:54:30 datechnoman: Skipped 194 very long URLs: https://transfer.archivete.am/DZq61/urls_batch_0000_002.txt.skipped.txt (tVXpvwov) 12:54:31 datechnoman: Deduplicating and queuing 17655467 items. (tVXpvwov) 12:59:22 datechnoman: Deduplicated and queued 24996241 items. (hqUYey9D) 13:07:42 datechnoman: Deduplicated and queued 24996110 items. (DXMBOCFu) 13:08:25 !status 13:08:25 datechnoman: Jobs running: 1, jobs waiting for a slot: 0. 13:09:02 !a https://transfer.archivete.am/tWwDY/urls_batch_0001_001.txt 13:09:03 !a https://transfer.archivete.am/5sgWk/urls_batch_0001_002.txt 13:09:03 !a https://transfer.archivete.am/jIydO/urls_batch_0002_000.txt 13:09:03 !a https://transfer.archivete.am/oOGvm/urls_batch_0002_001.txt 13:09:03 !a https://transfer.archivete.am/gzpA7/urls_batch_0002_002.txt 13:09:03 datechnoman: Registering Sx2vvZIW for '!a https://transfer.archivete.am/tWwDY/urls_batch_0001_001.txt' 13:09:04 datechnoman: Registering LH5O5J5N for '!a https://transfer.archivete.am/5sgWk/urls_batch_0001_002.txt' 13:09:05 datechnoman: Registering tWpMu5VX for '!a https://transfer.archivete.am/oOGvm/urls_batch_0002_001.txt' 13:09:06 datechnoman: Registering BQKwTjAY for '!a https://transfer.archivete.am/jIydO/urls_batch_0002_000.txt' 13:09:07 datechnoman: Registering wHXLosFP for '!a https://transfer.archivete.am/gzpA7/urls_batch_0002_002.txt' 13:19:03 datechnoman: Deduplicated and queued 17655470 items. (tVXpvwov) 13:41:11 datechnoman: can we wait a little here with queuing more until everything is rolling again? 13:41:23 and yes i can reduce the maximum number of jobs 13:41:36 time to deploy zramctl on the tracker 15:58:10 datechnoman: Skipped 2779 invalid URLs: https://transfer.archivete.am/11v07v/urls_batch_0001_002.txt.bad-urls.txt (LH5O5J5N) 15:58:17 datechnoman: Skipped 150 very long URLs: https://transfer.archivete.am/8BTt5/urls_batch_0001_002.txt.skipped.txt (LH5O5J5N) 15:58:19 datechnoman: Deduplicating and queuing 18129493 items. (LH5O5J5N) 16:44:38 datechnoman: Deduplicated and queued 18129495 items. (LH5O5J5N) 17:08:27 datechnoman: Skipped 3284 invalid URLs: https://transfer.archivete.am/ZK7mW/urls_batch_0002_002.txt.bad-urls.txt (wHXLosFP) 17:08:30 datechnoman: Skipped 148 very long URLs: https://transfer.archivete.am/B2o21/urls_batch_0002_002.txt.skipped.txt (wHXLosFP) 17:08:31 datechnoman: Deduplicating and queuing 18436424 items. (wHXLosFP) 17:46:19 datechnoman: Deduplicated and queued 18436426 items. (wHXLosFP) 17:58:01 datechnoman: Skipped 3999 invalid URLs: https://transfer.archivete.am/2oaj0/urls_batch_0001_001.txt.bad-urls.txt (Sx2vvZIW) 17:58:06 datechnoman: Skipped 198 very long URLs: https://transfer.archivete.am/6xwCr/urls_batch_0001_001.txt.skipped.txt (Sx2vvZIW) 17:58:07 datechnoman: Deduplicating and queuing 24995803 items. (Sx2vvZIW) 18:00:16 datechnoman: Skipped 4267 invalid URLs: https://transfer.archivete.am/158o3G/urls_batch_0002_001.txt.bad-urls.txt (tWpMu5VX) 18:00:20 datechnoman: Skipped 157 very long URLs: https://transfer.archivete.am/DsEnl/urls_batch_0002_001.txt.skipped.txt (tWpMu5VX) 18:00:21 datechnoman: Deduplicating and queuing 24995576 items. (tWpMu5VX) 18:09:38 datechnoman: Skipped 3583 invalid URLs: https://transfer.archivete.am/JzFFG/urls_batch_0002_000.txt.bad-urls.txt (BQKwTjAY) 18:09:40 datechnoman: Skipped 212 very long URLs: https://transfer.archivete.am/gsl10/urls_batch_0002_000.txt.skipped.txt (BQKwTjAY) 18:09:41 datechnoman: Deduplicating and queuing 24996204 items. (BQKwTjAY) 18:20:50 here it's mainly pdf files that are archived, right? :o 18:41:11 datechnoman: Deduplicated and queued 24995805 items. (Sx2vvZIW) 18:42:41 datechnoman: Deduplicated and queued 24995578 items. (tWpMu5VX) 18:47:55 datechnoman: Deduplicated and queued 24996208 items. (BQKwTjAY) 20:38:26 arkiver - Holding off queuing until things get rolling again as requested :) 21:47:03 the data, it flows 21:54:24 I'm suddenly moist 21:54:41 Thank you sooooo much rewby 21:54:58 wonder how long it will take us to chew through 825 million 21:55:38 Well most of them are edu urls and we archivd the outlinks so it will take a bit 21:55:53 Plus it's nearly the start of a new month so bulk sitemaps