01:23:38 !a https://transfer.archivete.am/bgybv/urls_batch_0005_002.txt 01:23:39 datechnoman: Registering LfMSW0gm for '!a https://transfer.archivete.am/bgybv/urls_batch_0005_002.txt' 01:27:50 datechnoman: Skipped 808 invalid URLs: https://transfer.archivete.am/zdtY4/urls_batch_0005_002.txt.bad-urls.txt (LfMSW0gm) 01:27:51 datechnoman: Skipped 6 very long URLs: https://transfer.archivete.am/5VLxh/urls_batch_0005_002.txt.skipped.txt (LfMSW0gm) 01:27:52 datechnoman: Deduplicating and queuing 2359959 items. (LfMSW0gm) 01:30:52 datechnoman: Deduplicated and queued 2359959 items. (LfMSW0gm) 04:01:56 more work https://transfer.archivete.am/mP4z5/pdf_makenyanyunhappy_8.txt.zst https://transfer.archivete.am/5ZuFD/pdf_makenyanyunhappy_9.txt.zst 04:04:27 !a https://transfer.archivete.am/mP4z5/pdf_makenyanyunhappy_8.txt 04:04:27 datechnoman: Registering aSNW6v3h for '!a https://transfer.archivete.am/mP4z5/pdf_makenyanyunhappy_8.txt' 04:16:58 ty 04:17:02 datechnoman++ 04:17:03 -eggdrop- [karma] 'datechnoman' now has 13 karma! 04:19:37 datechnoman: Deduplicating and queuing 9640943 items. (aSNW6v3h) 04:31:02 datechnoman: Deduplicated and queued 9640944 items. (aSNW6v3h) 04:33:19 !a https://transfer.archivete.am/5ZuFD/pdf_makenyanyunhappy_9.txt 04:33:22 datechnoman: Registering xVlUxrsp for '!a https://transfer.archivete.am/5ZuFD/pdf_makenyanyunhappy_9.txt' 04:33:29 imer - I got you fam :) 04:34:36 Feed the beast :P 04:45:19 Its nearly a new month. I wonder if nyany is going to poke their head out soon haha 04:50:08 datechnoman: Deduplicating and queuing 9640943 items. (xVlUxrsp) 05:00:32 datechnoman: Deduplicated and queued 9640943 items. (xVlUxrsp) 05:59:06 !a https://transfer.archivete.am/Oo2j1/urls_batch_0000_000.txt 05:59:07 datechnoman: Registering PBgANBoM for '!a https://transfer.archivete.am/Oo2j1/urls_batch_0000_000.txt' 05:59:47 I have no clue how you guys go so fast here. Last i checked, last night, there were 15M in the queue, then another like hundred+ million were queued over today, and there's 27 million in the queue now 07:34:52 We dont muck around here ;) 07:37:53 gotta reach that 8 PB somehow 07:42:21 datechnoman: Skipped 3780 invalid URLs: https://transfer.archivete.am/T1OdS/urls_batch_0000_000.txt.bad-urls.txt (PBgANBoM) 07:42:23 datechnoman: Skipped 50 very long URLs: https://transfer.archivete.am/OCrCT/urls_batch_0000_000.txt.skipped.txt (PBgANBoM) 07:42:24 datechnoman: Deduplicating and queuing 24996169 items. (PBgANBoM) 07:49:25 that_lurker - 8pb, that's child's play haha :P ;) 07:49:45 We aren't far off it though! 07:50:01 that is true, but that is the next milestone :-) 08:22:10 datechnoman: Deduplicated and queued 24996173 items. (PBgANBoM) 09:05:21 !a https://transfer.archivete.am/bHKEW/urls_batch_0000_001.txt 09:05:21 datechnoman: Registering S1cu3CKI for '!a https://transfer.archivete.am/bHKEW/urls_batch_0000_001.txt' 09:50:19 datechnoman: Skipped 3925 invalid URLs: https://transfer.archivete.am/118EMK/urls_batch_0000_001.txt.bad-urls.txt (S1cu3CKI) 09:50:21 datechnoman: Skipped 102 very long URLs: https://transfer.archivete.am/9Qp49/urls_batch_0000_001.txt.skipped.txt (S1cu3CKI) 09:50:22 datechnoman: Deduplicating and queuing 24995973 items. (S1cu3CKI) 10:23:30 datechnoman: Deduplicated and queued 24995981 items. (S1cu3CKI) 12:22:31 !a https://transfer.archivete.am/gIdoQ/urls_batch_0000_002.txt 12:22:31 datechnoman: Registering tEyEhj1K for '!a https://transfer.archivete.am/gIdoQ/urls_batch_0000_002.txt' 12:45:19 datechnoman: Skipped 2039 invalid URLs: https://transfer.archivete.am/MC9Nv/urls_batch_0000_002.txt.bad-urls.txt (tEyEhj1K) 12:45:20 datechnoman: Skipped 74 very long URLs: https://transfer.archivete.am/fWw2R/urls_batch_0000_002.txt.skipped.txt (tEyEhj1K) 12:45:21 datechnoman: Deduplicating and queuing 12055109 items. (tEyEhj1K) 13:02:58 datechnoman: Deduplicated and queued 12055111 items. (tEyEhj1K) 13:03:20 !a https://transfer.archivete.am/8LZzm/urls_batch_0001_000.txt 13:03:21 datechnoman: Registering zUPmiac7 for '!a https://transfer.archivete.am/8LZzm/urls_batch_0001_000.txt' 13:51:20 datechnoman: Skipped 3576 invalid URLs: https://transfer.archivete.am/JiDh6/urls_batch_0001_000.txt.bad-urls.txt (zUPmiac7) 13:51:22 datechnoman: Skipped 101 very long URLs: https://transfer.archivete.am/P66uR/urls_batch_0001_000.txt.skipped.txt (zUPmiac7) 13:51:23 datechnoman: Deduplicating and queuing 24996322 items. (zUPmiac7) 14:25:38 datechnoman: Deduplicated and queued 24996324 items. (zUPmiac7) 22:35:06 Will only add one more batch in as im going to let the queue drain right back due to the start of the month sitemap queuing that will be coming in a day or two :O 22:35:31 !a https://transfer.archivete.am/yqQuS/urls_batch_0001_001.txt 22:35:32 datechnoman: Registering gkBbZHo8 for '!a https://transfer.archivete.am/yqQuS/urls_batch_0001_001.txt' 23:42:56 I think I remember when we got up to 1PB here 23:43:16 and I thought "damn, that's a LOT of data" 23:55:42 datechnoman: Skipped 3425 invalid URLs: https://transfer.archivete.am/L67Dt/urls_batch_0001_001.txt.bad-urls.txt (gkBbZHo8) 23:55:46 datechnoman: Skipped 104 very long URLs: https://transfer.archivete.am/TIWxT/urls_batch_0001_001.txt.skipped.txt (gkBbZHo8) 23:55:47 datechnoman: Deduplicating and queuing 24996471 items. (gkBbZHo8)