00:01:00 datechnoman: Skipped 3679 invalid URLs: https://transfer.archivete.am/ZUAtV/urls_batch_0003_001.txt.bad-urls.txt (dNky4aw1) 00:01:02 datechnoman: Skipped 56 very long URLs: https://transfer.archivete.am/f2heH/urls_batch_0003_001.txt.skipped.txt (dNky4aw1) 00:01:03 datechnoman: Deduplicating and queuing 24996265 items. (dNky4aw1) 00:03:47 !a https://transfer.archivete.am/xlfj5/urls_batch_0003_002.txt 00:03:47 datechnoman: Registering JYDKFcm2 for '!a https://transfer.archivete.am/xlfj5/urls_batch_0003_002.txt' 00:26:12 datechnoman: Skipped 1459 invalid URLs: https://transfer.archivete.am/uU6Cs/urls_batch_0003_002.txt.bad-urls.txt (JYDKFcm2) 00:26:13 datechnoman: Skipped 18 very long URLs: https://transfer.archivete.am/RymH5/urls_batch_0003_002.txt.skipped.txt (JYDKFcm2) 00:26:14 datechnoman: Deduplicating and queuing 10438983 items. (JYDKFcm2) 00:40:11 datechnoman: Deduplicated and queued 10438985 items. (JYDKFcm2) 00:42:31 !a https://transfer.archivete.am/ZajK0/urls_batch_0004_000.txt 00:42:32 datechnoman: Registering 3sdLU2fj for '!a https://transfer.archivete.am/ZajK0/urls_batch_0004_000.txt' 00:43:36 datechnoman: Deduplicated and queued 24996267 items. (dNky4aw1) 01:10:37 datechnoman I am starting to think you are just making up these naming conventions XD 01:11:05 Lol. Couldnt be bothered fixing the names. They are all .edu url dumps 01:56:49 datechnoman: Skipped 3213 invalid URLs: https://transfer.archivete.am/ZBWtR/urls_batch_0004_000.txt.bad-urls.txt (3sdLU2fj) 01:56:52 datechnoman: Skipped 110 very long URLs: https://transfer.archivete.am/HqMjQ/urls_batch_0004_000.txt.skipped.txt (3sdLU2fj) 01:56:53 datechnoman: Deduplicating and queuing 24996676 items. (3sdLU2fj) 01:57:49 !a https://transfer.archivete.am/B3kMC/urls_batch_0004_001.txt 01:58:04 datechnoman: Registering tXMm0bAm for '!a https://transfer.archivete.am/B3kMC/urls_batch_0004_001.txt' 02:52:17 datechnoman: Skipped 3676 invalid URLs: https://transfer.archivete.am/d2r9j/urls_batch_0004_001.txt.bad-urls.txt (tXMm0bAm) 02:52:20 datechnoman: Skipped 93 very long URLs: https://transfer.archivete.am/x87Ye/urls_batch_0004_001.txt.skipped.txt (tXMm0bAm) 02:52:21 datechnoman: Deduplicating and queuing 24996231 items. (tXMm0bAm) 02:54:02 datechnoman: Deduplicated and queued 24996678 items. (3sdLU2fj) 03:21:57 datechnoman: Deduplicated and queued 24996238 items. (tXMm0bAm) 03:34:16 !a https://transfer.archivete.am/106IEs/urls_batch_0004_002.txt 03:34:16 datechnoman: Registering OsfvvfFA for '!a https://transfer.archivete.am/106IEs/urls_batch_0004_002.txt' 03:34:51 datechnoman: Something went wrong. (OsfvvfFA) 03:37:22 !status 03:37:23 datechnoman: Jobs running: 0, jobs waiting for a slot: 0. 03:37:31 !a https://transfer.archivete.am/106IEs/urls_batch_0004_002.txt 03:37:31 datechnoman: Registering JpujvJBu for '!a https://transfer.archivete.am/106IEs/urls_batch_0004_002.txt' 03:55:04 datechnoman: Skipped 1785 invalid URLs: https://transfer.archivete.am/h7glF/urls_batch_0004_002.txt.bad-urls.txt (JpujvJBu) 03:55:06 datechnoman: Skipped 83 very long URLs: https://transfer.archivete.am/16cz3t/urls_batch_0004_002.txt.skipped.txt (JpujvJBu) 03:55:07 datechnoman: Deduplicating and queuing 12458319 items. (JpujvJBu) 04:09:43 datechnoman: Deduplicated and queued 12458322 items. (JpujvJBu) 04:54:01 !a https://transfer.archivete.am/bfER9/pdf_urls_batch_2.txt 04:54:02 datechnoman: Registering mfcq2ffv for '!a https://transfer.archivete.am/bfER9/pdf_urls_batch_2.txt' 05:31:37 datechnoman: Skipped 7389 invalid URLs: https://transfer.archivete.am/yHHnk/pdf_urls_batch_2.txt.bad-urls.txt (mfcq2ffv) 05:31:39 datechnoman: Skipped 306 very long URLs: https://transfer.archivete.am/10OkrE/pdf_urls_batch_2.txt.skipped.txt (mfcq2ffv) 05:31:40 datechnoman: Deduplicating and queuing 24992305 items. (mfcq2ffv) 05:59:07 datechnoman: Deduplicated and queued 24992312 items. (mfcq2ffv) 06:01:04 !a https://transfer.archivete.am/QBeph/pdf_urls_batch_3.txt 06:01:04 datechnoman: Registering UOAwLCzb for '!a https://transfer.archivete.am/QBeph/pdf_urls_batch_3.txt' 06:40:16 datechnoman: Skipped 2018 invalid URLs: https://transfer.archivete.am/inMwy/pdf_urls_batch_3.txt.bad-urls.txt (UOAwLCzb) 06:40:18 datechnoman: Skipped 194 very long URLs: https://transfer.archivete.am/SiR92/pdf_urls_batch_3.txt.skipped.txt (UOAwLCzb) 06:40:19 datechnoman: Deduplicating and queuing 24997788 items. (UOAwLCzb) 07:09:43 datechnoman: Deduplicated and queued 24997788 items. (UOAwLCzb) 07:55:35 !a https://transfer.archivete.am/uRJWQ/pdf_urls_batch_4.txt 07:55:35 datechnoman: Registering gtW7ugBY for '!a https://transfer.archivete.am/uRJWQ/pdf_urls_batch_4.txt' 08:33:33 datechnoman: Skipped 5075 invalid URLs: https://transfer.archivete.am/Mul4B/pdf_urls_batch_4.txt.bad-urls.txt (gtW7ugBY) 08:33:35 datechnoman: Skipped 431 very long URLs: https://transfer.archivete.am/5IuL8/pdf_urls_batch_4.txt.skipped.txt (gtW7ugBY) 08:33:36 datechnoman: Deduplicating and queuing 24994494 items. (gtW7ugBY) 08:42:28 🤔My workers seem very unhappy with rsync right now 08:42:44 rsync: [sender] read error: Connection reset by peer (104) 08:42:44 rsync error: error in socket IO (code 10) at io.c(806) [sender=3.2.7] 09:02:02 datechnoman: Deduplicated and queued 24994495 items. (gtW7ugBY) 09:25:00 AK - We are all facing the same issue. Been happening for approximately 36 hours :( 09:25:08 !a https://transfer.archivete.am/DKEMb/pdf_urls_batch_5.txt 09:25:09 datechnoman: Registering Oi8pGKmM for '!a https://transfer.archivete.am/DKEMb/pdf_urls_batch_5.txt' 09:34:40 Damn, at least I know it's not just me haha 09:58:03 datechnoman: Skipped 1704 invalid URLs: https://transfer.archivete.am/pLKi9/pdf_urls_batch_5.txt.bad-urls.txt (Oi8pGKmM) 09:58:04 datechnoman: Skipped 15 very long URLs: https://transfer.archivete.am/NOUHw/pdf_urls_batch_5.txt.skipped.txt (Oi8pGKmM) 09:58:05 datechnoman: Deduplicating and queuing 24998281 items. (Oi8pGKmM) 10:19:14 !a https://transfer.archivete.am/lcV83/pdf_urls_batch_6.txt 10:19:14 datechnoman: Registering hCHYZbsW for '!a https://transfer.archivete.am/lcV83/pdf_urls_batch_6.txt' 10:24:59 datechnoman: Deduplicated and queued 24998282 items. (Oi8pGKmM) 10:58:32 datechnoman: Skipped 50 invalid URLs: https://transfer.archivete.am/Nzkoo/pdf_urls_batch_6.txt.bad-urls.txt (hCHYZbsW) 10:58:34 datechnoman: Skipped 70 very long URLs: https://transfer.archivete.am/wT0Pi/pdf_urls_batch_6.txt.skipped.txt (hCHYZbsW) 10:58:35 datechnoman: Deduplicating and queuing 24999880 items. (hCHYZbsW) 11:04:09 !a https://transfer.archivete.am/9Jk7s/pdf_urls_batch_7.txt 11:04:10 datechnoman: Registering WEJa5oB9 for '!a https://transfer.archivete.am/9Jk7s/pdf_urls_batch_7.txt' 11:45:27 datechnoman: Deduplicated and queued 24999889 items. (hCHYZbsW) 11:51:44 datechnoman: Skipped 208 invalid URLs: https://transfer.archivete.am/Z3nAz/pdf_urls_batch_7.txt.bad-urls.txt (WEJa5oB9) 11:51:46 datechnoman: Skipped 126 very long URLs: https://transfer.archivete.am/VOr7b/pdf_urls_batch_7.txt.skipped.txt (WEJa5oB9) 11:51:47 datechnoman: Deduplicating and queuing 24999666 items. (WEJa5oB9) 12:20:17 datechnoman: Deduplicated and queued 24999668 items. (WEJa5oB9) 12:25:08 !a https://transfer.archivete.am/oGx7w/pdf_urls_batch_8.txt 12:25:09 datechnoman: Registering nEwJPUok for '!a https://transfer.archivete.am/oGx7w/pdf_urls_batch_8.txt' 12:29:31 datechnoman: Skipped 100 invalid URLs: https://transfer.archivete.am/HeUHd/pdf_urls_batch_8.txt.bad-urls.txt (nEwJPUok) 12:29:33 datechnoman: Fixed 2 unprintable URLs: https://transfer.archivete.am/EnmrY/pdf_urls_batch_8.txt.not-printable.txt (nEwJPUok) 12:29:34 datechnoman: Skipped 2 very long URLs: https://transfer.archivete.am/yYQxw/pdf_urls_batch_8.txt.skipped.txt (nEwJPUok) 12:29:35 datechnoman: Deduplicating and queuing 2566467 items. (nEwJPUok) 12:32:07 datechnoman: Deduplicated and queued 2566467 items. (nEwJPUok) 13:45:33 AK: Yeah seems about right https://server8.kiska.pw/uploads/e2b1e7b3bbfb8af0/image.png 13:46:12 Moving average looks fine :D 22:22:34 Good Morning/Afternoon rewby rewby|backup. Whenever you have a spare moment, could I please request for a health check of optane10? Its moving data along but seems to be bottlenecking where we previously have not had issues. I am wondering if there are some stuck packer or upload containers for the #// project causing a performance decrease. We are 22:22:34 seeing constant "rsync: [sender] write error: Broken pipe (32)" and "rsync error: error in socket IO (code 10) at io.c(806) [sender=3.2.7]" where the target appears to be resetting/breaking the connections so workers are constantly reuploading the same items over and over. It has been occurring for the past 48 hours and hasn't seemed to clear up on 22:22:34 its own. Here is a screenshot showing the last 3 days and where traffic started going from a normal constant throughput to constantly spiking. https://transfer.archivete.am/dH5UU/worker_traffic_graph.jpg 22:22:35 inline (for browser viewing): https://transfer.archivete.am/inline/dH5UU/worker_traffic_graph.jpg 22:22:44 I hope this information helps! 22:33:07 I assume your incoming traffic would be very spiky :O 23:10:28 Workers are showing mostly max connections -1 but there are some -500 every now and then 23:11:18 Seem to be down to 12TB a day atm. Have previously run up to 30TB a day 23:11:43 I'll leave it in your capable hands :D Thanks in advance! 23:41:32 I'll give it a look in the morning