-
h2ibotdatechnoman: Skipped 3679 invalid URLs: transfer.archivete.am/ZUAtV/urls_batch_0003_001.txt.bad-urls.txt (dNky4aw1)
-
h2ibotdatechnoman: Skipped 56 very long URLs: transfer.archivete.am/f2heH/urls_batch_0003_001.txt.skipped.txt (dNky4aw1)
-
h2ibotdatechnoman: Deduplicating and queuing 24996265 items. (dNky4aw1)
-
datechnoman
-
h2ibotdatechnoman: Registering JYDKFcm2 for '!a transfer.archivete.am/xlfj5/urls_batch_0003_002.txt'
-
h2ibotdatechnoman: Skipped 1459 invalid URLs: transfer.archivete.am/uU6Cs/urls_batch_0003_002.txt.bad-urls.txt (JYDKFcm2)
-
h2ibotdatechnoman: Skipped 18 very long URLs: transfer.archivete.am/RymH5/urls_batch_0003_002.txt.skipped.txt (JYDKFcm2)
-
h2ibotdatechnoman: Deduplicating and queuing 10438983 items. (JYDKFcm2)
-
h2ibotdatechnoman: Deduplicated and queued 10438985 items. (JYDKFcm2)
-
datechnoman
-
h2ibotdatechnoman: Registering 3sdLU2fj for '!a transfer.archivete.am/ZajK0/urls_batch_0004_000.txt'
-
h2ibotdatechnoman: Deduplicated and queued 24996267 items. (dNky4aw1)
-
Flashfire42datechnoman I am starting to think you are just making up these naming conventions XD
-
datechnomanLol. Couldnt be bothered fixing the names. They are all .edu url dumps
-
h2ibotdatechnoman: Skipped 3213 invalid URLs: transfer.archivete.am/ZBWtR/urls_batch_0004_000.txt.bad-urls.txt (3sdLU2fj)
-
h2ibotdatechnoman: Skipped 110 very long URLs: transfer.archivete.am/HqMjQ/urls_batch_0004_000.txt.skipped.txt (3sdLU2fj)
-
h2ibotdatechnoman: Deduplicating and queuing 24996676 items. (3sdLU2fj)
-
datechnoman
-
h2ibotdatechnoman: Registering tXMm0bAm for '!a transfer.archivete.am/B3kMC/urls_batch_0004_001.txt'
-
h2ibotdatechnoman: Skipped 3676 invalid URLs: transfer.archivete.am/d2r9j/urls_batch_0004_001.txt.bad-urls.txt (tXMm0bAm)
-
h2ibotdatechnoman: Skipped 93 very long URLs: transfer.archivete.am/x87Ye/urls_batch_0004_001.txt.skipped.txt (tXMm0bAm)
-
h2ibotdatechnoman: Deduplicating and queuing 24996231 items. (tXMm0bAm)
-
h2ibotdatechnoman: Deduplicated and queued 24996678 items. (3sdLU2fj)
-
h2ibotdatechnoman: Deduplicated and queued 24996238 items. (tXMm0bAm)
-
datechnoman
-
h2ibotdatechnoman: Registering OsfvvfFA for '!a transfer.archivete.am/106IEs/urls_batch_0004_002.txt'
-
h2ibotdatechnoman: Something went wrong. (OsfvvfFA)
-
datechnoman!status
-
h2ibotdatechnoman: Jobs running: 0, jobs waiting for a slot: 0.
-
datechnoman
-
h2ibotdatechnoman: Registering JpujvJBu for '!a transfer.archivete.am/106IEs/urls_batch_0004_002.txt'
-
h2ibotdatechnoman: Skipped 1785 invalid URLs: transfer.archivete.am/h7glF/urls_batch_0004_002.txt.bad-urls.txt (JpujvJBu)
-
h2ibotdatechnoman: Skipped 83 very long URLs: transfer.archivete.am/16cz3t/urls_batch_0004_002.txt.skipped.txt (JpujvJBu)
-
h2ibotdatechnoman: Deduplicating and queuing 12458319 items. (JpujvJBu)
-
h2ibotdatechnoman: Deduplicated and queued 12458322 items. (JpujvJBu)
-
datechnoman
-
h2ibotdatechnoman: Registering mfcq2ffv for '!a transfer.archivete.am/bfER9/pdf_urls_batch_2.txt'
-
h2ibotdatechnoman: Skipped 7389 invalid URLs: transfer.archivete.am/yHHnk/pdf_urls_batch_2.txt.bad-urls.txt (mfcq2ffv)
-
h2ibotdatechnoman: Skipped 306 very long URLs: transfer.archivete.am/10OkrE/pdf_urls_batch_2.txt.skipped.txt (mfcq2ffv)
-
h2ibotdatechnoman: Deduplicating and queuing 24992305 items. (mfcq2ffv)
-
h2ibotdatechnoman: Deduplicated and queued 24992312 items. (mfcq2ffv)
-
datechnoman
-
h2ibotdatechnoman: Registering UOAwLCzb for '!a transfer.archivete.am/QBeph/pdf_urls_batch_3.txt'
-
h2ibotdatechnoman: Skipped 2018 invalid URLs: transfer.archivete.am/inMwy/pdf_urls_batch_3.txt.bad-urls.txt (UOAwLCzb)
-
h2ibotdatechnoman: Skipped 194 very long URLs: transfer.archivete.am/SiR92/pdf_urls_batch_3.txt.skipped.txt (UOAwLCzb)
-
h2ibotdatechnoman: Deduplicating and queuing 24997788 items. (UOAwLCzb)
-
h2ibotdatechnoman: Deduplicated and queued 24997788 items. (UOAwLCzb)
-
datechnoman
-
h2ibotdatechnoman: Registering gtW7ugBY for '!a transfer.archivete.am/uRJWQ/pdf_urls_batch_4.txt'
-
h2ibotdatechnoman: Skipped 5075 invalid URLs: transfer.archivete.am/Mul4B/pdf_urls_batch_4.txt.bad-urls.txt (gtW7ugBY)
-
h2ibotdatechnoman: Skipped 431 very long URLs: transfer.archivete.am/5IuL8/pdf_urls_batch_4.txt.skipped.txt (gtW7ugBY)
-
h2ibotdatechnoman: Deduplicating and queuing 24994494 items. (gtW7ugBY)
-
AK🤔My workers seem very unhappy with rsync right now
-
AKrsync: [sender] read error: Connection reset by peer (104)
-
AKrsync error: error in socket IO (code 10) at io.c(806) [sender=3.2.7]
-
h2ibotdatechnoman: Deduplicated and queued 24994495 items. (gtW7ugBY)
-
datechnomanAK - We are all facing the same issue. Been happening for approximately 36 hours :(
-
datechnoman
-
h2ibotdatechnoman: Registering Oi8pGKmM for '!a transfer.archivete.am/DKEMb/pdf_urls_batch_5.txt'
-
AKDamn, at least I know it's not just me haha
-
h2ibotdatechnoman: Skipped 1704 invalid URLs: transfer.archivete.am/pLKi9/pdf_urls_batch_5.txt.bad-urls.txt (Oi8pGKmM)
-
h2ibotdatechnoman: Skipped 15 very long URLs: transfer.archivete.am/NOUHw/pdf_urls_batch_5.txt.skipped.txt (Oi8pGKmM)
-
h2ibotdatechnoman: Deduplicating and queuing 24998281 items. (Oi8pGKmM)
-
datechnoman
-
h2ibotdatechnoman: Registering hCHYZbsW for '!a transfer.archivete.am/lcV83/pdf_urls_batch_6.txt'
-
h2ibotdatechnoman: Deduplicated and queued 24998282 items. (Oi8pGKmM)
-
h2ibotdatechnoman: Skipped 50 invalid URLs: transfer.archivete.am/Nzkoo/pdf_urls_batch_6.txt.bad-urls.txt (hCHYZbsW)
-
h2ibotdatechnoman: Skipped 70 very long URLs: transfer.archivete.am/wT0Pi/pdf_urls_batch_6.txt.skipped.txt (hCHYZbsW)
-
h2ibotdatechnoman: Deduplicating and queuing 24999880 items. (hCHYZbsW)
-
datechnoman
-
h2ibotdatechnoman: Registering WEJa5oB9 for '!a transfer.archivete.am/9Jk7s/pdf_urls_batch_7.txt'
-
h2ibotdatechnoman: Deduplicated and queued 24999889 items. (hCHYZbsW)
-
h2ibotdatechnoman: Skipped 208 invalid URLs: transfer.archivete.am/Z3nAz/pdf_urls_batch_7.txt.bad-urls.txt (WEJa5oB9)
-
h2ibotdatechnoman: Skipped 126 very long URLs: transfer.archivete.am/VOr7b/pdf_urls_batch_7.txt.skipped.txt (WEJa5oB9)
-
h2ibotdatechnoman: Deduplicating and queuing 24999666 items. (WEJa5oB9)
-
h2ibotdatechnoman: Deduplicated and queued 24999668 items. (WEJa5oB9)
-
datechnoman
-
h2ibotdatechnoman: Registering nEwJPUok for '!a transfer.archivete.am/oGx7w/pdf_urls_batch_8.txt'
-
h2ibotdatechnoman: Skipped 100 invalid URLs: transfer.archivete.am/HeUHd/pdf_urls_batch_8.txt.bad-urls.txt (nEwJPUok)
-
h2ibotdatechnoman: Fixed 2 unprintable URLs: transfer.archivete.am/EnmrY/pdf_urls_batch_8.txt.not-printable.txt (nEwJPUok)
-
h2ibotdatechnoman: Skipped 2 very long URLs: transfer.archivete.am/yYQxw/pdf_urls_batch_8.txt.skipped.txt (nEwJPUok)
-
h2ibotdatechnoman: Deduplicating and queuing 2566467 items. (nEwJPUok)
-
h2ibotdatechnoman: Deduplicated and queued 2566467 items. (nEwJPUok)
-
kiskaAK: Yeah seems about right server8.kiska.pw/uploads/e2b1e7b3bbfb8af0/image.png
-
kiskaMoving average looks fine :D
-
datechnomanGood Morning/Afternoon rewby rewby|backup. Whenever you have a spare moment, could I please request for a health check of optane10? Its moving data along but seems to be bottlenecking where we previously have not had issues. I am wondering if there are some stuck packer or upload containers for the #// project causing a performance decrease. We are
-
datechnomanseeing constant "rsync: [sender] write error: Broken pipe (32)" and "rsync error: error in socket IO (code 10) at io.c(806) [sender=3.2.7]" where the target appears to be resetting/breaking the connections so workers are constantly reuploading the same items over and over. It has been occurring for the past 48 hours and hasn't seemed to clear up on
-
datechnomanits own. Here is a screenshot showing the last 3 days and where traffic started going from a normal constant throughput to constantly spiking. transfer.archivete.am/dH5UU/worker_traffic_graph.jpg
-
eggdropinline (for browser viewing): transfer.archivete.am/inline/dH5UU/worker_traffic_graph.jpg
-
datechnomanI hope this information helps!
-
datechnomanI assume your incoming traffic would be very spiky :O
-
datechnomanWorkers are showing mostly max connections -1 but there are some -500 every now and then
-
datechnomanSeem to be down to 12TB a day atm. Have previously run up to 30TB a day
-
datechnomanI'll leave it in your capable hands :D Thanks in advance!
-
rewby|backupI'll give it a look in the morning