00:15:58 datechnoman: on that question - "it's complicated" 00:16:25 the queue is not ordered, so in the queue they'll be 'somewhere' 00:16:33 not beginning not end per se 00:16:57 and, if the queue if very large we offload batches of items to external storage, to be loaded in again when the queue becomes smaller 00:17:15 which items are chosen to be offloaded... happen by chance 00:17:40 All good. By the sounds its more of a bucket of items 00:17:48 Very interesting 00:19:04 Thanks for sharing 00:21:21 yeah, exactly! 04:43:19 Spinning down till those pesky spam domains are cleared out 15:44:53 datechnoman: yeah 17:51:21 !a https://transfer.archivete.am/R4754/goo-gl.2023-01-01-04-17-02.txt 17:51:24 datechnoman: & 17:51:27 ^* 17:56:56 arkiver: Skipped 28945 bad URLs: https://transfer.archivete.am/v8m1u/goo-gl.2023-01-01-04-17-02.txt.bad-urls.txt 17:57:30 arkiver: Deduplicating and queuing 9034161 items. 18:00:44 arkiver: Deduplicated and queued 9034161 items.