02:06:06 Hi, somebody please help me, how to get access to download this file https://archive.org/download/archiveteam_urls_20240717143359_16efb7e3? 02:06:39 arkiver: ^ 02:06:52 V2: It might help to state why you need access to the file. 02:08:33 got it, I am building a URL shortening service, so I really need database of goo.gl/* shortlinks, its 14gb, and listed here https://archive.org/download/archiveteam_urls_20240717143359_16efb7e3, and I see its Status as restricted 02:09:58 Uh 02:10:34 So first, please don't build another URL shortener that will die within a few years. And second, where did you find that that file contains a goo.gl database? 02:10:41 JAA lot of users facing challenge to find its target links as it may contain useful information, so they are all getting shutdown, so that dababase will find them 02:11:30 no, its not new URL shortener, but a small tool to help find its target links when existing links break.. 02:12:24 I see. We intend to archive all goo.gl URLs into the Wayback Machine (project channel #urlteamwasright) before they die, so that tool will exist automatically. :-) 02:12:40 I was browsing lot of links, but ended up there (https://archive.org/download/archiveteam_urls_20240717143359_16efb7e3) as it contains list of all goo.gl links, if not please advise me where it is available 02:13:18 I doubt that file contains any significant number of goo.gl URLs. Certainly not all of them. Our project for that hasn't started yet, and it wouldn't happen here either. 02:14:03 That's why I'm wondering why you ended up at that file to begin with. 02:14:17 oh ok, then should i ask in #urlteamwasright? I have been searching for weeks, pls help me give direct access to download it., any number of links also fine, as it definitely helps some users of the community 02:14:30 We do not have such a database yet. 02:15:03 all I need is just goo.gl's shortlink (or hash), and its target links, just two fields is enough 02:15:10 #urlteamwasright is where that project will be coordinated in the next months or so. 02:15:17 Yes, we would like to have that, too. 02:16:46 Here (https://tracker.archiveteam.org:1338/status), I see goo-gl status with almost 7B links found, it should have been stored somewhere.. pls help me find or give some contacts so that I can followup 02:19:26 And now we've come full circle and are back at URLTeam. Yes, that data exists, and it's incomplete. 02:19:39 See https://wiki.archiveteam.org/index.php/URLTeam#Archives for details on that data. 02:49:54 JAA Thanks, but its Publication date 2013-07, quite old, is there a latest version, at least few months old archive? 02:50:53 JAA, also its topic says "urlteam, 4url.cc, arseh.at, ff.im, kl.am, links.sharedby.co, litturl.com, surl.ws, tny.im, tr.im, t.co, ur1.ca,", does it definitely contains the 02:50:53 "goo.gl" shortlinks also? 03:24:15 JAA please clarify to my above two items 03:26:05 JAA that torrent address "torrent:urn:sha1:4cf5896b507f3ca6f50819a2788e99dfa5bcb58b" didn't work, tried with several cilents, nothing downloaded 08:00:00 !a https://transfer.archivete.am/GSgmc/urls_batch_0002_002.txt 08:00:02 datechnoman: Registering GuB0k6IH for '!a https://transfer.archivete.am/GSgmc/urls_batch_0002_002.txt' 08:30:15 datechnoman: Skipped 4702 invalid URLs: https://transfer.archivete.am/IZGfU/urls_batch_0002_002.txt.bad-urls.txt (GuB0k6IH) 08:30:17 datechnoman: Skipped 78 very long URLs: https://transfer.archivete.am/LHf68/urls_batch_0002_002.txt.skipped.txt (GuB0k6IH) 08:30:18 datechnoman: Deduplicating and queuing 19270666 items. (GuB0k6IH) 08:52:53 !a https://transfer.archivete.am/fsbxV/urls_batch_0003_000.txt 08:52:53 datechnoman: Registering tfzhZxEB for '!a https://transfer.archivete.am/fsbxV/urls_batch_0003_000.txt' 08:53:19 datechnoman: Deduplicated and queued 19270906 items. (GuB0k6IH) 10:42:15 datechnoman: Skipped 4561 invalid URLs: https://transfer.archivete.am/BV0M3/urls_batch_0003_000.txt.bad-urls.txt (tfzhZxEB) 10:42:20 datechnoman: Skipped 92 very long URLs: https://transfer.archivete.am/Kd4jv/urls_batch_0003_000.txt.skipped.txt (tfzhZxEB) 10:42:21 datechnoman: Deduplicating and queuing 24995346 items. (tfzhZxEB) 10:51:09 !a https://transfer.archivete.am/pNNUo/urls_batch_0003_001.txt 10:51:11 datechnoman: Registering H3eIJevB for '!a https://transfer.archivete.am/pNNUo/urls_batch_0003_001.txt' 11:23:47 datechnoman: Deduplicated and queued 24995621 items. (tfzhZxEB) 12:04:35 !a https://transfer.archivete.am/PoDN1/urls_batch_0003_002.txt 12:04:35 datechnoman: Registering YWdsTEyO for '!a https://transfer.archivete.am/PoDN1/urls_batch_0003_002.txt' 12:17:17 !status 12:17:18 datechnoman: Jobs running: 2, jobs waiting for a slot: 0. 12:17:29 !a https://transfer.archivete.am/APKKL/urls_batch_0004_000.txt 12:17:30 datechnoman: Registering Qff4OMGk for '!a https://transfer.archivete.am/APKKL/urls_batch_0004_000.txt' 12:17:32 !status 12:17:33 datechnoman: Jobs running: 3, jobs waiting for a slot: 0. 12:37:15 datechnoman: Skipped 5683 invalid URLs: https://transfer.archivete.am/jaKbl/urls_batch_0003_001.txt.bad-urls.txt (H3eIJevB) 12:37:17 datechnoman: Skipped 97 very long URLs: https://transfer.archivete.am/iQUB5/urls_batch_0003_001.txt.skipped.txt (H3eIJevB) 12:37:18 datechnoman: Deduplicating and queuing 24994220 items. (H3eIJevB) 13:19:55 datechnoman: Deduplicated and queued 24994516 items. (H3eIJevB) 14:03:42 datechnoman: Skipped 5055 invalid URLs: https://transfer.archivete.am/iRfoH/urls_batch_0003_002.txt.bad-urls.txt (YWdsTEyO) 14:03:48 datechnoman: Skipped 74 very long URLs: https://transfer.archivete.am/Tcyu4/urls_batch_0003_002.txt.skipped.txt (YWdsTEyO) 14:03:49 datechnoman: Deduplicating and queuing 21083865 items. (YWdsTEyO) 14:23:53 datechnoman: Skipped 4765 invalid URLs: https://transfer.archivete.am/14Vyfx/urls_batch_0004_000.txt.bad-urls.txt (Qff4OMGk) 14:23:56 datechnoman: Skipped 136 very long URLs: https://transfer.archivete.am/KZeFm/urls_batch_0004_000.txt.skipped.txt (Qff4OMGk) 14:23:57 datechnoman: Deduplicating and queuing 24995098 items. (Qff4OMGk) 14:39:07 datechnoman: Deduplicated and queued 21084163 items. (YWdsTEyO) 14:54:56 datechnoman: Deduplicated and queued 24995353 items. (Qff4OMGk) 16:49:57 arkiver: any chance you could look into this PR? going on holiday so would be great to have this merged in case of restart in here! https://github.com/ArchiveTeam/urls-sources/pull/36 16:50:12 let me know if it needs any changes 23:04:00 !a https://transfer.archivete.am/v7ZaF/urls_batch_0004_001.txt 23:04:01 datechnoman: Registering Dhc3Bsi7 for '!a https://transfer.archivete.am/v7ZaF/urls_batch_0004_001.txt' 23:04:18 !a https://transfer.archivete.am/Q9ZH4/urls_batch_0004_002.txt 23:04:18 datechnoman: Registering N2VG7BEl for '!a https://transfer.archivete.am/Q9ZH4/urls_batch_0004_002.txt' 23:05:21 datechnoman: '#// is sad, so let me slap another couple dozen million URLs on top!' 23:05:48 hey, this is how I make myself feel better while it isnt running :( 23:05:53 :-P 23:05:56 datechnoman++ 23:05:57 -eggdrop- [karma] 'datechnoman' now has 22 karma! 23:06:09 Alcoholics drinks, I queue URLs 23:06:10 :P 23:06:29 I have about 1 billions in a stash and it keeps growing while we standby lol 23:06:34 1 billion* 23:06:49 Oof 23:06:52 My 16 workers are keen to attack the queue... 23:08:06 Throw them at telegram 23:08:27 By the way, we averaged about 149 million snapshots and 20 TB per day before things ground to a halt. 23:09:39 We sure did :( those were the days