02:48:07 <arkiver> JAA: how many URLs is that list?
02:48:11 <arkiver> may be nice to queue yes
02:52:30 <JAA> arkiver: Don't remember exactly, need to try and find the list again, but I think it was a couple hundred million.
02:52:56 <arkiver> that's a lot
02:53:16 <arkiver> if you find it can you ping me with it? so I can glance over it before we put 100 million URLs in here
02:53:26 <JAA> Yeah, of course.
02:53:54 <arkiver> if these are all good outlinks (no other 'inlinks' we don't want here), it would be interesting to see how much is deduplicated away when queuing
02:58:14 <JAA> arkiver: Found it, and it's a giant mess, that's why I never pursued it further. There are 227M lines in the file, and they should all contain a URL, but there's also a lot of noise in them and it includes something like 60M truncated URLs due to faulty description extraction at the time.
02:58:39 <JAA> Noise meaning 'text:URL' and stuff like that.
03:00:51 <arkiver> right :/
03:20:16 <JAA> We could reprocess the dislikes project data. Doesn't look like we were feeding back to here at the time.
03:21:28 <JAA> I don't feel like cleaning up those 227M lines of mess.
13:03:15 <arkiver> JAA: can you send me the 227 million lines file?
18:09:56 <JAA> arkiver: Yes, will PM you in a sec. I also have another list that I don't think was queued either and is also somewhat messy. Will send you that one as well.
18:11:12 <arkiver> thanks
20:21:18 <arkiver> !a https://transfer.archivete.am/8mBAr/youtube-urls.txt
20:31:32 <TheTechRobo> how big is this file? :P
20:31:46 <arkiver> GB
20:32:32 <h2ibot> arkiver: Skipped 30108 bad URLs: https://transfer.archivete.am/PUde1/youtube-urls.txt.bad-urls.txt
20:32:35 <h2ibot> arkiver: Skipped 46706 unprintable URLs: https://transfer.archivete.am/fnF7r/youtube-urls.txt.not-printable.txt
20:33:23 <h2ibot> arkiver: Deduplicating and queuing 17769410 items.
20:33:36 <JAA> :-)
20:34:07 <arkiver> looking later into these unprintable URLs, I think we can fix most of them
20:36:43 <arkiver> !a https://transfer.archivete.am/fnF7r/youtube-urls.txt.not-printable.txt
20:36:47 <h2ibot> arkiver: Skipped 46706 unprintable URLs: https://transfer.archivete.am/AlG1t/youtube-urls.txt.not-printable.txt.not-printable.txt
20:36:48 <h2ibot> arkiver: Deduplicating and queuing 0 items.
20:36:49 <h2ibot> arkiver: Deduplicated and queued 0 items.
20:36:52 <arkiver> right
20:39:08 <h2ibot> arkiver: Deduplicated and queued 17769410 items.
20:39:31 <arkiver> moving to :secondary
20:40:55 <arkiver> Ryz: are you happy :P ^
20:41:21 <Ryz> Yaaaaayyyyyyy~
20:41:50 <arkiver> JAA: surprisingly few duplicates
20:41:57 <arkiver> may be due to http:// often being used
20:42:07 <arkiver> http/https are treated as unique
20:44:09 <JAA> How many after dedupe?
20:44:13 <arkiver> no idea
20:44:26 <arkiver> but looking at the number after queuing not that big of a difference
20:44:39 <JAA> Interesting
21:01:08 <arkiver> !a https://transfer.archivete.am/fnF7r/youtube-urls.txt.not-printable.txt
21:01:13 <h2ibot> arkiver: Skipped 95 bad URLs: https://transfer.archivete.am/NWevV/youtube-urls.txt.not-printable.txt.bad-urls.txt
21:01:16 <h2ibot> arkiver: Fixed 46706 unprintable URLs: https://transfer.archivete.am/P1NIO/youtube-urls.txt.not-printable.txt.not-printable.txt
21:01:17 <h2ibot> arkiver: Deduplicating and queuing 46611 items.
21:01:20 <h2ibot> arkiver: Deduplicated and queued 46611 items.
21:13:44 <arkiver> JAA: looks like we have lots of shortened URLs, that may be a reason for the few duplicates
21:15:46 <JAA> Ah