04:33:33 <JAA> Oh
04:34:06 <JAA> arkiver: I noticed that https://web.archive.org/web/collections/20230000000000*/https://www.svt.se/ does not display anything from archiveteam_urls, which seemed odd. Turns out the URL is commented out as a duplicate, even though that appears to be the only entry for it.
04:34:28 <JAA> Line 100125 in 43200_wikidata_Q11033_mass-media.wikidata.txt : #REASON=DUPLICATE#random=RANDOM;keep_random=1;all=1;keep_all=1;depth=1;url=http://www.svt.se/
04:34:45 <JAA> (As of 712433bc)
04:36:31 <JAA> Hmm, or does the collection list exclude archiveteam_urls? https://web.archive.org/web/20230000000000*/https://www.svt.se/nyheter/lokalt/uppsala/ (not commented out, same file) shows plenty of snapshots, nothing in the collection view.
04:37:11 <JAA> There are in fact very regular snapshots of the homepage from here. Huh
04:38:33 <JAA> Yeah, we certainly fetch it and its links regularly, so the bug is just the collection view on the WBM, I guess.
04:38:54 <JAA> Could you update the Wikidata files in urls-sources sometime please?
13:29:27 <imer> arkiver: were the random_chars.word.de domains supposed to be filtered out or just ignored? seeing some still, for example 7k9d.wk-giesa.de qo3x.unternehmen-mut.de 87oj.initiative-pro-gd.de (those are alive), can pull more from logs if needed
13:36:35 <imer> https://transfer.archivete.am/inline/8tbcL/2023_11_12_urls_spam_domains_de.txt
13:41:28 <imer> some filtering so only domains with lots of subdomains are listed: https://transfer.archivete.am/KjXoP/2023-11-12_13-41-04.txt
15:27:58 <AK> Oh are we paused?
15:32:03 <project10> Seems so
18:50:01 <arkiver> yeah, sorry for not pinging before
18:50:08 <arkiver> paused since the queue was growing fast due to spam
18:50:19 <arkiver> (hopefully unpaused soon again)
20:27:50 <MetaWonderrat> Hallo. I recently installed the Warrior and wanted to contribute to the URLs project. I have a fresh install, but the project does not move past "GettingItemFromTracker". I already rebooted my Warrior."
20:29:04 <MetaWonderrat> Is this a known issue? Is the project paused?
20:30:59 <that_lurker> paused currently
20:31:01 <fireonlive> hey MetaWonderrat :) - yeah it's currently paused due to an issue with some spam urls but should hopefully be unpaused soon
20:32:21 <MetaWonderrat> What URLs are on the list and how can one know which are spam?
20:33:35 <fireonlive> i believe, but could be wrong, it was one of those one link leads to another on the same site and it kinda becomes a infinite growing loop
20:33:49 <fireonlive> so just have to flush those out so the queue doesn't grow to infinity
20:36:04 <MetaWonderrat> Can I see the list somewhere? As I read it most of it is public somewhere.
20:43:00 <Flashfire42> feel free to contribute to the telegram project in the mean time until urls is back
20:44:22 <MetaWonderrat> I already switched over.
20:45:25 <MetaWonderrat> At firtst I tried URLTeam2 but that keeps giving me 404 errors
20:48:23 <MetaWonderrat> Is there a guess when "soon" is, regarding the URL project?
20:50:19 <Flashfire42> No idea but telegram will keep chugging along with items. I suggest running at concurrency 2 personally for minimal bans but the bans are semi random tbh
20:51:29 <MetaWonderrat> concurrency 2 personally? bans?
20:51:53 <MetaWonderrat> I am new in the Archive warrior thing
21:11:32 <vokunal|m> Yeah, if you run at high concurrency, the site you're scraping might trigger an automatic system that makes it give back an error, so running under that is sort of important.
21:12:36 <vokunal|m> If it happens, it's usually only for a short time, though that depends on the project. Some are as short as 20 minutes, and others up to days. I'm not sure about Telegram in particular
21:12:45 <MetaWonderrat> concurrency?
21:12:58 <MetaWonderrat> requests per minute?
21:13:54 <MetaWonderrat> or is it something else?
21:19:56 <vokunal|m> it's the amount of concurrent threads of the tool it runs at once. Not a particular requests per minute, which makes it a little more tricky to accurately see how many you can run at once, but 2 is fine
21:21:40 <arkiver> i'm going to take out the extraction of URLs from special interest pages
21:21:44 <arkiver> we're not able to handle it right now
21:21:51 <arkiver> but we'll try to put it back in sometimes later
21:24:01 <MetaWonderrat> Thank you. I will check back later. I got to go now. Thanks for the info.:]
23:55:58 <arkiver> update is in
23:56:06 <arkiver> we now queue to the imgur project
23:56:07 <arkiver> JAA: ^