-
PedrossoDoes running at high concurrency lead to many of the same domain being archived at once, or does each concurrent archive have a guaranteed different domain?
-
imerPedrosso: tasks are only handed out to one worker at a time (except for retries of course, but those have a wait inbetween)
-
imeralso applies for the concurrency setting
-
imereach task being one url to archive
-
Pedrossoso high concurrency means archiving one url faster?
-
imermultiple urls at once
-
Pedrossogood
-
JAAURLTeam is the one where the tracker rate-limits you per target site.
-
vokunal|mNot sure if we want them, but here's everything from the past few archives in Disgard transfer.archivete.am/9y8tN/urls-discard-nov-14-23.txt
-
vokunal|m~247K urls
-
vokunal|mDiscard*