-
h2ibot
JAABot edited List of websites excluded from the Wayback Machine (+0):
wiki.archiveteam.org/?diff=51830&oldid=51827
-
h2ibot
Bear edited List of websites excluded from the Wayback Machine (+72, Merged my duplicate with existing "www." entry…):
wiki.archiveteam.org/?diff=51831&oldid=51830
-
h2ibot
Bear edited List of websites excluded from the Wayback Machine (+191, Deno, Market-Ticker (not excluded as of 2019)):
wiki.archiveteam.org/?diff=51832&oldid=51831
-
h2ibot
Bear edited List of websites excluded from the Wayback Machine/Former exclusions (+137, unibo.it - excluded in 2014):
wiki.archiveteam.org/?diff=51833&oldid=51829
-
h2ibot
JAABot edited List of websites excluded from the Wayback Machine (+0):
wiki.archiveteam.org/?diff=51834&oldid=51832
-
tzt
did we grab telam.com.ar before it was shut down?
-
fireonlive
-
fireonlive
idk why, but this popped up in my search alerts. so have a look if you haven't seen it already maybe?
youtube-nocookie.com/embed/fYI_XtT7sgQ?rel=0
-
fireonlive
"DEF CON 19 - Jason Scott - Archive Team: A Distributed Preservation of Service Attack"
-
arkiver
JAA: would it be possible to get those complete notes online?
-
arkiver
it would be good to have a possibility again to archive onions
-
arkiver
it would not be used for other stuff thouhg
-
arkiver
though*
-
JAA
Oh, I forgor to actually say it, yeah, I'll try to get those notes into the repo soon.
-
arkiver
was this where we previously put the AB Tor WARCs in?
archive.org/details/archiveteam_tor
-
JAA
Yeah
-
arkiver
yeah we can push them to there again
-
arkiver
-
fireonlive
ah! i was just about to ask if onions would be visible in the WBM
-
fireonlive
:)
-
arkiver
fireonlive: absolutely
-
fireonlive
awesome
-
arkiver
-
» pabs wonders if we should let the Tor community know about this
-
fireonlive
:)
-
arkiver
pabs: i think we can treat it as we treat normal AB
-
arkiver
pabs: or do you mean the wayback machine viewable stuff?
-
arkiver
-
pabs
I mean that AB will be saving onions again
-
pabs
they might have some ideas of things to save
-
fireonlive
hmm, I2P would be interesting one day perhaps
-
fireonlive
oh, i was thinking of ipfs
-
fireonlive
two different things!
-
arkiver
fireonlive: i never looked into I2P much
-
arkiver
oh IPFS
-
» pabs wonders if IA will start sharing things over IPFS :)
-
arkiver
yeah maybe
-
arkiver
pabs: no idea, i didn't hear much about it
-
fireonlive
-
arkiver
pabs: yeah on the tor community. at the same time, the general "internet community" would have ideas about what to archive on the internet with AB, and we're not explicitly advertising AB to the world
-
arkiver
i think we could treat TorAB the same
-
pabs
true
-
arkiver
if it makes sense to mention somewhere, we could go "hey, you might want to let AT know, they have a Tor archiving operation running"
-
fireonlive
could an existing pipeline be used for tor as well or would it need a new dedi i wonder
-
fireonlive
pipeline's server* i guess
-
arkiver
probably best answered by JAA ^
-
arkiver
but i wonder if we can just disable the Tor one by default, and only enable it when explicitly given as pipeline to run on?
-
JAA
I'll have to look for the notes from the 2018 iteration. I think that did some iptables magic. Which *could* be made to work on an existing machine in principle, I guess.
-
magmaus3
for example as an parameter?
-
JAA
arkiver: Yes, that's how it worked in 2018. The Tor pipelines only processed jobs queued with `--pipeline tor`.
-
magmaus3
ah
-
arkiver
that could work
-
arkiver
maybe with a special !a-tor or so command for h2ibot which subsequently posts a `--pipeline tor` one
-
pabs
-p tor isn't too much to type that !a-tor seems needed?
-
JAA
Yeah, `--pipeline` = `-p`, so that's short enough, I'd say.
-
arkiver
alright
-
arkiver
SootBector: we'll likely look now first into getting a tor pipeline on one of the existing machine (or, well, JAA is going to check that)
-
pabs
arkiver JAA - another thing to consider is Tor exit node IP reputation varies quite a bit, so being able to rotate exit nodes or choose which country to exit from would be useful
-
pabs
the country stuff would be really useful for geoblocked sites that AB curren'tly can't access
-
JAA
pabs: IIRC, this kind of stuff is a massive pain to do programmatically. Last time I played with that, the only and poor solution to 'I need a new circuit for X' was 'restart Tor entirely'.
-
pabs
I think there are things to do that now. selektor for eg is a GUI. python3-stem for Python
-
fireonlive
hmm there's newnym, but it's not granular
-
JAA
> Stem is mostly unmaintained.
-
JAA
Sounds fun.
-
fireonlive
-
pabs
-
pabs
oh, only v2 onions
-
JAA
Oh, you can't change the circuit for just a single site at all? TIL.
-
JAA
So we'd need a separate Tor instance per job to be able to control the circuit at that granularity at least, I guess. Ugh...
-
fireonlive
not up on the new stuff but i don't think so
-
JAA
Yeah, I don't see that happening soon.
-
fireonlive
yeah indeed..
-
arkiver
pabs: does that matter much for .onion addresses?
-
pabs
no, only for clearnet stuff
-
pabs
exit nodes aren't involved in onion services
-
katia
needs kubernetes
-
inti83
hi all, how are you? I've been in and out here in regards to Argentina archive
-
inti83
So yesterday the main public news provider and archive Telam was taken down, most of it was archived by AB and others luckily
-
inti83
And now they are talking of taking down cine.ar, which has most nationally produced cinema. This site is behind Cloudflare so wasn't archived. It was spoken about here that you may have contacts in Buenos Aires to host a local ArchiveBot... Did anythingcome of it?
-
inti83
I think JAA mentioned they had contacts in
cabase.org.ar
-
inti83
We have some automation in place to use as a way to download the content, but we don't have the space to do it
-
inti83
Just to reiterate, the site we are talking about is cine.ar - it has most if not all of the INCAA produced cinema, another organism they are taking down
-
pabs
looks like you have to register to watch anything?
-
inti83
anyone can register
-
inti83
Anyone in Argentina at least
-
inti83
I understand at least grab-site and wpull have an authn option
-
SootBector
arkiver: JAA - great. ping me in any case if the install notes get updated - I have a server mostly sitting idle and would like to try installing just to see how things fit together.
-
SootBector
re: selecting Tor exits/countries, an approach might be to configure a list in torrc each on its own SocksPort then have AB just change which port it connects to.
-
SootBector
sending a different socks username:password (which can be anything) will give you a fresh circuit but not necessarily a different exit
-
SootBector
manpage seems to say you can't select ExitNodes [countrycode] per-SocksPort, but a tor process could be spun up on demand with that option set on the commandline
-
SootBector
tor --SocksPort 9090 --ExitNodes {ES} yes that works
-
SootBector
-
arkiver
queuing bot will now note when it is started and stopped
-
inti83
I have to go now will check logs for any updates, thanks!
-
arkiver
there was also a big that caused slots to be hardly found, effectively making one job run at a time, that is fixed now
-
fireonlive
dl.fireon.live/irc/d6790cdce49fe76f/image.png < “Hidden Birds” site/wiki (from the “TeraLeak” (ugh)) to go down
-
fireonlive
oh and forum. not sure at the moment where they reside…
-
arkiver
forum is discord here? :P
-
fireonlive
-
fireonlive
:P
-
arkiver
some day people will discover we archived a ton of imgur, label it a leak, put some stupid name on it, and advertise it as something new they found
-
arkiver
fireonlive: they did make the front page of
hiddenbirds.ultra0.ar pretty nice thouhg
-
fireonlive
ugh yeah xD
-
fireonlive
indeed!
-
fireonlive
*someone finds our wiki* “holy crap, leak motherload”
-
JAA
inti83: That wasn't me.
-
JAA
SootBector: Good info, thanks!
-
SootBector
I'll investigate if changing ExitNodes is possible via commands to ControlPort
-
JAA
FWIW, integrating that into AB still sounds painful and unlikely to happen soon, but yeah, would be nice.
-
SootBector
also found a feature where you append an exit's fingerprint and .exit to the URL's domain to set it on-the-fly. not yet sure if you can enable that for *
-
JAA
inti83: I can access cine.ar fine, but it looks like accessing the content may require registration.
-
fireonlive
-
eggdrop
-
fireonlive
-
fireonlive
8.9k videos on their main channel wow lol
-
fireonlive
threw the main channel to DTT but will have to return to it later
-
mikolaj|m
pabs: might be worth archiving all this, some mailing lists (such as agda and agda-dev) are public
lists.chalmers.se/mailman/listinfo
-
lunik1
+ 3.1k on Achievement Hunter, + 4k on Let's Play and there's more besides…
-
fireonlive
lunik1: if you have a list of their youtube channels please do let #down-the-tube know
-
fireonlive
i haven't been able to return to looking up who/what they are quite yet
-
lunik1
will do. won't be exhaustive but I'll send what I remember
-
lunik1
what's the preferred pastebin to use?
-
katia
-
fireonlive
-
fireonlive
-e is "explain" in this instance
-
fireonlive
(anything can be added if it falls within the "scope" of #down-the-tube, explain is to say why it does:
wiki.archiveteam.org/index.php/YouTube#Scope) but let's take further talk there