-
datechnoman
We seem to be back on track. Been running smooth the last 12 hours
-
datechnoman
fireonlive i have mentioned the same thing before actually. I have a bunch of compute I could make us of at home for it. Just won't run this project on my home ip for obvious reasons but bulk processing pdf's I could do for sure. Would save me a bunch on cloud compute costs atm
-
fireonlive
datechnoman: i'm in kinda the same situation as well!
-
fireonlive
i'm sure others could lend compute as well but don't nesc want to run urls proper
-
fireonlive
great minds etc :3
-
datechnoman
I believe we can have a different tracker but still feed the watch into the #// Collection so it's all together. Would be up to arkiver :)
-
datechnoman
I've got a dual CPU server at home that would happily chew through them
-
fireonlive
although I suppose if you didn't want those hosts 'reaching out randomly to the web' it would have to get the PDFs from IA...
-
fireonlive
🤔
-
fireonlive
fetch previously uploaded $thing from $safelocation; process it; feed backlinks to urls; repeat
-
TheTechRobo
well it could work if the project waits until the PDFs are available from the WBM
-
TheTechRobo
though the logistics of that might be difficult to code
-
datechnoman
It would be a matter of pdfs being discovered in the #// project which are then queues to a pdf tracker which the workers download the pdf (from all over the internet), discover the urls and then queue them back in the #// tracker
-
arkiver
fireonlive: what do you mean with "more people participating"? because of possible problematic stuff going through #// ?
-
arkiver
if we find a document, we would probably just send the URL of that document to the alternative project to be processed, meaning there could still be any random stuff in there that people might get in trouble for
-
arkiver
ah i see this was already discussed ater
-
fireonlive
arkiver: indeed, thinking abuse reports, or some people's stories here of three-letter-agencies/police showing up asking wtf they're accessing, that kinda thing
-
michaelblob
last time the fbi bought me a hot chocolate, maybe this time they'll splurge and buy me a coffee?
-
arkiver
hot chocolate sounds more likely splurging to me than a coffee :P
-
arkiver
fireonlive: yeah
-
shodan
Is there an effort to archive the content of sketchup's 3d warehouse before it disappears for good behind a paywall ? Same for extensions, especially extensions with Sketchup 8, the last free for all purposes of sketchup ?
-
JAA
That wouldn't be appropriate for this project, but please bring it up in #archiveteam-bs instead.
-
shodan
ok thanks