-
datechnoman
Fusl arkiver Request to blacklist the following domain returning thoughts of Server returned 0 (CONERROR)
-
datechnoman
-
datechnoman
-
datechnoman
Got a few million dead link hits for those domains *throws fist up*
-
arkiver
looking into it
-
arkiver
ah!
-
arkiver
yeah we're not going to block those domains but some pattern they have
-
arkiver
thanks datechnoman )
-
arkiver
:)
-
datechnoman
Thanks arkiver! All good. Whatever makes it more efficient :)
-
arkiver
yeah in general I try to avoid filtering an entire domain
-
datechnoman
Also on a side note how did you and JAA_ go with the wget tweak to reduce cpu consumption with an increase to the ram buffer?
-
arkiver
there is usually some reason the URLs of that domain are problematic, and that should be fixed (so that it is fixed for all other domains it could appear on as well)
-
arkiver
nothing yet
-
arkiver
i am not home right now... and forgot to bring the the Wget-AT with FTP updates
-
arkiver
in a week I'll be home again, push the FTP code out and after that we'll get an update in for increased buffers
-
datechnoman
Excellent thanks for the fix then!
-
datechnoman
Thanks for the update on the code updates also. Will standby till next week :)
-
arkiver
let's see how well it works!
-
arkiver
might ask you to try out some versions and see if it makes any difference for your setup
-
arkiver
(either positive or negative difference)
-
arkiver
imagine CPU not being the bottleneck anymore...
-
datechnoman
Well my dedicated servers have lots of free RAM so that is great :D
-
datechnoman
CPU has been the killer for quite a long time
-
arkiver
bandwidth is never the problem?
-
datechnoman
nope not even close. Averaging 200mbps/300mbps in/out at peak which is nothing for dedicated 1Gbps links.
-
arkiver
nice
-
datechnoman
CPU just sits at 95%+ the whole time
-
arkiver
yeah let's focus on CPU usage improvements over the coming time
-
arkiver
I think that'll be the next big set of updates for Wget-AT (after FTP)
-
datechnoman
Bandwidth and RAM is a hell of a lot cheaper with plenty of overhead
-
datechnoman
Sounds like a plan. That would be great. Would make it much more efficient
-
arkiver
other bottlenecks might be in compression (zstd or gz) and hashing (sha1), so we'll see
-
arkiver
but we're not reading the files multiple times with 8K buffers, if we take those out it should already be a big improvement
-
datechnoman
Baby steps is fine. Dont want to make the code worse or less efficient
-
datechnoman
no point going backwards!
-
arkiver
yes!
-
arkiver
we won't be going backwards :P
-
arkiver
at some point will want to look into HTTP2 stuff
-
datechnoman
Sounds like a plan. How does the code handle HTTP2 as it stands? Can it still process it or just skip over it?
-
arkiver
not supported
-
arkiver
at all
-
datechnoman
roger. Just threw in some cloud compute to chew through the backlog
-
datechnoman
Not that we have had much backlog for the past few weeks
-
datechnoman
been quiet!
-
arkiver
been stable :P
-
datechnoman
joys of dedicated servers on the project to keep a baseline
-
datechnoman
Mind you we were doing bulk imports a month or so ago with massive lists and stuff of URL's
-
datechnoman
but you cant complain about things being stable!
-
arkiver
we might queue some more large lists again some time soon
-
arkiver
will let you all know when that is queued
-
datechnoman
Cheers ping me anytime
-
datechnoman
Processing 100,000 URL's a minute at the moment lol
-
datechnoman
With 4 servers
-
arkiver
onlyfiles coming up in a little bit too
-
datechnoman
Keen! I will be online for another 4 or so hours. So if I can atleast pull the docker container and get it prepped it will start processing when the tracker starts allowing ID's
-
datechnoman
Plently of spare bandwidth atleast. Will just come down to having a target
-
arkiver
right the container
-
arkiver
pinged in -bs for building that
-
datechnoman
Cheers :)
-
datechnoman
1 day 5 hours. Hopefully enough time!
-
arkiver
should be fine if they can hold up
-
datechnoman
Always the challenge!