04:51:47 <datechnoman> Fusl arkiver Request to blacklist the following domain returning thoughts of Server returned 0 (CONERROR)
04:51:49 <datechnoman> "https://olesadebonesvalls.eadministracio.cat/*"
04:54:59 <datechnoman> "https://www.inakiecheverria.com/*"
04:59:34 <datechnoman> Got a few million dead link hits for those domains *throws fist up*
05:35:46 <arkiver> looking into it
05:36:08 <arkiver> ah!
05:36:14 <arkiver> yeah we're not going to block those domains but some pattern they have
05:39:28 <arkiver> thanks datechnoman )
05:39:30 <arkiver> :)
06:45:43 <datechnoman> Thanks arkiver! All good. Whatever makes it more efficient :)
06:46:08 <arkiver> yeah in general I try to avoid filtering an entire domain
06:46:35 <datechnoman> Also on a side note how did you and JAA_ go with the wget tweak to reduce cpu consumption with an increase to the ram buffer?
06:46:38 <arkiver> there is usually some reason the URLs of that domain are problematic, and that should be fixed (so that it is fixed for all other domains it could appear on as well)
06:46:48 <arkiver> nothing yet
06:47:15 <arkiver> i am not home right now... and forgot to bring the the Wget-AT with FTP updates
06:47:37 <arkiver> in a week I'll be home again, push the FTP code out and after that we'll get an update in for increased buffers
06:54:32 <datechnoman> Excellent thanks for the fix then!
06:54:56 <datechnoman> Thanks for the update on the code updates also. Will standby till next week :)
06:57:08 <arkiver> let's see how well it works!
06:57:24 <arkiver> might ask you to try out some versions and see if it makes any difference for your setup
06:57:31 <arkiver> (either positive or negative difference)
06:57:49 <arkiver> imagine CPU not being the bottleneck anymore...
07:03:07 <datechnoman> Well my dedicated servers have lots of free RAM so that is great :D
07:03:21 <datechnoman> CPU has been the killer for quite a long time
07:04:39 <arkiver> bandwidth is never the problem?
07:06:46 <datechnoman> nope not even close. Averaging 200mbps/300mbps in/out at peak which is nothing for dedicated 1Gbps links.
07:06:54 <arkiver> nice
07:06:55 <datechnoman> CPU just sits at 95%+ the whole time
07:07:07 <arkiver> yeah let's focus on CPU usage improvements over the coming time
07:07:19 <arkiver> I think that'll be the next big set of updates for Wget-AT (after FTP)
07:07:31 <datechnoman> Bandwidth and RAM is a hell of a lot cheaper with plenty of overhead
07:07:40 <datechnoman> Sounds like a plan. That would be great. Would make it much more efficient
07:08:24 <arkiver> other bottlenecks might be in compression (zstd or gz) and hashing (sha1), so we'll see
07:08:50 <arkiver> but we're not reading the files multiple times with 8K buffers, if we take those out it should already be a big improvement
07:09:31 <datechnoman> Baby steps is fine. Dont want to make the code worse or less efficient
07:09:37 <datechnoman> no point going backwards!
07:11:57 <arkiver> yes!
07:12:02 <arkiver> we won't be going backwards :P
07:12:13 <arkiver> at some point will want to look into HTTP2 stuff
07:14:09 <datechnoman> Sounds like a plan. How does the code handle HTTP2 as it stands? Can it still process it or just skip over it?
07:15:26 <arkiver> not supported
07:15:30 <arkiver> at all
07:19:11 <datechnoman> roger. Just threw in some cloud compute to chew through the backlog
07:19:18 <datechnoman> Not that we have had much backlog for the past few weeks
07:19:30 <datechnoman> been quiet!
07:20:17 <arkiver> been stable :P
07:21:42 <datechnoman> joys of dedicated servers on the project to keep a baseline
07:21:59 <datechnoman> Mind you we were doing bulk imports a month or so ago with massive lists and stuff of URL's
07:22:13 <datechnoman> but you cant complain about things being stable!
07:22:28 <arkiver> we might queue some more large lists again some time soon
07:22:51 <arkiver> will let you all know when that is queued
07:23:32 <datechnoman> Cheers ping me anytime
07:23:47 <datechnoman> Processing 100,000 URL's a minute at the moment lol
07:23:57 <datechnoman> With 4 servers
07:24:10 <arkiver> onlyfiles coming up in a little bit too
07:25:43 <datechnoman> Keen! I will be online for another 4 or so hours. So if I can atleast pull the docker container and get it prepped it will start processing when the tracker starts allowing ID's
07:26:06 <datechnoman> Plently of spare bandwidth atleast. Will just come down to having a target
07:26:14 <arkiver> right the container
07:26:47 <arkiver> pinged in -bs for building that
07:27:57 <datechnoman> Cheers :)
07:28:17 <datechnoman> 1 day 5 hours. Hopefully enough time!
07:29:45 <arkiver> should be fine if they can hold up
07:29:58 <datechnoman> Always the challenge!