!a transfer.archivete.am/CbXSO/kemono-sites-google-com-fixed.txt

michaelblob_: Invalid privileges, need one of ('@', '+').

sad

JAA: may i be blessed with the power to !a

michaelblob_: The list is small, but this project isn't well-suited for lists of single hosts. There's also zero feedback about the success. Are you sure you don't want AB instead?

oh fair enough, i can try AB

would i have to do a job for each host?

In the context of that list, you'd probably want one !a job per 'site', i.e. sites.google.com/foobar/.

got it, that seems manageable

may i be blessed with the power to !a in AB

Let's take this over there.

I spun up urls on one of my laptops here at the house... "Why did it default to the max of 64 workers, and why am I using no network and no CPU?"

I should've known something was off when the default traffic interface was "nordlynx"...

!a transfer.archivete.am/NFlCU/sciencedirect_urls.txt

datechnoman: Registering mA3WSx29 for '!a transfer.archivete.am/NFlCU/sciencedirect_urls.txt'

datechnoman: Deduplicating and queuing 15 items. (mA3WSx29)

datechnoman: Deduplicated and queued 15 items. (mA3WSx29)

nice speed :)

arkiver: we made some good progress here

That said I think I might be saturating my line with concurrent connections

I have 32 workers with 5 concurrency running on my home connection, and I'm used to seeing my CPU being thrashed but I'm actually observing loads under 10 and very low core utilization

load average: 115.89, 113.19, 112.29 looking about normal :D

fourone$:

load average: 36.39, 35.41, 35.96

Ubuntu-2204-jammy-amd64-base$:

load average: 37.22, 34.79, 32.96

these are both running 20 workers with 5 concurrency each

nice loads :)

arkiver: I'd probably be able to do more if the D-1521 that fourone has isn't a flaming piece of intel garbage

The hetzner dedi is running circles around it with an i5-7700 albeit NVMe and not spinning rust like fourone

well at this speed we'll already be up to date very soon

Just in time for the first of the month sitemaps rush ;)

Oh hey I passed Fusl on the tracker. There's something I didn't think I'd ever be able to say lol

Congrates :)

Nice, I'm close to climing two more spots soon as well 👀

We're doing really good progress wise

I think we were at like 260 something on todo this morning when I peeled myself out of bed