rewby - Looks like we are target bound again. I didnt add any additional workers so I guess we filled buneary

:(

probably AB should archive all the .well-known etc stuff too

!a transfer.archivete.am/R6JJo/sitemap_urls_january_february_2023.txt

datechnoman: Deduplicating and queuing 254459 items. (for 'transfer.archivete.am/R6JJo/sitemap_urls_january_february_2023.txt')

datechnoman: Deduplicated and queued 254459 items. (for 'transfer.archivete.am/R6JJo/sitemap_urls_january_february_2023.txt')

More sitemaps to chew on :)

datechnoman: awesome :)

and many of these (except sitemaps) are very low in required resources

yeah they are all pretty small :)

Over the coming week I will be working on more batches including robots.txt files etc

!a transfer.archivete.am/J2KuE/goo-gl.2023-02-12-00-17-01.txt

datechnoman: Skipped 27938 invalid URLs: transfer.archivete.am/qypPB/goo-gl.…023-02-12-00-17-01.txt.bad-urls.txt (for 'transfer.archivete.am/J2KuE/goo-gl.2023-02-12-00-17-01.txt')

datechnoman: Deduplicating and queuing 8801833 items. (for 'transfer.archivete.am/J2KuE/goo-gl.2023-02-12-00-17-01.txt')

datechnoman: Deduplicated and queued 8801833 items. (for 'transfer.archivete.am/J2KuE/goo-gl.2023-02-12-00-17-01.txt')

oh no, my urls grabber ip has been greylisted by bitninja.. for.. doing a GET on random images once every few minutes? :D

should I try to contest that or do I just roll with it?

some of the user agents of those "bad requests" they blocked look ancient btw, is that intended?

mh, list is two years old, maybe time to update?

Urls is a problematic project that often triggers lists. Bitninja is often tripped here

urls is a "opt-in-only" project due to that, and it has a warning in its description

I am aware and don't care about the ip reputation, more asking from the project point of view if I should ask them to unlist me

or just do nothing

Bitninja are clowns. Unless your hosting provider is complaining, I wouldn't worry about them.

Hetzner used to require a statement for them. Then they would forward the email as "informational" and not require a statement. I honestly haven't seen anything from them in a while so I'm not sure if Hetzner even forwards the complaints anymore

!a transfer.archivete.am/MUYL3/telegram.me_urls_processed.txt

datechnoman: Skipped 3 invalid URLs: transfer.archivete.am/RoehB/telegra….me_urls_processed.txt.bad-urls.txt (for 'transfer.archivete.am/MUYL3/telegram.me_urls_processed.txt')

datechnoman: Deduplicating and queuing 3739 items. (for 'transfer.archivete.am/MUYL3/telegram.me_urls_processed.txt')

datechnoman: Deduplicated and queued 3739 items. (for 'transfer.archivete.am/MUYL3/telegram.me_urls_processed.txt')

Echoing what Craigle said, bitninja honestly isnt' even worth reading past "bitninja" if you see the email

They're absolute clowns who greylist you just for saying hello to the wrong server

thanks :)

!a transfer.archivete.am/jRn4X/telegram.me_share_urls.txt

datechnoman: Something went wrong. (for 'transfer.archivete.am/jRn4X/telegram.me_share_urls.txt')

oops wrong copy and paste >.<

!a transfer.archivete.am/jRn4X/telegram.me_urls.txt

datechnoman: Deduplicating and queuing 15339 items. (for 'transfer.archivete.am/jRn4X/telegram.me_urls.txt')

datechnoman: Deduplicated and queued 15339 items. (for 'transfer.archivete.am/jRn4X/telegram.me_urls.txt')

Before anyone panic's on the name of the file, they are telegram.me share urls, processed to be standard urls for #// consumption