hackint #//

7 Jun 2023

Grabbing 'interesting' URLs from a variety of places | Got suggestions? Let's hear 'em! | Ideas involving many URLs at the same source are likely not suitable here; please use #archiveteam-bs instead for those. | wiki.archiveteam.org/index.php/URLs | tracker.archiveteam.org/urls

• 02:20
  pabs
  I sent a PR adding lobste.rs (similar to HN but smaller) to urls-sources, is that reasonable? can the PR get merged?
• 02:21
  pabs
  ArchiveTeam/urls-sources #22
• 11:07
  datechnoman
  Running so much cleaner now for the past 12+ hours. Thanks rewby|backup :)
• 11:07
  rewby|backup
  datechnoman: I did nothing. Looked at the servers and they looked fine.
• 11:07
  rewby|backup
  This is why i ask you to just keep slamming it
• 11:08
  rewby|backup
  That way i can see where the bottlenecks are
• 11:08
  rewby|backup
  And it's not causing a significant delays
• 11:08
  rewby|backup
  Because the "errors" are just the way it applies backpressure
• 11:08
  rewby|backup
  Net throughput on my end doesn't change
• 11:09
  rewby|backup
  Yes it has thundering herd. That's intended
• 11:09
  datechnoman
  Interesting and weird. It was super choppy for few days on the worker side
• 11:09
  datechnoman
  Ahh well. It came good. Whatever lol
• 11:09
  datechnoman
  *throws hands in the air*
• 11:09
  datechnoman
  Take the win :P
• 11:09
  rewby|backup
  As I said, don't back off if that happens. Just let me know and I can figure out where it goes wrong
• 11:09
  rewby|backup
  If i need some backoff i have controls on the tracker to do that
• 11:10
  datechnoman
  No worries at all. Will do :)
• 11:10
  rewby|backup
  Usually it's an intermix of targets shared between projects
• 11:10
  rewby|backup
  And things fluctuate
• 11:11
  datechnoman
  Which makes total sense. No point dedicated servers to projects as they go up and down depending on what is being processed
• 11:21
  rewby
  The real tldr is that I need to find the time to unfuck my monitoring
• 11:33
  rewby
  datechnoman: It does look like optane9 is having some load issues
• 11:33
  rewby
  Although it's uploading at 5gbps
• 11:33
  rewby
  Consistently
• 12:22
  datechnoman
  Yeah monitoring would be good. Makin me feel like im crazy lol
• 12:22
  datechnoman
  Optane is beast
• 12:37
  rewby|backup
  Yeah, I'm exploring options for a other similar box
• 12:37
  rewby|backup
  But that's at least a month or so out
• 22:06
  datechnoman
  Sounds like a great plan. Everything costs money and time so all good. Good to know a plan is in the works as im sure the demand is just going to keep increasing over time