-
knecht4Hi! I'm currently looking to archive the comments section on xrel.to as the mods are a bit trigger happy with deleting stuff and to preserve edits and similar. New comments show up on the frontpage, there is no other feed of sort.
-
JAAxREL still exists. :o
-
knecht4If it helps, I could provide an up to date list somewhere as I'm grabbing the links from IRC anyway.
-
knecht4It's still going well yeah! Piece of digital heritage tbh.
-
JAASo what we could do here is retrieve the homepage regularly and follow one layer of links. That would include the comment links.
-
JAAWhen we aren't backlogged due to *gestures broadly in the general direction of datechnoman*, that should be fairly reliable.
-
knecht4That would be great! How regularly are we talking, just out of curiosity?
-
JAAThe lowest we've gone is intervals of one minute. Realistically, it doesn't exactly work out to actually being one minute, but normally every couple minutes at least: web.archive.org/web/20240000000000*…://news.ycombinator.com/newcomments
-
JAAYou can see in the graph when we started this. :-P
-
knecht4Haha yeah, quite the spike. That sounds great and I would really appreciate it. Done my share of TBs in various projects here so I'm stoked to contribute something content wise.
-
imerin datechnoman's defense, if we could queue to lower priority lists via bot that wouldn't be an issue ;p
-
JAAknecht4: Note that we'd only fetch each comment once, shortly after it's posted. (Although we'd probably catch some edits via the embedding on the homepage.)
-
JAAAre edits announced on IRC?
-
knecht4Archiving the comments in the frontpage would at least catch edits in comments above the most recent I guess.
-
knecht4And no, only new comments, reviews and forum posts are announced.
-
knecht4The comments are where the "value" lies I would argue.
-
datechnomanI'm always to blame these days lol!
-
datechnomanI haven't queued anything in like 48 hours here :P
-
datechnomanI should be given a medal
-
JAA2024-08-14 05:29:40, that's not 48 hours. :-P
-
JAABut yeah, prioritising/queueing to secondary or whatever would be nice.
-
JAAI know it's been on the wishlist for years.
-
datechnomanAlways pulling out the logs to be right. I thought we were friends....!
-
JAA<3
-
datechnomanIn my defense I typically let's the backlog clear out before adding more. This batch just has seemed to keep growing
-
knecht4Sooo when can I turn off my half baked irc archivebot? :^)