01:23:50 <h2ibot> Flashfire42 edited List of websites excluded from the Wayback Machine (+25): https://wiki.archiveteam.org/?diff=51396&oldid=51390
01:34:52 <h2ibot> OrIdow6 edited Google Drive (+1594, Make some of my research useful for future…): https://wiki.archiveteam.org/?diff=51397&oldid=50420
01:55:58 <fireonlive> OrIdow6++
01:55:59 -eggdrop- [karma] 'OrIdow6' now has 1 karma!
01:56:34 <fireonlive> sites do privately appear in folders at leaast but hm
02:00:57 <h2ibot> JAABot edited List of websites excluded from the Wayback Machine (+0): https://wiki.archiveteam.org/?diff=51398&oldid=51396
02:15:55 <JAA> ETA for OneHallyu is 4 days, 3 hours. Probably not going to finish in time.
02:22:01 <h2ibot> OrIdow6 edited Google Drive (+86, New discoveries involving Sites in Drive): https://wiki.archiveteam.org/?diff=51399&oldid=51397
02:53:35 * fireonlive waits
02:54:00 <fireonlive> cron pls
02:54:07 <h2ibot> FireonLive edited Current Projects (+95, attempt to clean up/make easier to read the…): https://wiki.archiveteam.org/?diff=51400&oldid=51243
02:54:10 <fireonlive> cc JAA/arkiver
02:56:06 <arkiver> looks good fireonlive
02:56:19 <fireonlive> =]
02:56:31 <arkiver> i'm not sure if we still need the ukraine/russian sites project there, it's not running since a long time
02:56:56 <fireonlive> ah good point
02:57:35 <fireonlive> Long-term, perpetual projects?
02:57:41 <fireonlive> did we have a word for 'basically forever'
02:57:49 <fireonlive> an internal word that is
02:58:07 <fireonlive> "occurring repeatedly; so frequent as to seem endless and uninterrupted."
02:58:09 <fireonlive> that works
02:58:30 <arkiver> i'd just say long term
02:58:40 <arkiver> can't promise we'll keep them running forever
02:58:43 <JAA> I've used 'continuous' before, but doesn't really say much.
02:58:50 <JAA> Yeah, 'long term' is good.
02:59:46 <fireonlive> ah ok
03:00:10 <JAA> Nothing will keep running forever. The heat death of the universe will consume it all.
03:00:17 <fireonlive> yep :)
03:00:28 <arkiver> so we might as well fade out now?
03:00:28 <fireonlive> 50/50 on leaving a blank section, but will leave an empty medium for now, to show that it 'can' exist
03:00:34 <fireonlive> arkiver: that's my dream
03:00:55 <arkiver> ouch
03:01:02 <JAA> '(none currently)'?
03:01:15 <fireonlive> ah that works
03:01:27 <JAA> Rather than just an empty section, which may look weird.
03:01:56 <JAA> I plan on hanging out on this channel until the heat death. :-)
03:02:06 <fireonlive> :)
03:02:15 <fireonlive> have we had a scripts only project in the past N years
03:02:33 <fireonlive> *removes commented out section*
03:04:12 <h2ibot> FireonLive edited Current Projects (-133): https://wiki.archiveteam.org/?diff=51401&oldid=51400
03:05:07 <fireonlive> arkiver:  "2019-202? coronavirus outbreak: Documenting and preserving data, events, and impacts of the virus on society. IRC Channel #coronarchive (on hackint)" < would you call this one not running as well?
03:05:19 <arkiver> yes
03:05:22 <fireonlive> kk
03:06:16 <h2ibot> FireonLive edited Current Projects (-167, remove coronavirus): https://wiki.archiveteam.org/?diff=51402&oldid=51401
03:06:36 <fireonlive> wow i cured the world i guess
03:06:37 <fireonlive> :p
03:07:10 <TheTechRobo> Photobucket did the purge *long* ago, right?
03:07:14 <TheTechRobo> Should that be removed from upcoming?
03:07:47 <TheTechRobo> also I feel like some of these hiatuses will never be unhiatused (is that a word?)
03:07:50 <TheTechRobo> ex. Audit 2014
03:08:28 <TheTechRobo> finally, https://wiki.archiveteam.org/index.php/NewsGrabber has been largely replaced with #//, right? should the wiki page be updated with that info?
03:08:29 <fireonlive> there was a line on the audit 2014 haitus bullet that it would be done in 2016 that i removed a few months ago
03:08:47 <TheTechRobo> oh, it does say under project status
03:08:50 <fireonlive> re: NewsGrabber it does say "Archiving status	Project superseded by URLs"
03:08:51 <fireonlive> ye
03:09:03 <fireonlive> lemme just...
03:10:18 <h2ibot> TheTechRobo edited NewsGrabber (+51, replaced with #//): https://wiki.archiveteam.org/?diff=51403&oldid=50757
03:10:43 <fireonlive> hey you
03:10:48 <fireonlive> conflicting my edit
03:10:52 <fireonlive> .
03:11:19 <h2ibot> FireonLive edited NewsGrabber (+74): https://wiki.archiveteam.org/?diff=51404&oldid=51403
03:11:39 <fireonlive> oops forgot a message lol
03:13:19 <h2ibot> TheTechRobo edited NewsGrabber (+1, Add a period): https://wiki.archiveteam.org/?diff=51405&oldid=51404
03:13:20 <h2ibot> TheTechRobo edited URLs (+222, Add urls-sources): https://wiki.archiveteam.org/?diff=51406&oldid=50427
03:14:19 <h2ibot> FireonLive edited Current Projects (+0, alphabetize "on hiatus"): https://wiki.archiveteam.org/?diff=51407&oldid=51402
03:14:19 <arkiver> oh yeah newsgrabber was kind of out predecessor to #//
03:15:01 <fireonlive> https://wiki.archiveteam.org/index.php/Project_Newsletter < neat idea
03:15:36 <TheTechRobo> > yt-dlp can be used to download article URLs, making it possible to preserve news in video-form just as well as news in text-form.
03:15:36 <TheTechRobo> I don't think we have that in URLs, do we? I suppose the storage would get unwieldy
03:15:52 <TheTechRobo> Might be nice for high-value stuff, though
03:16:17 <fireonlive> archivebot used to use youtube-dl (before the fork) but not any longer
03:16:42 <TheTechRobo> Yeah
03:16:52 <TheTechRobo> That integration was always jank IIRC though
03:16:59 <arkiver> we don't use yt-dlp in any project, except for the bot in #down-the-tube to discover videos of a channel for queuing
03:17:06 <arkiver> err
03:17:09 <TheTechRobo> arkiver: I thought yt-dlp was replaced in the bot?
03:17:20 <arkiver> any Warrior project i should say
03:17:24 <fireonlive> arkiver: is the bot on git :p
03:17:29 <arkiver> TheTechRobo: only partially replaced
03:17:40 <arkiver> fireonlive: no, it has keys that i didn't separate out yet
03:17:44 <arkiver> but yes i should get it on git
03:17:44 <fireonlive> ahh np
03:17:47 * TheTechRobo asked about that before :P
03:17:52 <fireonlive> no rushy
03:18:03 <JAA> Yes please :-)
03:18:03 <TheTechRobo> arkiver: should Photobucket be removed from upcoming/proposed? or is it still planned?
03:18:09 <arkiver> just have to free up some time for that
03:18:13 <TheTechRobo> Can we have the tracker next?
03:18:19 <arkiver> TheTechRobo: i don't think it's planned at the moment
03:18:24 <fireonlive> good luck for tracker :P
03:19:00 <arkiver> TheTechRobo: it current tracker is so very duck taped together (with sensitive stuff spread across it), that it will likely not be released publicly any time soon
03:19:01 <TheTechRobo> I've been asking ever since I touched Seesaw. Universal-tracker is, despite the name, not very universal
03:19:01 <fireonlive> arkiver: oh, one more q: are the IDs it generates stored in a database or something somewhere alongside the explanation provided/project/etc?
03:19:10 <arkiver> i believe the old tracker on github should still somewhat work?
03:19:16 <fireonlive> or is it mainly for irc logs?
03:19:17 <TheTechRobo> arkiver: Somewhat is right.
03:19:26 <TheTechRobo> fireonlive: I also have the same question about `-e`
03:19:37 <arkiver> fireonlive: the bot for queuing you main? they are currently only in the logs
03:19:43 <TheTechRobo> arkiver: No backfeed, slow, no offloader, etc
03:19:43 <arkiver> together with the explanation, only in the logs
03:19:45 <fireonlive> ye indeed
03:19:51 <fireonlive> ah ok :)
03:19:57 <arkiver> TheTechRobo: yeah
03:20:14 <fireonlive> eventually ™
03:20:17 <fireonlive> :D
03:20:21 <arkiver> i guess :/
03:20:23 <fireonlive> tracker is more understandable
03:20:32 <fireonlive> so i don't hold that one against y'all lol
03:20:46 <arkiver> :)
03:20:49 <fireonlive> :)
03:20:50 <TheTechRobo> The lack of an offloader was the main reason I never archived very much of Strawpoll. Whenever tracker was running, even idle, ~4GB of RAM usage because everything was in memory
03:21:39 <arkiver> i could have setup a project for that, if i was aware
03:21:41 <TheTechRobo> Maybe a project for 2024. Building Universal-tracker 3 :P
03:21:52 <arkiver> set up*
03:21:52 <TheTechRobo> arkiver: No shutdown notice, I just felt like archiving it
03:21:58 <arkiver> ah okey
03:22:03 <arkiver> it's still online?
03:22:05 <TheTechRobo> No
03:22:07 <fireonlive> i'm sure someone will set something big on fire on 2024
03:22:14 <arkiver> went offline without shutdown notice?
03:22:18 <fireonlive> well a lot of somethings
03:22:24 <TheTechRobo> arkiver: No idea
03:22:33 <TheTechRobo> I thought "y'know maybe I should continue archiving strawpoll" and it was ded
03:22:47 <arkiver> fireonlive: maybe, i expected more to burn down with higher interest rates. maybe that will come next year still as the rates stay somewhat high and companies need to refinance
03:22:59 <arkiver> TheTechRobo: sad :/
03:23:03 <TheTechRobo> Yeah
03:23:16 <fireonlive> https://support.fandom.com/hc/en-us/articles/7951865547671-August-2022-StrawPoll-me-closure
03:23:16 <fireonlive> updated 2023-02-09, closed 2022-08
03:23:18 <TheTechRobo> I did get a bunch of polls, but nowhere near everything :/
03:25:07 <arkiver> are they on IA?
03:25:54 <fireonlive> https://nitter.net/StrawPollme
03:25:58 <TheTechRobo> arkiver: i think? this was when I was very new to ATY
03:25:59 <TheTechRobo> *AT
03:25:59 <fireonlive> their twitter kinda died lol
03:26:10 <TheTechRobo> ah: https://archive.org/details/strawpoll-my-grab
03:26:22 <h2ibot> TheTechRobo edited Strawpoll.me (+2, Update info): https://wiki.archiveteam.org/?diff=51408&oldid=49804
03:26:24 <fireonlive> apparently they were having technical issues? and i guess didn't want to spend resources into fixing it
03:26:35 <TheTechRobo> fireonlive: lol
03:26:58 <fireonlive> https://old.reddit.com/r/NoStupidQuestions/comments/rzwnpk/is_strawpollme_going_to_be_broken_forever/
03:27:19 <fireonlive> i vaguely remember others saying 'not to use the .me' version as well
03:28:44 <fireonlive> ooh, dark IA items with poll data :3
03:29:26 <TheTechRobo> fireonlive: Yeah, not sure what's up with that
03:41:49 <arkiver> TheTechRobo: how did you archive it?
03:41:57 <arkiver> meaning how was the WARC created
03:43:29 <fireonlive> looks like v1.20.3-at of https://github.com/archiveteam/wget-lua
03:43:39 <fireonlive> from https://git.thetechrobo.ca/TheTechRobo/strawpoll-grab/src/branch/master/get-wget-lua.sh anyhow
03:45:20 <fireonlive> https://git.thetechrobo.ca/TheTechRobo/strawpoll-grab/src/branch/master/pipeline.py#L161
03:50:56 <arkiver> thanks
03:51:11 <arkiver> TheTechRobo: i've moved the strawpoll item to archiveteam-fire , it will soon be in the Wayback Machine
03:51:43 <fireonlive> i have my own collection?
03:51:47 <fireonlive> :D
03:52:07 <fireonlive> also sweet news :)
03:53:21 <arkiver> hah i guess so :)
03:53:54 <fireonlive> :3
04:00:55 <TheTechRobo> arkiver: Holy shit lmao
04:04:21 <arkiver> TheTechRobo: ?
04:05:04 <TheTechRobo> arkiver: My shitty code made it into the WBM! :P
04:05:56 <arkiver> well as long as the records are fine, it should be good :)
04:06:41 <TheTechRobo> I don't even think Wget-AT *lets* you write invalid records :P
04:07:00 <TheTechRobo> Well, I guess you could override DNS
04:07:02 <arkiver> yeah :)
04:07:09 <arkiver> for DNS yes i guess
04:07:11 <TheTechRobo> But you could do that anyway
04:08:51 <TheTechRobo> Wget-AT is amazing
04:09:00 <TheTechRobo> Wget-AT++
04:09:01 -eggdrop- [karma] 'Wget-AT' now has 2 karma!
04:09:16 <arkiver> thanks :)
04:09:21 <arkiver> many improvements coming up!
04:09:35 <fireonlive> =]
04:09:52 * arkiver is preparing a response to the recent responses from the TLS working group on our proposed mime types and URIs for SSL/TLS
04:10:58 <fireonlive> good luck with those IETF types
04:11:25 <TheTechRobo> I'd also suggest adding some sort of unit testing
04:12:02 <fireonlive> i'm sure it's on mind
04:12:11 <TheTechRobo> Yeah
04:26:40 <arkiver> thanks...
04:44:08 <fireonlive> 🆕 !tell now supports hostmasks (nick!user@host) e.g. !tell *!*@balls.example hello
04:44:21 <fireonlive> (with wildcards)
05:20:27 <Ryz> Mmm, welp, from a random checking of links to ignore on some ArchiveBot jobs, sadly https://forum.mobilelegends.com/ has shut down earlier this year on April 30
05:20:50 <Ryz> ...I don't think we have much people in the mobile game area of things strongly about it :c
18:09:45 <SketchCow> Hi Jason,
18:09:45 <SketchCow> Sorry to bother you, but Joe Baugher has died. He wrote up so much about
18:09:45 <SketchCow> aviation throughout the years and his articles are invaluable. Would you
18:09:45 <SketchCow> mind asking the Archive team to archive his home page one last time?
18:09:45 <SketchCow> https://www.joebaugher.com/
18:09:47 <SketchCow> All the best,
18:09:50 <SketchCow> Chris
18:44:25 <c3manu> SketchCow: I’m not jason, but i think this is something we can do :)
18:45:45 <c3manu> oh wait. you're jason >.<
18:47:08 <c3manu> it’s queued :)
18:49:57 <Nulo|m> is there something easy to quickly (multi connection) download a list of urls (without following links) into a warc?
18:51:48 <c3manu> Nulo|m: i’m not as experienced as other users here (which might have better answers for you), but you shouldd just be able to use wget for that
18:52:04 <Nulo|m> i guess just have to make a script to run many wget right?
18:52:17 <Nulo|m> also i can't find a flag to not download files into a file when i'm already downloading them into a warc in wget
18:53:54 <c3manu> Nulo|m: well, it has to download them, but there's the --delete-after flag which gets rid of them once they're in the warc
18:54:33 <Nulo|m> thanks!
18:54:35 <c3manu> wget has a --background mode, but i would assume they then cannot write into the same warc file
18:54:47 <c3manu> if you have multiple running i mean
18:56:15 <c3manu> there's also wpull, a wget fork, which archivebot also uses to download things. that one supports concurrency, but depending on your python version it might be a little fidddly to set up: https://github.com/ArchiveTeam/wpull
18:57:01 <c3manu> correction: it's not a fork, just another tool. my bad
18:57:39 <c3manu> what do you need the warc for, if i may ask?
18:58:00 <Nulo|m> i'm downloading product pages to then scrap them offline
18:58:51 <c3manu> why the warc then, and not just the pages themselves?
18:59:27 <Nulo|m> because if i need to pull more info later that i wasn't scrapping before, i can still just pull from the warc
18:59:46 <Nulo|m> also my scraper is kind of hacky so if it's bad i can just re-run it on the WARCs
19:00:05 <c3manu> i see, that makes sense.
19:00:07 <Nulo|m> also i should be able to run the scraper on WARCs from archive.org or other sources :)
19:02:51 <c3manu> ok. apart from wpull i am running out of ideas. hopefully someone else can give you better answers when they're back :)
19:02:59 <c3manu> did you have a look at https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem already?
19:04:13 <Nulo|m> no, thank you!
19:04:22 <Nulo|m> i think i'll make a script based on wget though
19:04:49 <fireonlive> there’s wget-at to :)
19:04:50 <fireonlive> too
19:05:25 <Nulo|m> yah but wget works fine for me and i believe wget-at doesn't have multi-connection, just improved warc stuff?
19:05:40 <c3manu> ah, that would probably be the fork than that i confused wpull with earlier
19:06:19 <fireonlive> improved warc stuff sounds pretty paramount :3
19:07:05 <Nulo|m> hehe but the warcs generated by gnu wget work fine with warcio.js which is what i'm using so 👍️
20:53:19 <JAA> I'm averaging 5k OneHallyu topics per hour now. They went read-only at 2023-12-20T11:23Z or so (date of the last post by an admin). If they shut it down at the same time of day, I expect to have covered about 81% of the topics.
20:54:09 <nicolas17_bot> more parallelism/IPs unlikely to help?
20:54:27 <JAA> Their potato is too slow.
20:54:39 <JAA> 6 second average response time.
20:59:45 <JAA> Let's see what happens if I throw more at it...
21:00:19 * Barto observes an explosion in the horizon
21:00:41 <nicolas17_bot> also try less, if there's resource contention on the server it could have weird effects
21:02:15 <JAA> Can't easily go to less, but yeah, I might if this makes it worse.
21:02:33 <nicolas17_bot> ("half the threads, 2 second response time" would be a net win, though unlikely)
21:02:34 <JAA> Average response time now: 8404 ms  ._.
21:02:40 <fireonlive> x_x
21:03:14 <JAA> Throughput still went up a bit though.
21:04:00 <nicolas17> hm how's your network-layer latency to their server?
21:04:51 <JAA> They hide behind Buttflare, so no idea.
21:05:08 <nicolas17> oh :|
21:05:33 <nicolas17> that latency is also irrelevant if they're in CF
21:06:33 <JAA> Depends on what their backend looks like, but the point is rather that I can't measure it anyway.
21:07:11 <nicolas17> if there wasn't CF, doing the crawl from somewhere closer could help
21:07:51 <JAA> Possibly, although it can usually be balanced by higher concurrency.
21:54:30 <JAA> I'm back down to the same throughput from before I increased the concurrency.
23:21:34 <tech234a> “Bluesky makes web view public, login no longer required to read posts” https://news.ycombinator.com/item?id=38739130
23:22:38 <fireonlive> nicolas17: feel free to use #fire-spam for testing
23:23:03 <fireonlive> everyone got to witness the bee movie so what's a bit more :p