00:25:14 <JAA> After over 40 million requests and 29.7 TiB downloaded, the qwarc process finally reached the memory limit and stopped. This is the normal intended behaviour due to a 'memory leak' (probably not an actual leak but heap fragmentation or similar), but I didn't expect it to take that long.
00:28:33 <pokechu22> qwarc process on what?
00:28:55 <JAA> Knowledge Adventure CDN
00:29:53 <h2ibot> Arkiver uploaded File:Tiki-video-logo.png: https://wiki.archiveteam.org/?title=File%3ATiki-video-logo.png
00:47:56 <h2ibot> Yts98 edited Current Projects (+156, Propose Game Atsumaru): https://wiki.archiveteam.org/?diff=50020&oldid=49956
00:49:27 <arkiver> yts98: on game atsumaru - since it's a limited number of games, could we simulate the browser and capture the URLs?
00:52:51 <yts98> sure we can stay on the game homepage longer to caputure the URLs. but frameworks like RMMV, RMMZ only load resource per request, and few indie games have anti-debugging to detect headless browsers
00:53:48 <arkiver> yts98: feel free to post a request in #archiveteam , that has a better chance of succeeding than the wiki
00:53:57 <arkiver> thought not much time left so I would not get my hopes up
00:55:10 <thuban> arkiver: yeah, the resource url extraction is pretty bespoke, so idk if it's realistic to duplicate in a warrior project
00:56:42 <thuban> *but* yts98's existing scripts also output url lists. can't just toss them in ab because of the session cookie thing, but definitely possible as dpos, right?
00:58:04 <thuban> yts98: new error https://transfer.archivete.am/14x6N/atsumaru_496.log
00:58:22 <yts98> not sure if the current wget-at can make multipart/form-data POST request properly.
00:58:51 <arkiver> no
00:59:12 <thuban> remind me, is that required to get the session cookie?
00:59:19 <arkiver> or well
00:59:19 <yts98> yes
00:59:29 <arkiver> if you have the exact data and headers to send I guess you could do it
00:59:59 <arkiver> but is it actually needed?
01:00:05 <arkiver> not everything the browser does is actually needed
01:01:34 <yts98> i did not try with application/x-www-form-urlencoded
01:01:34 <yts98> a possible approach is make request in python and them pass the cookies to wget-at as environment variables
01:01:59 <arkiver> what percentage of games does this affect?
01:03:17 <yts98> every game is guarded by the "ticket" session cookie
01:04:46 <thuban> arkiver: https://github.com/yts98/game-atsumaru-discovery/blob/master/step6-MV_iterate_.py#L61-L77
01:05:18 <thuban> lol oops didn't reload. 62-78
01:08:30 <JAA> (Press y to get a permalink to the exact revision rather than the mutable branch name.)
01:09:09 <yts98> https://github.com/yts98/game-atsumaru-discovery/blob/f5a1b46fa3bc76196b3b42c7bf3d61e5df8efe61/step6-MV_iterate_.py#L62-L78
01:24:13 <thuban> yts98: https://pad.notkiska.pw/p/game-atsumaru possibly helpful
03:00:28 <h2ibot> JAABot edited CurrentWarriorProject (-8): https://wiki.archiveteam.org/?diff=50021&oldid=49989
03:29:33 <h2ibot> Vokunal edited Tiki (+123): https://wiki.archiveteam.org/?diff=50022&oldid=50016
03:31:33 <h2ibot> Vokunal edited Tiki (+0): https://wiki.archiveteam.org/?diff=50023&oldid=50022
03:32:43 <vokunal|m> Figures my second wiki edit would be fixing a problem in my first one. I see I have a bright future
04:16:32 <fireonlive> vokunal|m: brighter than mine!
04:23:03 <vokunal|m> Thanks!
04:23:03 <vokunal|m> Wait
04:26:51 <fireonlive> :)
06:29:03 <h2ibot> FireonLive edited Current Projects (+135, add Tiki): https://wiki.archiveteam.org/?diff=50024&oldid=50020
06:33:34 <fireonlive> (not sure if <!-- Urgent projects is supposed to be alphabetical or in order of urgency there to feel free to slap the line around)
06:35:49 <fireonlive> (or chronological but it doesn't appear to be)
08:05:27 <h2ibot> Systwi created Tjournal (+22, Created an article redirecting "Tjournal" to…): https://wiki.archiveteam.org/?title=Tjournal
08:06:27 <h2ibot> Systwi edited Tjournal (-6, Corrected redirect target from "Tjournal" to "TJ."): https://wiki.archiveteam.org/?diff=50026&oldid=50025
08:07:52 <systwi_> Bah, I meant "TJournal" in that last message...in case those messages _really_ matter. :-P
08:09:44 <systwi_> Hehe, even I make mistakes. No need to worry, fire and vokunal. :-)
08:10:32 <systwi_> It's all a learning experience, and thankfully corrections can be made a click or two away.
08:14:15 <yts98> Script for 1332 RMMZ games is ready. Still looking for operators to claim gameId ranges on https://pad.notkiska.pw/p/game-atsumaru in 19 hours!
08:16:29 <h2ibot> Systwi edited Topics of Archiving Interest (+44, /* Multimedia Franchises */): https://wiki.archiveteam.org/?diff=50027&oldid=49224
08:17:29 <h2ibot> Systwi edited Topics of Archiving Interest (+10, /* Multimedia Franchises */ Added 1 entry.): https://wiki.archiveteam.org/?diff=50028&oldid=50027
08:17:40 <systwi_> Made another erratum, hehe. Right Shift turned into Right Shift + Enter.
08:19:29 <h2ibot> Systwi edited Topics of Archiving Interest (+54, /* Work/Occupations */ Added 1 entry.): https://wiki.archiveteam.org/?diff=50029&oldid=50028
08:21:30 <h2ibot> Systwi edited Topics of Archiving Interest (+60, /* Celebrities/Famous Individuals */ Added 1…): https://wiki.archiveteam.org/?diff=50030&oldid=50029
08:23:30 <h2ibot> Systwi edited Topics of Archiving Interest (+65, /* Leisure */ Added 1 entry.): https://wiki.archiveteam.org/?diff=50031&oldid=50030
08:37:33 <h2ibot> Systwi edited Topics of Archiving Interest (+642, /* Celebrities/Famous Individuals */ Added 9…): https://wiki.archiveteam.org/?diff=50032&oldid=50031
09:45:42 <Misty|m> What's the channel for : https://github.com/yts98/game-atsumaru-discovery?
10:02:34 <yts98> Misty|m: not decided yet. all previous discussions happened here.
10:29:07 <yts98> thuban: now I'm asking STWP for help, so the currently unclaimed gameids may be saved later.
12:50:41 <VickoSaviour> is the blog.harvard.edu saved?
13:17:52 <yts98> Script for 1019 Akashic Engine games is ready. Still looking for operators to claim gameId ranges on https://pad.notkiska.pw/p/game-atsumaru in 14 hours.
13:32:35 <rktk> yts98, does it require a JP IP address for proper discovery?
13:32:38 <rktk> VPN JP IP
13:34:57 <yts98> rktk: probably not. I am using a residential IP outside of Japan.
13:39:05 <rktk> Oh that's good
13:39:15 <rktk> I was going to offer help but, I have 500GB Free of 110TB so. I need to free up some space
13:46:23 <immibis> just buy more hard drives :)
14:00:35 <h2ibot> JAABot edited CurrentWarriorProject (+8): https://wiki.archiveteam.org/?diff=50033&oldid=50021
14:07:15 <Hans5958> Someone should move "Google Drive" on the main page from active to hiatus
14:07:32 <Hans5958> *warrior-based projects
14:20:39 <h2ibot> PaulWise edited Mailman2 (+28, lua-l kind of looks like a pipermail archive…): https://wiki.archiveteam.org/?diff=50034&oldid=50007
14:44:39 <rktk> immibis, it's a rented dedicated server :(
14:44:45 <rktk> no upgrades
14:44:56 <rktk> I will probably buy a set of HDDs for a new build soon, it's just... so expensive
15:15:36 <immibis> you rent a server with 110TB of storage?
16:42:29 <thuban> yts98: new error https://transfer.archivete.am/NrXdp/atsumaru_8573.log
16:55:00 <yts98> thuban: commit 35954f2 fix 8573
16:58:06 <thuban> ty
16:59:12 <rktk> immibis, yes
16:59:21 <rktk> it was the best option when I was travelling
16:59:28 <rktk> cheaper than glacier s3 too
17:00:12 <fireonlive> from where?
17:00:20 <arkiver> Stitcher has sitemaps
17:00:27 <arkiver> we might do a project for this one
17:00:42 <thuban> yts98: by the way, what is the plan for storage? i assume this is going to go to the internet archive, but do you have a way to transfer the data and enough space to consolidate it before upload?
17:00:57 <rktk> fireonlive, it's one of the SX line of servers from hetzner
17:01:04 <fireonlive> ahh :)
17:01:08 <fireonlive> tks
17:01:34 <rktk> I will stop renting the server soon since I will self host at home, and instead shift that monthly cost towards another glacier s3 compatible provider
17:01:44 <rktk> so then I have 2 copies and 1 is offsite, so I am missing the 3rd copy.
17:01:52 <rktk> but, 3rd copy doesn't have to be for everything like that...
17:02:00 <rktk> in the cloud i mean
17:02:15 <arkiver> yeah i see signs sitemaps are being updated
17:02:26 <arkiver> thuban: yts98: how much is this?
17:03:31 <thuban> arkiver: hundreds of gigs (uncompressed and with duplication--bare files + warcs)
17:04:57 <arkiver> nice, yeah that can be just stored on IA
17:31:13 <h2ibot> Manu edited Deathwatch (+277, Adding Oryx, a news blog that will shut down…): https://wiki.archiveteam.org/?diff=50035&oldid=50018
18:05:23 <rktk> if an entire website is saved, with wget, is it better to upload the .warc.gz file to archive, or the plain HTML files themselves
18:05:32 <rktk> HTML + associated files
18:06:15 <pokechu22> The .warc.gz file is useful (though if you're a random user it probably won't be added to web.archive.org but instead only to https://archive.org/details/warczone). If the size isn't too big, you could upload both of them
18:15:12 <rktk> it's close ish to 50GB total
18:15:19 <rktk> but I'm not the one that archived it
18:38:46 <masterX244> pokechu22 warcs are more convenient for grepping thru. backl when we hunted imgur links the warczone was sweeped, too
19:02:31 <h2ibot> Entartet edited List of websites excluded from the Wayback Machine/Partial exclusions (+43): https://wiki.archiveteam.org/?diff=50036&oldid=49650
19:05:35 <thuban> yts98: new error https://transfer.archivete.am/OrIsi/atsumaru_17614.log
19:42:04 <immibis> really, hetzner? I pay $60 for a server from hetzner with only 1TB of storage, but it was CPU-optimized
19:42:29 <immibis> Glacier Deep Archive is about $1/TB/month and I see Hetzner SX prices are not much higher than that
19:43:19 <Maakuth|m> Azure too has a cheap cold storage tier
19:43:30 <immibis> $97/mo for 64TB is surely better than any hot cloud storage out there. Of course, that's not accounting for replication
19:51:37 <fireonlive> (link for the rest of yall: https://www.hetzner.com/dedicated-rootserver/matrix-sx)
19:53:05 <thuban> yts98: new error https://transfer.archivete.am/13nVsi/atsumaru_19117.log
19:55:50 <threedeeitguy> yts98 How many instances is it safe to run per IP? Im up to 5 and still have bandwidth spare but don't wanna get banned.
19:56:53 <thuban> i've done a dozen and haven't seen anything like rate limiting
20:01:15 <threedeeitguy> cool. guess il power on another server then. wget can really hammer a CPU.
20:10:34 <yzqzss|m> thuban: threedeeitguy: I changed the MV_iterate_.py  script to use multiple threads (40 by default).
20:10:34 <yzqzss|m> run `git pull` to update :)
20:12:18 <imer> server action has some decent storage ones as well: https://www.hetzner.com/sb?drives_count_from=15&drives_size_to=16000 15x10TB for 150€ (excl. tax)
20:12:55 <imer> mh, that link doesnt work. just filter by drive size 10tb^
20:13:05 <fireonlive> ooh
20:13:28 <fireonlive> if only EUR to CAD wasn't such a ***** **** lol
20:15:41 <masterX244> immibis: my main archival crap runs on the 64TB one, too. 48 usable only, one HDD for parity
20:18:57 <threedeeitguy> cool. Il update once this lot finish.
20:28:28 <thuban> yts98: new error https://transfer.archivete.am/12A7U8/atsumaru_8298.log
20:54:58 <upintheairsheep> The following Discord server will be shutting down and moving to Telegram: https://discord.gg/cEH6cqMgBd
20:56:35 <fireonlive> hi/bye
20:59:46 <masterX244> doesnt even tell what the server is about, another issue: you can't check out invites in a private window while the app is open, iot pokes a connection to localhost
21:05:55 <fireonlive> 'hang out with 122 other members'
21:06:04 <fireonlive> logo the "windows 10" logo
21:06:05 <fireonlive> o_O
21:06:30 <fireonlive> 'kazlandia // new beginning'
21:31:07 <threedeeitguy> yts98 looks like all of MV has been claimed. I take it we just moved onto the next one?
21:38:43 <TNN> Hey guys - probably stupid question. What do each of these mean?
21:39:00 <TNN> claims: 17361101done: 7471389todo: 105762705todo:backfeed: 3782832todo:redo: 0todo:secondary: 0unretrievable: 0
21:40:03 <threedeeitguy> yts98 also nice update, going even quicker now.
21:42:15 <JAA> Looks like the Knowledge Adventure CDN download should finish at around 05:00.
21:43:32 <threedeeitguy> TNN Claims: workers have claimed the item but have not done it yet. This can include items that have been claimed but will never be finished (maybe the worker was switched off). These are eventually moved to another queue to be reclaimed. Done: Done :) todo: items to be done. IIRC this queue has the highest priority. Backfeed: items discovered by
21:43:32 <threedeeitguy> (or you could say fed back) by the workers. todo:redo already been done once but needs doing again. Probably because it temporarily failed. todo:secondary low priority to do queue jaa feel free to correct anything :)
21:44:15 <JAA> todo:secondary has a higher priority than :redo, it appears in the wrong order on the tracker page. Otherwise, correct.
21:45:12 <TNN> Thanks very much, super helpful
21:47:07 <JAA> The meaning of the queues might vary by project. We sometimes shuffle items around and just use them as priority queues, i.e. to process certain things before others.
21:52:20 <threedeeitguy> Worth adding an explanation to the wiki? It feels like there should be a dedicated page for the leaderboard, not everyone wants to go digging through the tracker docs.
21:53:21 <thuban> that sounds like a good idea (especially as "the tracker docs" is a rather aspirational phrase)
21:53:49 <JAA> ++
21:55:09 <thuban> JAA: where should it go? obvious answer is 'the tracker wiki page' but a lot of that stuff is wrong at present and i don't have enough details to correct it
21:58:47 <JAA> thuban: Yeah, not sure, maybe just the Warrior FAQ?
21:59:20 <JAA> I feel like we should restructure that at some point though. A lot of the stuff there isn't really related to the warrior but to DPoS projects more generally.
22:02:31 <threedeeitguy> I hate the wiki editor already :(
22:03:05 <thuban> JAA: you're technically correct, but from a usability perspective i'm kind of reluctant to direct warrior users to a secondary page for troubleshooting
22:03:45 <thuban> mediawiki magic to transclude relevant items on both the 'warrior' and 'running with docker' pages?
22:07:35 <nicolas17> immibis: Backblaze is really cheap compared to competition... and it would be $320/mo for 64TB
22:07:50 <JAA> Even just splitting everything warrior-specific into a separate section would do, I think.
22:08:29 <thuban> not sure what you mean; specify?
22:08:45 <thuban> (also, i will repeat my request for a toclimit)
22:08:49 <nicolas17> what's DPoS?
22:09:12 <thuban> nicolas17: Distributed Preservation of Service
22:09:54 <thuban> (what we often refer to as a 'project' or 'warrior project')
22:11:59 <threedeeitguy> yts98 seems to hang on completion now. Last line:
22:11:59 <threedeeitguy> 2023-06-27 22:57:06 URL:https://resource.game.nicovideo.jp/games/gm27293/26/img/tilesets/Dungeon_C.png [649991/649991] -> "games/gm27293/26/img/tilesets/Dungeon_C.png" [1]
22:11:59 <threedeeitguy> Is it safe to close the terminal at that point?
22:12:22 <JAA> thuban: E.g. 4.4 to 4.8 on https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior are not really warrior-specific.
22:13:13 <JAA> And people ask about them all the time, too.
22:14:37 <thuban> JAA: i understand that, just not sure what you mean by 'splitting into a separate section'. it wouldn't solve the duplication with the 'running with docker' page, and it wouldn't really be more helpful, would it?
22:16:33 <JAA> We could have a 'DPoS FAQ' with sections 'warrior' and 'not warrior'.
22:17:46 <JAA> This would depend on people actually using the names consistently though. I try to only use 'warrior' when I actually mean the warrior VM or container (or, technically, the seesaw parts driving that, although that's basically never discussed).
22:18:39 <thuban> we could, but i think that would be less usable for newbies
22:19:52 <nicolas17> oh I didn't know the tracker etc was called that
22:28:52 <thuban> JAA: do you have rights to edit https://wiki.archiveteam.org/index.php/MediaWiki:Common.css ? i think last time we talked about this you said you didn't, but mediawiki sez you're an admin and should
22:29:56 <JAA> I don't remember us talking about this, which suggests it was quite some time ago, possibly before I had admin rights, yeah.
22:30:02 <JAA> I should be able to.
22:30:22 <thuban> i actually checked and it was after :P but yeah
22:30:38 <thuban> can you add https://en.wikipedia.org/wiki/Template_talk:TOC_limit#Steps_to_limit_the_TOC_in_your_mediawiki ?
22:32:19 <JAA> thuban: Ah, I was probably confused by 'Add the following code *in the file* ...', thinking it required editing an actual file in the MW config dir etc. That I don't have access to.
22:32:38 <thuban> i figured it was something like that
22:32:56 <JAA> > You do not have permission to edit this CSS page because it may affect all visitors.
22:33:03 <JAA> (╯°□°)╯︵ ┻━┻
22:33:34 <thuban> ugh! i guess you have to make yourself an "interface administrator" as well? https://wiki.archiveteam.org/index.php/Special:ListGroupRights
22:34:18 <JAA> Yeah, looks like it, editsitecss permission.
22:34:59 <JAA> There we go.
22:35:11 <h2ibot> JustAnotherArchivist changed the user rights of User:JustAnotherArchivist (Need to edit [[MediaWiki:Common.css]])
22:36:11 <h2ibot> JustAnotherArchivist edited MediaWiki:Common.css (+474, Add TOC limit rules from…): https://wiki.archiveteam.org/?diff=50037&oldid=24389
22:36:12 <h2ibot> Switchnode created Template:TOClimit (+57, Created page with "<div…): https://wiki.archiveteam.org/?title=Template%3ATOClimit
22:41:12 <h2ibot> Switchnode edited ArchiveTeam Warrior (+7, limit toc depth for better readability): https://wiki.archiveteam.org/?diff=50039&oldid=50013
22:41:27 <thuban> thanks! <3
23:01:59 <imer> wouldn't hurt to have an explainer for the components of the dpos stack then? i made a (bad) flowchart-thingy https://transfer.archivete.am/Jqui7/2023-06-28_01-01-08_UKocGw5qlp.png probably needs some naming corrections
23:03:09 <arkiver> https://wiki.archiveteam.org/index.php/File:Archiveteam_warrior_infrastructure.png
23:04:24 <imer> yeah, saw that - less technical though which was the point, right?
23:04:37 <arkiver> :P
23:04:38 <imer> way better graphically of course :)
23:04:40 <arkiver> i just like the image
23:04:45 <imer> yeah its great
23:04:48 <arkiver> yes we could create a more technically correct image
23:11:17 <h2ibot> Switchnode edited ArchiveTeam Warrior (+668, /* Warrior FAQ */ add counter explanation here…): https://wiki.archiveteam.org/?diff=50040&oldid=50039
23:13:37 <thuban> https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#Can_I_use_whatever_internet_access_for_the_Warrior? have the recent quad9 updates made the item about dns here unnecessary?
23:17:15 <datechnoman> Correct. To my knowledge all projects are now using quad9 DNS by default set within the container arkiver?
23:18:56 <yts98> thuban: I'll inspect 8298 17614 19117 later.
23:19:35 <yts98> threedeeitguy: you can kill 27293 and redo it later.
23:23:28 <thuban> yts98: new error https://transfer.archivete.am/Nrh4B/atsumaru_9445.log
23:24:31 <yts98> thuban: got it. I'm investigating other frameworks, so debugging woulg be at lower priority.
23:24:48 <thuban> understood
23:25:14 <yts98> if anyone is able to resolve exceptions, feel free to make a pull request.
23:25:49 <yts98> s/woulg/would :p
23:45:22 <threedeeitguy> yts98 ive set 4 large jobs going with the rest of rmmz. There's 750gb of space left on that disk so I guess were gonna find out how large it is. Il let you know how it went in the morning. It is going a lot slower, looks like its only single threaded? If it needs an update and then a restart lmk, il be back in 8 hrs or so.
23:47:07 <yts98> the deadline is in 3 hours, so I would also do some duplicate ranges
23:47:40 <threedeeitguy> ah I thought it was later on today. np il see about speeding things up.
23:48:26 <flashfire42> 3tt38940863qg137qhybq2xbe there is also this AB job
23:48:45 <thuban> threedeeitguy: why are some of your ranges listed as "done?" with question marks?
23:49:21 <threedeeitguy> Those jobs hung on a few items. Im just checking the warcs to make sure it did get everything.
23:49:40 <threedeeitguy> If there are any missing il shout shortly.
23:50:33 <thuban> depending on the error(s) the script may have downloaded some files but not all
23:53:15 <threedeeitguy> Looks like the jobs completed fine, all files match up (I just re-ran all 4 affected items in a new folder to compare). Since the update a few hours ago my terminal window just hangs once the script finishes. I guess its related to the multithread changes. Does not appear to have affected the data in anycase.
23:55:37 <thuban> i'm a little concerned that i encountered half a dozen errors (some of which have not yet been fixed) whereas you have apparently seen none over a similar number of games.
23:55:53 <thuban> is it possible error messages got lost in the parallel output?