00:08:33 <threedeeitguy> potentially. There are only two lines in my MV-Errors.log file.
00:08:33 <threedeeitguy> gm26791 (MV): 'animation1Name'
00:08:33 <threedeeitguy> gm25516 (MV): Unterminated string starting at: line 24 column 1489 (char 94827)
00:09:38 <thuban> ah ok
00:10:05 <thuban> 26791 looks the same as my 17614
00:10:10 <thuban> not sure about 25516
00:17:23 <threedeeitguy> K, ive split it into a lot (136) of smaller jobs and am running them in parallel. Its still slower but it is vastly improved.
00:17:44 <thuban> nice
00:29:31 <h2ibot> Wickedplayer494 edited Surrender at 20 (+367, SS is paying attention to the looming expiry…): https://wiki.archiveteam.org/?diff=50041&oldid=49232
01:28:00 <yts98> thuban: commit f14a67e fix 8298. I'm doing it.
01:32:38 <yts98> Looking for someone to put https://transfer.archivete.am/OrLXQ/api_requests.txt.zst (900 Game Atsumaru comment / scoreboard API requests) quickly before 3AM UTC.
01:32:54 <yts98> i.e. in 1.5 hours.
01:33:04 <yts98> s/900/900K/
01:33:56 <thuban> you say 'requests'; are these just urls that can go in archivebot?
01:34:57 <yts98> Yes. they're GET request URLs without session requirement.
01:35:58 <thuban> i see you've asked in #archivebot, so someone will get to it shortly
01:39:10 <thuban> job id: askbyh40bwvdmpjqb5v0l3por
01:55:52 <thuban> yzqzss|m: mv 22500-24999 ok?
01:57:49 <JAA> Summary of #archivebot: AB is far too slow for this, but I'm grabbing them with qwarc instead, which does more brrrr.
01:58:27 <thuban> is that wbm certified or nah?
01:58:27 <JAA> Almost all responses are empty, but I guess that's expected.
01:59:08 <JAA> Where empty = {"meta":{"status":200},"data":{}}
02:02:09 <JAA> I just ran into a couple thousand 400s, random example: https://api.game.nicovideo.jp/v1/rpgtkool/games/gm10857/scoreboards/25.json
02:03:14 <Jake> "boardIdが範囲外です" == "boardId is out of range"
02:05:34 <yzqzss|m> <thuban> "yzqzss: mv 22500-24999 ok?" <- it's ok
02:11:01 <JAA> Those invalid URLs slow me down significantly due to how qwarc works on errors, but it should still finish in time at the current rate.
02:46:20 <JAA> It's done.
02:49:56 <h2ibot> FireonLive edited Egloos (+172, Egloos is offline. Data been partially saved.…): https://wiki.archiveteam.org/?diff=50042&oldid=49938
03:00:31 <fireonlive> lets see if minor edits show up too
03:00:58 <h2ibot> FireonLive edited Egloos (+50, tweaks, add lead): https://wiki.archiveteam.org/?diff=50043&oldid=50042
03:01:00 <fireonlive> so it do
03:01:12 <fireonlive> can't get anything past h2ibot
03:01:43 <JAA> Edits in the User and User_talk namespaces get ignored, but otherwise, no.
03:01:53 <fireonlive> ahh ok :)
03:02:13 <fireonlive> also, curse you mediawiki for making every first letter capital :p
03:08:37 <thuban> yts98: did you get 9445? it still errors out for me
03:08:44 <thuban> (after pulling, i mean)
03:09:23 <yts98> thuban: not able to get 9445 yet
03:09:48 <thuban> ok. sorry, wasn't sure what strikeout meant
03:13:37 <yzqzss|m> DDL was reached and the front-end pages are HTTP 30X to <https://blog.nicovideo.jp/niconews/194994.html>
03:22:34 <yts98> strikeout means I got it myself
03:23:59 <thuban> ? 9445 was struck out, did it work or no
03:24:22 <fireonlive> re IRL; it's already past 12pm PDT
03:24:27 <fireonlive> hm
03:24:34 <fireonlive> (from #archiveteam)
03:25:33 <fireonlive> seeing 500s for profiles and 404s for post links nulldata: https://www.irl.com/meg-myers/bTduA5Al5N
03:26:12 <fireonlive> where's what sketch quote about sites losing as much data as possible
03:26:24 <fireonlive> what→that
03:28:31 <JAA> I like this Foone quote: https://twitter.com/Foone/status/1146136479573299200
03:29:18 <fireonlive> ah! yes it was foone not sketch :)
03:29:21 <yzqzss|m> The Game Atsumar's api endpoint has also been redirected.
03:29:21 <yzqzss|m> To prevent wget from accidentally overwriting the previous warc files, don't run any step6-* script now.
03:30:06 * thuban nods
03:30:11 <thuban> where are we sending files?
03:34:53 <yts98> I don't know if we should create an IA project and upload to it respectively, or transfer all data to one place to generate magawarcs first.
03:51:37 <yts98> So I'm looking for help from volunteers experienced in running staging servers.
04:01:17 <datechnoman> yts98 - you would want to speak with JAA_ and arkiver_ regarding this. They can assist in getting a formal project put together
04:02:49 <thuban> datechnoman: the time for that is over, site is already dead. we're just talking about how to get manually collected data onto ia in a convenient format
04:06:09 <h2ibot> Yts98 edited Niconico (+228, Game Atsumaru was down): https://wiki.archiveteam.org/?diff=50044&oldid=50015
04:06:10 <h2ibot> FireonLive edited Current Projects (-447, move Egloos to recently finished, remove…): https://wiki.archiveteam.org/?diff=50045&oldid=50024
04:09:10 <h2ibot> Yts98 edited Current Projects (-156, Game Atsumaru finished): https://wiki.archiveteam.org/?diff=50046&oldid=50045
04:13:21 <datechnoman> Ahh gotcha no worries at all. Apologies!
04:13:55 <nulldata> Should we add whatever Android app textfiles is referring to to the Deathwatch list? :P https://digipres.club/@textfiles/110619602655711553
04:14:54 <thuban> datechnoman: none needed! that's probably whom i'd ask about how to format an ia item anyway
04:19:40 <datechnoman> For sure. The experts for sure :)
07:03:19 <pabs> company acquiry https://www.networkworld.com/article/3700616/cisco-to-buy-network-monitoring-firm-samknows-for-better-last-mile-visibility.html
07:03:49 <pabs> two actually
07:09:56 <pabs> https://techcrunch.com/2023/06/26/decentralized-social-networking-app-damus-to-be-removed-from-app-store-will-appeal-decision/
07:43:49 <pabs> https://www.hollywoodreporter.com/business/business-news/siriusxm-to-shut-down-stitcher-podcast-app-1235524707/
13:15:00 <arkiver> yts98: how many WARCs is this?
13:19:53 <h2ibot> Jack Thompson edited Deathwatch (+440, Added Showbuzz Daily and IRL): https://wiki.archiveteam.org/?diff=50047&oldid=50035
14:01:22 <yts98> arkiver: each game produces 1~4 WARCs. I have 12556 files in 62GiB, but other operators including thuban_ and 3 STWP members have claimed more IDs, so they will have more.
14:06:02 <yts98> We also keep the files besides WARCs. Since the script overwrites WARC generated by previous runs, we cannot guarantee that every crawled file will appear in the WARC.
14:06:11 <arkiver> yts98: feel free to upload to IA as items with 1000 WARCs each
14:06:22 <arkiver> note that this is only okey in this case, other cases maybe not
14:29:43 <arkiver> yts98: actually, each of these games, since you also get them outside of WARCs. are they playable somehow easily with the files you archived?
14:29:56 <arkiver> for example like an index.html you can run and then the game plays, or similar?
14:35:15 <yts98> I'm sad to find out that some major browsers reject localhost CORS, so some games should be uploaded to an http server like WAMP to be playable.
14:39:16 <arkiver> got it
14:39:43 <threedeeitguy> arkiver I got almost 3000 warcs, ~60gb. plus the extracted assets yts98 mentioned. The assets are ~67gb and 260,000 files.
14:41:34 <arkiver> alright
14:42:14 <arkiver> yts98: let's do the following
14:42:39 <arkiver> each individual game can be upload in it's own item with mediatype=software
14:43:41 <arkiver> perhaps identifier game-atsumaru-ID, or atsumaru-ID, whatever is more appropriate
14:43:53 <arkiver> and with proper metadata
14:44:30 <arkiver> and, the WARC can be uploaded separately into items with 1000 WARCs in each item
14:44:33 <arkiver> does that sound good?
14:47:28 <yts98> STWP also proposed game-atsumaru-warc-{game_type}-{range_start}-{range_end} .
14:47:29 <yts98> Should we schedule each item to contain nearly 1000 WARCs, or use a predictable range as the index?
15:20:39 <arkiver> yts98: let's say max 1000 WARCs per items, how many items with WARCs would you create in your 'predictable range'?
15:47:46 <yts98> arkiver: Oh, I see: if each game generates an average of 3 WARCs, then 30 items with WARC will be created; If I conservatively map 250 game IDs to an item, there will be 119 items.
15:49:05 <arkiver> yts98: ah that is no problem. let's say max 1000 items with WARCs, each max 1000 WARCs
15:49:28 <arkiver> and the games files for each individual game can be uploaded to a separate item
15:54:15 <yts98> Sound good.
15:54:26 <yts98> I wonder if it's possible for two IA accounts to upload files to the same item? If not, is it safe to share S3 access keys with others?
15:54:56 <arkiver> why do you need two people uploading to the same item?
15:55:00 <arkiver> the WARCs?
15:55:10 <arkiver> IA does not recommend sharing keys
15:58:38 <yts98> Got it, so each operator should decide on the identifier separately.
15:58:56 <nicolas17> hm
15:59:15 <nicolas17> does IA use AWS-like authentication and signing?
16:06:50 <yzqzss|m> nicolas17: <https://archive.org/developers/ias3.html>
16:14:20 <masterX244> afterwards flashpoint should be crosschecked, too. (flashpoint does html games, too so its worth to get the games into that, too). having them already grouped at the IA makes that work easier since no filehunt needed anymore
17:00:43 <JAA> Initial Knowledge Adventure CDN download finished just over 12 hours ago, as predicted. I'm now relisting the bucket and will grab anything that was added in the past couple weeks.
17:01:28 <JAA> 32.23 TiB downloaded into 3.22 TiB of WARCs :-)
17:06:31 <JAA> There were 65907 403s, i.e. files which appear to be inaccessible.
17:08:34 <JAA> Also 14662 404s which I need to deal with later. Many of those are 'directories', which aren't accessible through the media* domains. Some are because I forgor to encode stuff correctly.
17:11:38 <phaeton> Is there a better way to see how long your worker has been at the same task than viewing the folder creation time in the data folder?
17:17:02 <pokechu22> On the warrior web page there's an elapsed time at the bottom right of each task (below the log)
18:06:53 <phaeton> thanks, I should have specified in the docker images
18:07:24 <JAA> Container logs would be another option, but otherwise, not really.
18:07:54 <JAA> Well, I guess you could check when the wget-at process was started, also.
18:14:25 <phaeton> ok, had a long running task and was curious how far back it went... thanks for the proc runtime suggestion. Hadn't considered doing it outside of the container
18:32:40 <fireonlive> https://archive.org/details/archiveteam_tracker_2020
18:32:44 <fireonlive> i can't say i'm not surprised
18:32:47 <fireonlive> but nice
18:33:50 <fireonlive> ah it's not official-dicial
18:45:23 <fireonlive> hello, world!
18:45:53 <fireonlive> has digitize.archiveteam.org been discontinued? "This server could not prove that it is digitize.archiveteam.org; its security certificate is from internetarchive.archiveteam.org. This may be caused by a misconfiguration or an attacker intercepting your connection."
18:46:08 <fireonlive> (ref: https://wiki.archiveteam.org/index.php/Storage_Media)
18:52:25 <pokechu22> I've never heard of that project before, hmm
18:53:05 <pokechu22> It might have become http://fileformats.archiveteam.org/wiki/Main_Page ?
18:53:08 <fireonlive> if you bypass the security warning it shows "Hello archive team.org!"
18:53:29 <fireonlive> hmmm i don't think so
18:53:36 <fireonlive> this one was "a wiki dedicated to digitizing many types of storage media including paper media, CDs, tapes, video, slides, and floppies"
18:53:45 <fireonlive> more on the ingest side
18:54:09 <pokechu22> hmm, some fairly dubious recent captures having captchas and stuff: https://web.archive.org/web/20230301000000*/https://digitize.archiveteam.org/
18:54:11 <fireonlive> though there is a 'file creation software' on there
18:54:18 <pokechu22> but, https://web.archive.org/web/20190212144505/http://digitize.archiveteam.org/index.php/Main_Page does look like a different project
18:54:49 <pokechu22> http://succeed-project.eu/wiki/index.php/Main_Page seems to be dead too :|
18:55:01 <fireonlive> DNS points to 213.184.85.58 which has a PTR of archiveteam.org (which can be set by anyone with no validation but it's a hint at least)
18:55:19 <fireonlive> run on a 'Hosting4Real' server
18:55:19 <fireonlive> hm
18:55:40 <fireonlive> not that it is an emergency or anything i was just curious
18:55:46 <nicolas17> archiveteam.org resolves to the same IP
18:55:54 <fireonlive> i saw it and was like ooh ingest methods
18:56:01 <fireonlive> nicolas17: ah! thanks
18:56:05 <pokechu22> There is a somewhat old export at https://archive.org/details/wiki-digitizearchiveteamorg (which is incorrectly in warczone when it should be in wikiteam it seems)
18:56:26 <pokechu22> based on https://web.archive.org/web/20210329052052/http://digitize.archiveteam.org/index.php/Main_Page I don't think it ever really got much content
18:57:38 <fireonlive> as does wiki.archiveteam.org; so maybe something jrwr ran?
18:58:04 <fireonlive> ooh that logo on Main_Page great though
18:59:22 <fireonlive> ye there's a few interesting pages on there
18:59:26 <fireonlive> but ye
18:59:32 <fireonlive> neat though :)
19:05:29 <arkiver> i pinged jason about it
19:12:35 <fireonlive> ah! thanks
19:12:51 <arkiver> it's been off since last year
19:36:55 <thuban> yts98: it might be easier to keep things consistent if one person consolidates + uploads
20:19:10 <h2ibot> FireonLive edited Tiki (+782, The torches have been blown out): https://wiki.archiveteam.org/?diff=50048&oldid=50023
20:20:10 <h2ibot> FireonLive edited Tiki (-27, fix infobox syntax): https://wiki.archiveteam.org/?diff=50049&oldid=50048
20:50:32 <threedeeitguy> I've no issues with sending everything onto someone else for upload.
23:29:20 <fireonlive> good afternoon good afternoon
23:29:29 <fireonlive> more fires for this hellscape we call earth
23:29:35 <fireonlive> National Geographic lays off its last remaining staff writers: https://www.washingtonpost.com/media/2023/06/28/national-geographic-staff-writers-laid-off/
23:31:17 <fireonlive> earth? life.
23:33:18 <fireonlive> seems like their online edition needs an account to 'see more' https://www.nationalgeographic.com/
23:40:38 <joepie91|m> sigh. and of course the wapo article uncritically repeats the narrative that it has "fallen on hard times" in the same breath as stating it's the most-read magazine in the US.
23:40:48 <joepie91|m> anything to not call out what it really is, I guess