00:02:36 cc pokechu22? 00:03:26 There's been an archivebot job on all of macgui.com but it's been blocked for a long time because the site has banned us 00:04:08 A new one probably is a good diea 00:04:17 aha, i thought i remembered some discussion 00:04:30 would a new job not also be b&? 00:04:56 If we run it slower it'd probably be fine 00:05:21 sounds good 00:22:09 There aren't that many files, even a really slow download rate would probably be fine 00:25:58 qwertyasdfuiopghjk: Trump's jail page is live fyi 00:28:49 qwertyasdfuiopghjkl: ^ 00:34:15 Not sure if I should have my warriors on xuite because we have less of it storage wise or gfycat because we have less of it item wise 00:35:08 flashfire42: xuite, gfycat should be fine with current workers on it 00:35:19 gfycat is tracker limited as well 00:37:53 ^ so cant go any faster than it currently is 00:41:00 mugshot: https://www.cnn.com/politics/live-news/trump-georgia-surrender-indictment-08-24-23/index.html 00:46:30 direct link: https://www.cnn.com/politics/live-news/trump-georgia-surrender-indictment-08-24-23/h_4f2ed97bfb6e6c9bd51e556c1ea0a6c0 00:59:34 thoughts on removing the sticker plug from https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior ? 00:59:58 "This is an unofficial product, but if there is enough interest, a portion of proceeds will be dona" (I'm not cutting it off, that's literally where the redbubble description ends.) 01:03:26 looks like it was added in 2015 by "Realhumanbeing" https://wiki.archiveteam.org/index.php?title=ArchiveTeam_Warrior&diff=24033&oldid=23582 01:04:07 https://wiki.archiveteam.org/index.php/Special:Contributions/Realhumanbeing hmm 01:04:15 was just about to link that :3 01:04:18 but ye 01:04:19 idk 01:09:52 imho, as long as redbubble still delivers, it's fine to leave it up 01:14:51 They do, I got a shirt from them not too long ago 01:26:09 qwertyasdfuiopghjkl: When you're around, please crawl this - Trump's jail info page: http://justice.fultoncountyga.gov/PAJailManager/JailingDetail.aspx?JailingID=1472740 01:27:35 Looks like someone else already did it: https://web.archive.org/web/20230825010801/http://justice.fultoncountyga.gov/PAJailManager/JailingDetail.aspx?JailingID=1472740 01:27:56 oh nice 01:32:52 Do I understand this right? Someone uploaded the warrior logo to Redbubble, but other than that, they're not involved as Redbubble directly prints the stuff whenever it's ordered? 01:33:32 qwertyasdfuiopghjkl: Awesome 01:33:40 I also wonder about the copyright situation there. I don't see any licence on https://wiki.archiveteam.org/index.php/File:Archive_team.png at least. 01:33:59 JAA: AIUI, yes. they also get a kickback 01:34:34 so just upload file and collect cheques 01:34:46 Lovely! 01:34:53 *creates redbubble account* 01:35:48 i guess someone came by *checks image link* ~3 years later and decided they'd upload it there and edit the wiki 01:36:08 and 'if money from here is enough perhaps can donate it back to AT' 01:36:09 lol 01:36:12 Lmao. Why would anyone use something so obscure as the AT logo... on generic consumer products? 01:36:38 Do you hate free money? 01:36:38 you for example might slap it on your laptop 01:36:58 or someone could hand them out after a passionate talk about archiveteam kinda thing 01:37:07 Well, sure. I guess I might, heh 01:37:13 They also uploaded the ReactJS logo and formerly the CoffeeScript logo. 01:37:37 The latter was removed at some point. 01:37:47 interesting 01:38:50 Unless that person is familiar with archiveteam, seems completely random. And if they are familiar, chances are they'd be in here already, no? 01:39:30 Or at least they would've been around that time. I don't see anything in the logs. 01:40:40 one edit to a angelfire/project page (i don't think there was approval back then? maybe there was?) then after that right to redbubble 01:41:12 No, the mod thing only exists since about three years. 01:41:24 ah ok 01:41:33 i assume after the m*****f***** incident 01:41:34 Previously, there was a safeword on signup and then you could just edit. 01:41:51 Nah, much later, around the time we also got rid of EFnet. 01:41:58 ahh 01:42:38 The Angelfire edit was also two years earlier than this sticker thing. 01:42:59 ahh yes it was 01:45:49 Based on how I understand it, this can go to hell and we should get Redbubble to take down the listing, too. I'd have a different opinion if this was run by a regular, but like that? No. 01:46:18 Might not be a terrible idea to ask Jason about it though. 01:46:55 Not that I have much of a say, given I've only been volunteering here a few years. But, would it be time for an upgraded logo anyway? 01:46:59 seems sensible to me 01:47:14 it's kinda a 'sub' logo 01:47:22 Also to sort out the copyright/licence matter. I assume Jason knew Penelope or similar. 01:47:23 but i like it :3 01:47:31 knew/knows* 01:47:48 It's cool, agree, but it seems very 2010s or earlier. 01:48:02 HP_Archivist: I don't think anyone will stop you if you want to design one. :-) 01:48:58 :3 true 01:49:06 JAA: I am a terrible draw-er. Most you'll get outta me is a stick figure or two, ha 01:49:11 i like the logo, chasing trends is for suckers 01:49:24 here's our masthead logo for reference: https://wiki.archiveteam.org/index.php/Talk:Main_Page#Logo_ideas 01:50:18 hmm I have been thinking and we are gonna lose webs sooner than we will lose the orange isp sites. should I focus more on some webs stuff even with the knowledge we did a pretty good grab of webs a few years ago 01:50:44 hmm 01:50:57 i think it depends how much we expect webs to have changed over the years 01:51:25 The orange web pages are fairly obscure but webs has had people updating their pages and making new pages up to and including in the last few days afaik 01:52:09 (no license on that one it seems either) 01:52:17 (well that one or its variations) 01:53:40 HP_Archivist: Yep, same. In general, if I had to guess, I'd say there aren't many designers here. Just doesn't seem like a demographic we'd attract. 01:56:19 I don't like logo changes for the sake of logo changes and newer ones are almost always worse 01:56:28 Should #archiveteam channel just be renamed to #at-announcements / #archiveteam-news or similar? Every third message is always telling the users to go here to discuss further whatever is it that they said 01:57:12 On the wiki it says that the channel is for announcements but the default name obviously makes new people forget 02:00:11 i was going to suggest an entrymsg but that already exists lol 02:05:03 anyway... the sticker bootlegging is slimy-but-harmless*, and i don't support going after it unless we intend to offer an 'official' replacement (even an equally crappy redbubble replacement) 02:05:10 * if penelope shenck wants royalties on a likely-single-digit number of sticker sales, i will personally cut her a check 02:07:51 https://twitter.com/mattjay/status/1694757805838811473 / https://nitter.net/mattjay/status/1694757805838811473 02:08:10 "Danish hosting giants CloudNordic and AzeroCloud have been hit by a massive ransomware attack, resulting in a catastrophic loss of customer data." 02:08:26 Yup, I archived what was left of their websites earlier. 02:08:37 "Recovery instructions include sending customers to the Wayback Machine since their own backups are hosed." 02:08:39 JAA: nice :) 02:08:58 It was basically only the homepages with the notices quoted there. 02:09:04 Everything else was gone. 02:09:24 :( 02:13:29 https://www.nordicbots.com (long-running IRC bot service on QuakeNet) had all bots go down, and suffered a total data loss with all backups encrypted as well. I wonder if this is the same thing 02:15:53 "Things aren't looking good. Harddisks _and_ backups have been encrypted by a crypto locker. Sorry. Looking into it." 02:16:03 might be worth an AB 02:16:30 > Latest news 02:16:31 > 013-08-30 02:16:36 > 2013-08-30 * 02:16:45 Yeah, probably worth that anyway. 02:20:15 ye :x 04:36:35 HP_Archivist: Current status on saving those pages: https://transfer.archivete.am/inline/wpkNM/list 04:36:37 I'll check the not found ones again later today to see if they've been added. 05:51:39 hey, wowturkey archiver says "paused". any issues? 05:52:43 expected item count for wowturkey is about 9.4M, not 9.15M 05:58:10 PaulWise edited Deathwatch (+249, Feb 29 Amazon Honeycode shutdown, AB jobs launched): https://wiki.archiveteam.org/?diff=50584&oldid=50575 06:06:43 :) 06:30:45 erkinalp: the websocket for the archivebot dashboard is currently misbehaving. Jobs are still being worked on, but the page only updates when refreshed, not automatically. 06:45:29 pokechu22: is there any page that we could watch the progress for that fix? 13:07:57 "The nordic cloud experts" 13:11:36 tbh this got me wondering 13:11:41 what if IA got hit with this 13:14:44 The easy defense is to have data be append only, so no overwriting or deleting, not sure how data distribution at IA works internally though 13:16:15 well, "easy" as things at scale always are ;) 13:22:01 shouldn't the secondary backup be minimally accessible, if they somehow managed to wipe secondary backup, then they must have done some massive mistakes 15:00:46 qwertyasdfuiopghjkl: Sounds good 15:00:59 I think that's everyone, in your list 17:22:24 wowturkey archival still going strong 17:24:15 btw there's a quirk of wowturkey that's **very important** for us: when the website closes for db vacuum, it returns a static error page with the text "bakım, temizlik, badana vesaire nedeniyle site kapalı, birkaç dakika içinde yeniden açılacak" 17:25:06 qwertyasdfuiopghjkl: Along similar lines, I heard last evening that Trump's mugshot was going to be released in higher resolution today by Fulton County Jail. I'm not sure if they will post on their site or elsewhere. But a high resolution crawl of it from the 'official source' should be done. 17:25:35 I'm AFK for a few hours. 17:25:48 which means "closed for maintenance, cleanup, paint etc and reopen in a few minutes" 17:26:14 if there are any such responses in the archive, those individual pages will need to be manually recrawled 17:27:57 that "badana vesaire..." page is meant as a 503 page 17:32:12 HP_Archivist: All the previously missing ones have appeared and been saved: https://transfer.archivete.am/inline/8iDFM/list 17:36:30 sorry, if there were any replies to the "badana vesaire" thing, i've missed that as my client crashed 17:42:18 There were no replies to that yet. If that page actually has the http status code 503, ArchiveBot will automatically retry it later. 17:46:59 Also, this channel has a public log at https://hackint.logs.kiska.pw/archiveteam-bs , so you can check that to see any messages you missed while disconnected. 17:49:29 erkinalp: do you know if the site also gives a different HTTP status code when that happens? If it's a 5xx error code it'll be retried 17:51:35 i need to check that out 17:54:42 No 50x responses from wowturkey.com in the log of 3di34a3v4nwzjuejzb82e336n. 19:05:04 need to look at that "badana vesaire" thing at 2UTC to 2:05UTC, that's normally when it vacuums 19:42:02 either everything has been stalled for an hour or the websocket stats are stuck 19:44:21 looking stalled on my end :( 19:46:24 @ERROR: max connections (-1) reached -- try again later 19:46:28 think we hit the wall for now 19:46:35 I think IA is struggling again 19:46:49 can concurrency be adjusted on an already running docker-grab or can it be set/reset only on container create/recreate? 19:46:55 all abord the struggle bus 19:47:27 s3 stats seems to be recovering slowly, so maybe it'll catch up soon? 19:47:34 phaeton: need to recreate afaik 19:48:27 that's what I thought. thanks for confirming 19:48:52 qwertyasdfuiopghjkl: You're awesome, thanks for doing all of them 19:49:27 yeee 19:49:29 qwertyasdfuiopghjkl++ 19:49:36 rip karma bots :( 19:56:25 The image here on this page used by the Associated Press seems of higher quality than the one released last evening. The Fulton County Jail text is legible and sharper 19:56:36 https://apnews.com/article/mug-shot-donald-trump-indictment-839920116a244df3e55bdedf33820a80 19:56:56 Image link: https://dims.apnews.com/dims4/default/9f3bcb1/2147483647/strip/true/crop/2700x2700+0+0/resize/1440x1440!/format/webp/quality/90/?url=https%3A%2F%2Fassets.apnews.com%2F92%2F03%2F069ad1da98c67475656d4bf59318%2F8a78038c24a84b58846d4024c2e25136 19:57:34 Trump himself is still blurry somehow 19:59:05 they don't seem to have the best camera there 19:59:11 that or they don't use it well 19:59:19 the other photos are not well taken 19:59:46 Yeah, or it's just terrible lighting, too 20:00:01 ahyeah 20:00:31 several newspapers including AP posted AI-upscaled bullshit 20:01:03 nicolas17: Do you have a source for that? 20:02:20 https://assets.apnews.com/92/03/069ad1da98c67475656d4bf59318/8a78038c24a84b58846d4024c2e25136 20:02:56 look at the logo, that's clearly upscaling 20:03:33 oh, gross 20:03:57 but there's artifacts all over the place on his face 20:05:07 feels like some failed ai upscaling 20:05:20 it upscaled the artifact too 20:05:32 I just commented on the legibility of that text (above). I mean, it could be an upscaled version. But if you look at the dimensions of that photo compared to what was posted online last evening, it IS larger. 20:06:43 Honestly, unless someone has evidence of upscaling, I'm inclined to take this at face value. It's the AP, too. Something tells me they would mention that below the photo used. 20:08:24 Also - that 'artifacting' is there in the one released light evening. It's just an overall bad photo with bad lighting. But this one from the AP is definitely of higher resolution. 20:13:30 "it could be upscaled, but you gotta admit it's larger" what. 20:14:27 nicolas17: I'm just trying to archive the highest resolution that was officially provided by said original source. 20:14:50 yeah a friend is trying to obtain that 20:14:55 AP is not an original source 20:14:56 If you compare Trumps face from the first released photo yesterday to the one in that AP link, his face and those artifacts look the same 20:15:54 apparently you can get it from http://justice.fultoncountyga.gov/PAJailManager/JailingDetail.aspx?JailingID=1472740 but I think that needs a US IP address, it doesn't load for me 20:16:32 That errors out for me, too, and I'm in the US 20:16:41 That AP image is so obviously upscaled and smoothened it isn't even funny. 20:17:18 JPEG is lossy so you should convert it to PNG if you want the highest resolution :D 20:17:59 PNG? But BMP files are larger! 20:18:02 JAA: https://media.discordapp.net/attachments/244190952107933697/1144723192456429658/Screen_Shot_2023-08-24_at_7.png 20:18:11 Well, okay, I'm wrong then, heh. I still regard AP as the last bastion of hope for new reporting. Or, so I'd like to think 20:18:17 news* 20:19:20 nicolas17: If you can get the link for us that leads to Fulton County's site providing the image, we can get it crawled. But that link errors out for me, too. I searched the site and didn't find a section for mugshots 20:26:21 Btw, nicolas17 - you were right about source being with Fulton County Jail. Usually though, they release 'source' to the news outlets and in turn they use that in their reporting. My thinking was if we couldn't get source from, well, source, then next best would be a trustworthy news outlet 20:28:22 I have a US ip address and http://justice.fultoncountyga.gov/PAJailManager/JailingDetail.aspx?JailingID=1472740 doesn't have the image. (Also, that link only works if you find it via the search page - directly using the link doesn't work. The site is super jank.) 20:29:05 "I'd like to track down the OG, but you have to request it by email after filling out a legal form" 20:29:06 see https://web.archive.org/web/20230825010801/http://justice.fultoncountyga.gov/PAJailManager/JailingDetail.aspx?JailingID=1472740 (saving that requires doing some jank cookie stuff, see discussion yesterday) 20:37:43 I don't know it any regular citizen can request the image or if it's only reserved for media and news outlets. Not sure the legality of it. 20:37:59 Mugshots are obviously public record, but no idea about requesting copies 20:42:16 https://www.nfoic.org/georgia-foia-laws/ if you can get someone in GA to ask 20:42:35 i wonder if it's on a press release hidden away somewhere 20:42:41 or hm 20:42:51 oh could just be everyone FOIA'd it? 20:42:54 interesting 20:44:44 Good point, fireonlive 20:46:33 > Release of booking information to include mugshots will occur daily at approximately 4:00 p.m. (EST) via media advisory. 20:47:05 From https://fcsoga.org/wp-content/uploads/2023/08/FCSO-Media-Advisory__._Election-Fraud-Defendants-Booking-Update-8.23.23.pdf linked on https://fcsoga.org/press-releases/ 20:50:14 Thanks JAA, and I think this is the form to request one? https://fcsoga.org/wp-content/uploads/2023/08/FCSO-Letter-of-Affirmation-Booking-Photos-8.21.23.pdf 20:50:40 Something like that, yeah. 20:52:16 I mean, there's nothing wrong or no harm in doing so. But I'm still leery about having my actual name associated with Trump in any way or context, lmao. 20:52:33 Someone else can if they feel so inclined for archival purposes... 21:26:56 HP_Archivist: 21:27:01 "so here's the raw file that was sent out to the press" 21:27:06 https://cdn.discordapp.com/attachments/1144306432628502669/1144740140200243342/DTJ_A.png 21:27:43 >the logo was, unsurprisingly, added separately, apparently using Canva (based on metadata) 21:27:44 >the JPEG macroblocks present indicate that the original resolution was somewhere around 454x454px 21:27:46 >(there are no macroblock artefacts on the logo) 21:27:47 >and the actual exported resolution is 1080x1080px 22:01:09 gmsracing.net - "a former championship-winning organization, plan to cease operations at the end of the 2023 NASCAR Truck Series season." https://beyondtheflag.com/2023/08/24/nascar-gms-racing-shutting-end-2023/ 22:48:58 nicolas17: Unless you have solid evidence for this, we can agree to disagree. Borderline conspiratorial, and, to borrow a quote from a famous scientist, "Extraordinary claims, require extraordinary evidence". 22:49:59 what is *your* theory? that the image this guy got from the .gov site is downscaled and AP has the full original resolution? 22:53:44 nulldata: that's squarespace and the sitemap has 610 URLs - should be easy for archivebot 22:55:02 What nicolas17 wrote sounds about right to me. The photo is clearly much lower resolution than the logo. 22:55:26 My only question would be: ... but why? 22:56:13 Then again, the other photos were horrible as well, so Hanlon's razor probably applies. 22:56:16 JAA: thing is, the jail seems to have upscaled it from whatever was the original (454?) to 1080x1080, but AP blew it to 2700x2700 22:56:49 and the latter is what shows signs of generative AI gone wild 22:56:50 Yeah, that's what I'm saying, JAA. If nicolas17 is right then pretty much wtf/why is what I'm wondering 22:56:52 Yeah, that too. 22:57:38 Taking a mugshot at that resolution in the first place makes no sense to me though. Maybe they didn't want to release the original photo for some reason, so they downscaled it, then upscaled it again. 22:57:47 The jail text does not look superimposed. It looks like it's part of the wall. I mean, I'm no expert on spotting fake / AI pics, but nobody else agrees? 22:59:40 Wired kinda addresses this https://www.wired.com/story/trump-mug-shot/ 23:01:02 "“I think it’s a certainty there will be context collapse around any prominent image that can be interpreted or misinterpreted by different communities, both wilfully and accidentally,” says Sam Gregory, executive director of Witness, a nonprofit organization focused on using images and videos for protecting human rights. And with numerous versions of a Trump mug shot circulating online, people may 23:01:02 have different memories and associations of the historical event. “We’ll remember the one that we saw in a context that made it memorable to us,” Gregory says." 23:01:04 Both types of artefacts, the regular upscaling in the 'original' image and the AI stuff in the AP image, are pretty obvious. 23:01:09 It looks superimposed to me, particularly with regards to the JPEG artefacts 23:01:24 But it's hard to explain why. 23:01:34 https://i.kym-cdn.com/photos/images/original/000/024/183/500pxShopped.jpg 23:02:33 JAA: Lol ^ 23:03:17 it's the only part of the image that doesn't show signs of chroma subsampling when I split it into HSV 23:03:56 also... what's the copyright status of that mugshot? I know government images in Florida are public domain but most other state government images aren't, but mugshots might be special? 23:04:45 Wikipedia is treating it as fair use and has some context on the description, it seems: https://en.wikipedia.org/wiki/File:Donald_Trump_mug_shot.jpg 23:10:28 Myusernameisanything edited URLTeam (+141, /* "Official" shorteners */ mod.lk): https://wiki.archiveteam.org/?diff=50585&oldid=50543 23:10:29 Vokunal edited Frequently Asked Questions (+321, Added a section on why uploads are less than…): https://wiki.archiveteam.org/?diff=50586&oldid=50275 23:11:58 According to this post on Reddit, the image unofficially leaked right before media outlets were showing it: https://www.reddit.com/r/pics/comments/160j0z0/donald_trumps_mugshot/jxmnluj/ 23:12:57 So, the one that leaked didn't have the logo on it. Idk how anything leaks in that short amount of time 23:13:41 looks like a photo of a screen 23:14:14 But to nicolas17' credit, I now see how the logo is superimposed 23:14:24 Yeah it does 23:14:43 I tried to make the image full-lightness full-saturation keeping only the hue 23:15:05 and the result looked fucking terrifying, like his face was a ball of fire 23:15:06 >.> 23:15:16 Because we don't know the setup and number of personnel at this jail, we might never know how it leaked, exactly or what the sequence of events for pre-release, leak, and official release came to be. 23:15:35 e.g. who had access to the pic before the public/media did, etc 23:16:13 Kinda odd though that no media outlets mention it leaked before the media was able to show it officially 23:19:11 nicolas17: A sub-comment on that post, "I think the fact that it doesn't have the seal adds some credibility to it being a leak. It would make sense the seal is added when they release the mugshot to the public, but they keep a clean one internally." 23:19:38 Which makes sense 23:43:40 hello! 23:44:18 i am new to archiveteam, i once ran the early version of the bot on a VPS but it wasn't a good idea since it was just a very cheap one 23:45:01 are there any laws that would make it difficult for me to do this from my home in EU/DE? 23:46:31 I believe there are some people around here in the EU that do run it from home; I run mine from home in the US. This doesn't account for relevant laws though which I haven't researched 23:47:22 Some sites will block/rate limit you if you request pages too fast, and some archiveteam projects tend to get you blocked on a lot of sites (the URLs project in particular warns about this) 23:47:24 well, there would be websites like project gutenberg or the internet archive which are kind of restricted from germany 23:48:04 Germany restricts access to Archive.org ? 23:48:06 oh, and can i find more information about that in the wiki? 23:48:21 hmm, some items would be illegal to download 23:48:41 Most projects target specific sites. URLs aka #// is really the only one that 'might grab anything' and requires care in that regard. 23:49:09 ah, okay 23:49:16 i cannot get myself blocked too much 23:49:21 my isp would be angry at me 23:51:59 would it wear down a SSD quickly? it would probably process many files 23:52:11 i just got a new one for my raspberry pi 4 23:53:01 It's not impossible that the targeted sites serve content that's illegal in some jurisdictions. Beyond some scale, it's basically unavoidable. Whether this is a concern for you depends on your level of paranoia. It shouldn't be a problem since the chance of hitting such content is generally very small and being part of an automated retrieval is usually somewhat protected legally (although Germany has 23:53:07 a special kink for shitty court decisions in that area, so...). HTTPS being very common nowadays also helps, although we sometimes have to archive old sites which lack it, of course. 23:53:23 the warrior/grabber currently can't run on ARM processors like raspberry pi 23:53:35 i see! 23:53:39 i have a PC too =) 23:53:42 ...why can't it? 23:54:05 because nobody took the work to extensively test if wget-at produces valid results 23:54:11 maybe because of some instructions that ARM cannot do? 23:54:29 SSDs get a decent amount of wear, but you can use a tmpfs to do it in RAM instead on most projects. I'm not sure this has been documented on the wiki yet, but quite a few people are using it. 23:55:12 JAA: i can try to use the tmpfs idea with a lot of google 23:55:32 i still have regular hdds 23:55:32 Someone can give you the right options for Docker for sure. I don't have them handy right now. 23:56:09 ah that could be really nice, thanks JAA 23:57:07 do i need to be an "expert" to test wget -at on my raspberry pi? 23:57:27 i have a 4B model 23:57:33 and i use DietPi 23:57:54 Please don't attempt to do it. 23:58:09 Krume: "--mount type=tmpfs,tmpfs-size=1G,destination=/grab/data" (1G being a safety limit, most projects use very little) 23:58:11 okay! 23:58:12 Yes, it requires a lot of knowledge about how WARCs and zstd work. 23:58:33 Why would a program like that produce inconsistent results based on CPU architecture? 23:58:38 i don't know zstd and WARC was something familiar which i don't remember right now 23:58:57 thank you imer 23:59:37 appledash: Because it's very easy to write C code that depends on undefined behaviour, and it was written and tested on x86. 23:59:40 i will copy it for now, since i only have windows installed and i need to reinstall linux again 23:59:50 * appledash scared