00:29:15 <nicolas17> JAA: is it possible to set a cookie/header in archivebot?
00:29:22 <JAA> nicolas17: No
00:29:50 <nicolas17> dammit
00:29:55 <nicolas17> <a id="logo" class="logo" href="./index.php?sid=34673d68f5d91e04536838e3b7a446a6">
00:30:14 <nicolas17> if there's no session ID cookie in the request, the response adds a sid= to all links
00:31:00 <JAA> Yeah, we can add a dummy request to https://foro.argenteam.net/?archiveteam or similar at the start to make it set the cookie before fetching the pages of interest.
00:31:13 <nicolas17> oh it does preserve received cookies between requests?
00:31:31 <JAA> Yes,
00:31:35 <JAA> s/,//
00:32:14 <JAA> Setting a value manually probably wouldn't work anyway for the session ID.
00:32:24 <nicolas17> yeah it might be per IP?
00:36:18 <nicolas17> https://transfer.archivete.am/BRoPl/foro.argenteam.net-public-forums.txt.zst okay, I think this will do it
00:38:28 <nicolas17> lmk if you do the !ao< so I add the job ID to the wiki
00:39:04 <JAA> Already submitted.
00:40:26 <h2ibot> Nicolas17v2 edited ARGENTeaM (+61, Add forum AB job): https://wiki.archiveteam.org/?diff=51285&oldid=51284
00:42:14 <fireonlive> v2 on the wiki eh
00:42:20 <fireonlive> oh nvm
00:42:35 <JAA> What was your request rate for the scraping?
00:42:38 <fireonlive> User:... tells all
01:30:05 <nicolas17> JAA: I used 10 parallel wgets, I just timed it and it did 22 req/s
01:31:06 <nicolas17> I see the queue is barely going down, probably collecting lots of page requisites :P
01:31:52 <JAA> Not bad. :-)
01:53:46 <pabs> Barto Nulo|m - #gitgud is mostly about JAA saving git bundles to archive.org (and I think also web issues/PRs to the WBM), but I also lurk there and am doing SWH SCN requests for GitHub users, the SWH SCN API is open to everyone else though, but with rate limits
01:57:40 <h2ibot> FireonLive edited Telegram (-803, remove notable channels section.…): https://wiki.archiveteam.org/?diff=51286&oldid=51176
02:00:11 <that_lurker> fireonlive: Welp thats better than what I did :P
02:00:57 <fireonlive> did you make an edit? xP
02:04:04 <that_lurker> Yep. I just wanted to clarify the section somewhat more, but just deleting it was also on my mind :P
02:34:14 <fireonlive> ah :3
02:51:59 <nicolas17> looks like there's lots of argenteam.net/resources/covers/*.jpg images on forum posts, which means they will be duplicated with the previous job that grabbed the webpages
02:52:21 <nicolas17> hmph, damn dupes
03:47:32 <JAA> nicolas17: And it's done. That was faster than expected.
04:26:13 <nicolas17> JAA: it looks like there's a similarly-sized number of URLs corresponding to forum threads in login-required forums :/
04:26:34 <nicolas17> idk what's up with that
04:26:53 <nicolas17> why make some forums available anonymously, and others require login but *any* login is fine?
04:51:51 <Nulo|m> <pabs> "Barto Nulo - #gitgud is mostly..." <- i have no clue what SWH SCN means
04:52:33 <nicolas17> "doing a Save Code Now request in Software Heritage"
04:53:08 <pabs> yep
04:53:40 <Nulo|m> okay, i believe you already picked them from #gitgud no? otherwise https://pastebin.com/bDAqGRsn / https://pastebin.com/6vk81xnE
04:53:51 <pabs> correct
04:54:00 <pabs> and also the gitlabs JAA mentioned on #codearchiver
04:55:56 <Nulo|m> i'm doing my own overengineered archiving effort for all datasets in argentinian gov CKANs and related
04:56:03 <Nulo|m> https://github.com/catdevnull/transicion-desordenada-diablo in spanish
04:58:44 <nicolas17> https://wiki.archiveteam.org/index.php/Argentina
05:00:00 <Nulo|m> i don't see any of the data hub thingies, here's my list: https://github.com/catdevnull/transicion-desordenada-diablo/blob/master/downloader/config.js
05:00:06 <Nulo|m> there's more, these are the CKAN ones
05:00:10 <nicolas17> yes so maybe we should add them :)
05:00:42 <Nulo|m> it's ~150 GB uncompressed and ~85 GB gzipped
05:01:09 <JAA> Oh dear
05:01:21 <JAA> I've thrown a couple of those into ArchiveBot, but...
05:02:34 <Nulo|m> i am downloading them myself and i will distribute them later (and should probably upload to archive.org) so i'm not personally concerned
05:02:54 <JAA> I'm assuming you're just downloading plain files though, not as WARCs.
05:03:10 <JAA> So no integration into the Wayback Machine.
05:03:13 <Nulo|m> yes indeed
05:03:22 <Nulo|m> i made my own fancy javascript frontend to navigate it
05:03:23 <JAA> But yeah, at least there'll be a copy. :-)
05:03:57 <JAA> datos.gob.ar is either horribly overloaded or hosted on a potato.
05:04:06 <nicolas17> for all I know CKAN might involve POST requests :P
05:04:17 <nicolas17> (I didn't check)
05:04:45 <JAA> It *seems* to work reasonably well in AB, apart from the expected filter faceting.
05:04:45 <Nulo|m> i don't think so,
05:05:19 <Nulo|m> my method is that most of these ones have a /data.json path and those have downloadURLs, it's all GETs. the API is also all GETs (for the ones that don't have the data.json plugin)
05:05:51 <JAA> I was just about to ask what you're fetching exactly. Sounds good then.
05:06:44 <Nulo|m> the data.json (that has most metadata) (or i scrap the api and generate one with less data from that) and all the downloadURLs
05:07:06 <JAA> AB doesn't seem to find /data.json. Not that I'm terribly surprised.
05:07:30 <JAA> But there probably shouldn't be a difference to what's listed on the web pages.
05:07:43 <JAA> Just more efficient to fetch everything that way.
05:07:50 <Nulo|m> yep
05:08:57 <Nulo|m> i'll be uploading my new dump with 147GB as i added a few big ones (they didn't have the data.json plugin so i had to write the glue code), does dropping a torrent here make sense?
05:09:20 <Nulo|m> i'm inexperienced with archive.org, can i just.. upload ~90GB? or do i need permission
05:12:30 <Nulo|m> is https://www.youtube.com/@AudiovisualTelam getting archived? i don't see it in the wiki page
05:13:04 <nicolas17> is it multiple files?
05:13:15 <nicolas17> uploading 90GB as a single file has a big risk of being interrupted
05:13:31 <Nulo|m> they are multiple files of sizes varying very small to a few GB
05:14:04 <nicolas17> uploading 90GB as a single item with multiple files is probably a bad idea anyway, you shold probably do one per server or something?
05:15:08 <Nulo|m> well, i've been taking the nuclear approach and automating downloading and dumping all together because they are so many. but yes, i can upload them separately, can that be scripted?
05:16:11 <nicolas17> https://archive.org/developers/internetarchive/cli.html
05:16:21 <pabs> https://github.com/jjjake/internetarchive
05:16:48 <JAA> (Same thing, to avoid confusion)
05:27:37 <h2ibot> JacksonChen666 edited URLTeam (+367, 2137.pl URL shortener): https://wiki.archiveteam.org/?diff=51287&oldid=51193
05:27:38 <h2ibot> Flama12333 edited BetaArchive (+97, same of ftp access): https://wiki.archiveteam.org/?diff=51288&oldid=38443
05:27:39 <h2ibot> Taka edited Deathwatch (+181, /* 2024 */ Added Wezzy): https://wiki.archiveteam.org/?diff=51289&oldid=51267
05:27:40 <h2ibot> Usernam edited Deathwatch (+128, /* 2024 */): https://wiki.archiveteam.org/?diff=51290&oldid=51289
05:27:41 <h2ibot> Inti83 edited Argentina (+455, /* Memory and Human Rights */ add some new sites): https://wiki.archiveteam.org/?diff=51291&oldid=51278
05:27:42 <h2ibot> Rad XXXack edited Blip.tv (-5, searchable index url fixed): https://wiki.archiveteam.org/?diff=51292&oldid=47802
05:27:45 <JAA> If anyone understands what that BetaArchive edit is trying to say exactly, please edit.
05:29:37 <h2ibot> JustAnotherArchivist changed the user rights of User:Usernam
05:29:38 <h2ibot> JustAnotherArchivist changed the user rights of User:JacksonChen666
05:31:48 <pokechu22> JAA: I'm guessing they're saying something like the FTP interface was shut down and replaced with something else with the same data on April 23, but I'm not sure
05:32:10 <JAA> Yeah, also my guess, but couldn't find anything on the website to confirm it.
05:32:58 <JAA> Oh, should've checked the forums: https://www.betaarchive.com/forum/viewtopic.php?t=44644
05:33:05 <project10> could I trouble you for wiki edit rights on user:Project10 while you're in there, JAA?
05:40:39 <h2ibot> JustAnotherArchivist edited BetaArchive (+149, Rephrase FTP shutdown and add reference; update…): https://wiki.archiveteam.org/?diff=51293&oldid=51288
05:42:39 <h2ibot> JustAnotherArchivist changed the user rights of User:Project10
05:42:40 <h2ibot> JustAnotherArchivist changed the user rights of User:That lurker
05:42:56 <project10> thanks :)
06:44:08 <Nulo|m> off to a bad start re: uploading my dumps to IA:  error uploading /datos.csjn.gov.ar_/[..]/creditos-y-ejecucion-2019-mensual-saf-335---febrero-2020.pdf.gz: Uploaded content is unacceptable. - error checking pdf file
06:44:24 <Nulo|m> i have all files gzipped (irrespective of type) to save space in the torrent and in the server
06:47:49 <nicolas17> yeah at least for PDFs you probably shouldn't...
06:48:11 <nicolas17> IA will let you search text in PDFs, even if they are image PDFs (eg. scanned) by doing OCR
06:48:29 <nicolas17> if gzipped that won't work
06:53:56 <Vokun> If a post on the wiki is quoting something, should a date in the quote still be swapped to the new date format, or should it be left alone?
06:54:43 <Nulo|m> <nicolas17> "if gzipped that won't work" <- why doesn't it just disable that functionality :(
06:54:57 <nicolas17> oh I don't know *why* it doesn't let you upload
06:55:00 <h2ibot> Vokunal edited Kephost.com (+31, /* Closure */): https://wiki.archiveteam.org/?diff=51294&oldid=47946
06:55:11 <Nulo|m> i'll try with the uncompressed copy i guess
06:55:15 <nicolas17> I'm saying even if it let you upload .pdf.gz, it seems like a good idea to not gzip PDFs
06:55:37 <Nulo|m> yeah it's not great, it just was a lazy decision i made
06:56:09 <Nulo|m> thank you all! going to sleep
06:57:01 <h2ibot> Vokunal edited Ghostbin (+52, updated to new date system): https://wiki.archiveteam.org/?diff=51295&oldid=46969
07:02:01 <h2ibot> Vokunal edited Early projects (+26, updated to new date format): https://wiki.archiveteam.org/?diff=51296&oldid=27674
07:03:01 <h2ibot> Vokunal edited Nin.com (+5, /* remix.nin.com */  new date format): https://wiki.archiveteam.org/?diff=51297&oldid=29051
07:11:03 <h2ibot> Vokunal edited Nin.com (+12, /* phm.nin.com */  new date format): https://wiki.archiveteam.org/?diff=51298&oldid=51297
07:18:05 <h2ibot> Vokunal edited Elections/2019 Swiss federal election/Candidates/Luzern (+552, /* Nationalrat */  New date format): https://wiki.archiveteam.org/?diff=51299&oldid=43129
07:32:08 <h2ibot> Vokunal edited Sploder (+35, new url format): https://wiki.archiveteam.org/?diff=51300&oldid=48899
07:32:09 <h2ibot> Vokunal edited Google Business Sitebuilder (+9, New date format): https://wiki.archiveteam.org/?diff=51301&oldid=47446
07:34:08 <h2ibot> Vokunal edited Google Business Sitebuilder (+23, new date format): https://wiki.archiveteam.org/?diff=51302&oldid=51301
07:36:09 <h2ibot> Vokunal edited CVE References (+11, date format): https://wiki.archiveteam.org/?diff=51303&oldid=49862
07:38:09 <h2ibot> Vokunal edited Magazines and journals (+26, date format): https://wiki.archiveteam.org/?diff=51304&oldid=48762
07:40:09 <h2ibot> Vokunal edited MediaFire (+16, date format): https://wiki.archiveteam.org/?diff=51305&oldid=50263
07:42:10 <h2ibot> Vokunal edited Gna!/code and downloads (+13, date format): https://wiki.archiveteam.org/?diff=51306&oldid=29362
07:43:10 <h2ibot> Vokunal edited Gna!/code and downloads (+34, /* Sizes by project */  date format): https://wiki.archiveteam.org/?diff=51307&oldid=51306
07:50:11 <h2ibot> Vokunal edited Canv.as (+36, date format): https://wiki.archiveteam.org/?diff=51310&oldid=47811
08:07:20 <h2ibot> Vokunal edited Ex.ua (+33, date format): https://wiki.archiveteam.org/?diff=51311&oldid=47516
08:09:21 <h2ibot> Vokunal edited Bitbucket (+61, date format): https://wiki.archiveteam.org/?diff=51312&oldid=47800
08:10:21 <h2ibot> Vokunal edited Xanga (+8, /* Xanga 2.0 */  date format): https://wiki.archiveteam.org/?diff=51313&oldid=47740
08:13:22 <h2ibot> Vokunal edited Sweb.cz (+78, date format): https://wiki.archiveteam.org/?diff=51314&oldid=49237
08:15:22 <h2ibot> Vokunal edited TalkTalk (+78, /* Personal Webspace Closure */  date format): https://wiki.archiveteam.org/?diff=51315&oldid=47544
08:16:23 <h2ibot> Vokunal edited 2019 Swiss women's strike (+11, date format): https://wiki.archiveteam.org/?diff=51316&oldid=43131
08:21:26 <h2ibot> Vokunal edited Sola.ai (+7, date format): https://wiki.archiveteam.org/?diff=51317&oldid=47674
08:23:27 <h2ibot> Vokunal edited Mininova (+13, date format): https://wiki.archiveteam.org/?diff=51318&oldid=41223
08:27:30 <h2ibot> Vokunal edited Google Video (+102, date format): https://wiki.archiveteam.org/?diff=51319&oldid=38184
08:28:31 <h2ibot> Vokunal edited Mastodon/Outdated instances/2019-03-24 (+26, date format): https://wiki.archiveteam.org/?diff=51320&oldid=35987
08:29:32 <h2ibot> Vokunal edited Turkey Media Crackdown (+13, /* Endangered Newspapers: */ date format): https://wiki.archiveteam.org/?diff=51321&oldid=30123
08:33:34 <h2ibot> Vokunal edited DNS History (+121, date format): https://wiki.archiveteam.org/?diff=51322&oldid=47777
10:50:33 <thuban> Vokun: imo, if the date isn't otherwise given in the article, do use the datetime template but wrap it in square brackets; if it is, leave the quotation as-is
11:14:58 <missaustraliana> can someone approve my edit to Deathwatch? I added Studio 10 to the axe
11:15:06 <missaustraliana> axe list*
13:34:40 <h2ibot> That lurker uploaded File:Sad-thumbs-up-cat.jpg: https://wiki.archiveteam.org/?title=File%3ASad-thumbs-up-cat.jpg
16:33:21 <nulldata> https://d2iq.com/, an Enterprise Kubernetes Management Platform, is shutting down/selling assets. https://www.theinformation.com/articles/a16z-backed-startup-that-once-rejected-150m-sale-to-microsoft-shuts-down
17:24:00 <Pedrosso> If you have an archive of a site, be it by grab-site or other tools more specific to that site, it's made clear that it can't be trusted to be put into wayback but is it still expected / good to upload as an item?
17:39:47 <thuban> Pedrosso: yes!
17:40:30 <Pedrosso> Awesome. Are there any specific metadata suggestions in the case of grab-site?
17:44:11 <thuban> i don't upload warcs much, so i don't have any, but someone else might
18:13:35 <hhhb> Did anyone noticed that twitter changed its media section? If you are on there and logged in, the media section now displays a grid of images, one for each tweet. If tweet has 2+ images, it only shows the first one. Extracting URLs will only get that image URL since the 2nd and beyond URLs aren't on the HTML until you click on the image, and press
18:13:36 <hhhb> left/right to view additional images
18:16:39 <hhhb> If you want to save tweets, which by the way could fail silently - the WBM itself does not emit any errors, but viewing the playback URL shows that the page itself is loaded, but the tweet content will not load with a message saying "something went wrong" message. You'll want to save nitter front end instead.
18:17:54 <hhhb> if a tweet fails to load on nitter, it will show an error page, I would assume would error 4XX or 5XX, which causes an error to appear when saving the page without the need to check the playback URL
22:09:27 <h2ibot> Vokunal edited ISP Hosting (+860, Updated to new date format): https://wiki.archiveteam.org/?diff=51325&oldid=50970
22:11:27 <h2ibot> Vokunal edited ISP Hosting (+1): https://wiki.archiveteam.org/?diff=51326&oldid=51325
22:12:49 <fireonlive> Vokun++
22:12:50 -eggdrop- [karma] 'Vokun' now has 1 karma!
22:13:28 <h2ibot> Vokunal edited Portalgraphics.net (+32, updated date format): https://wiki.archiveteam.org/?diff=51327&oldid=30338
22:15:28 <h2ibot> Vokunal edited WikiApiary (+59, date format): https://wiki.archiveteam.org/?diff=51328&oldid=49585
22:16:28 <h2ibot> Vokunal edited Twaud.io (+10, /* Announcement */  date format): https://wiki.archiveteam.org/?diff=51329&oldid=28598
22:17:29 <h2ibot> Vokunal edited Xeno-canto (+13, date format): https://wiki.archiveteam.org/?diff=51330&oldid=29439
22:19:29 <h2ibot> Vokunal edited Bayimg (+10, /* Going offline */  date format): https://wiki.archiveteam.org/?diff=51331&oldid=47798
22:20:29 <h2ibot> Vokunal edited ESPN Forums (+10, date format): https://wiki.archiveteam.org/?diff=51332&oldid=28887
22:21:30 <h2ibot> Vokunal edited Instacast (+20, date format): https://wiki.archiveteam.org/?diff=51333&oldid=47933
22:22:30 <h2ibot> Vokunal edited Moegirlpedia (+37, date format): https://wiki.archiveteam.org/?diff=51334&oldid=49240
22:23:30 <h2ibot> Vokunal edited Moegirlpedia (+1, spelling): https://wiki.archiveteam.org/?diff=51335&oldid=51334
22:25:30 <h2ibot> Vokunal edited List of websites excluded from the Wayback Machine/Former exclusions (+8, date format): https://wiki.archiveteam.org/?diff=51336&oldid=51211
22:34:53 <thuban> at some point i've got to go through the 'endangered' category, because i know for a fact that some of them are in fact dead
22:35:33 <h2ibot> Vokunal edited URLTeam (+117, /* Non-warrior projects */  date format): https://wiki.archiveteam.org/?diff=51337&oldid=51287
22:35:41 <thuban> i want to revamp fire drill with my proposed defcon system too, but... should get work done first...
22:42:25 <JAA> Vokun: EDT is UTC-4, not -5. (EST is -5.)
22:56:24 <missaustraliana> Still waiting for someone to approve my edit. Not sure if theres an ID type thing or reference number i need to provide
23:00:39 <h2ibot> Amphitryon edited URLTeam/Dead (+0, /* Dead or Broken */ Update last known…): https://wiki.archiveteam.org/?diff=51338&oldid=51192
23:00:39 <h2ibot> Missaustraliana edited Deathwatch (+160, Add Studio 10): https://wiki.archiveteam.org/?diff=51339&oldid=51290
23:01:41 <h2ibot> HHHB edited Talk:Twitter (+1683, /* Twitter have changed its media section, it…): https://wiki.archiveteam.org/?diff=51341&oldid=50606
23:02:38 <h2ibot> JustAnotherArchivist changed the user rights of User:Amphitryon
23:03:38 <h2ibot> JustAnotherArchivist changed the user rights of User:Missaustraliana
23:03:39 <h2ibot> JustAnotherArchivist changed the user rights of User:HHHB
23:03:40 <h2ibot> JustAnotherArchivist changed the user rights of User:Magmaus3
23:06:14 <that_lurker> Seems like JAA got tired of approving all the rapid edits :3
23:09:02 <JAA> The mod system was never intended as a barrier to submission but as a spam filter without a registration lock. So... yeah :-)
23:09:28 <JAA> I wish the mod interface had a button to directly make someone automoderated.