00:25:26 https://github.com/ArchiveTeam/ArchiveBot/issues/564 00:32:14 akamai bad 00:37:07 Hi, I was the one who asked about divested.dev's exclusion a couple days ago or so 00:37:20 would like to note that the AB snapshot showed up for opinionplatform.org but still no snapshots for divested 00:37:27 on WBM 02:12:07 JAA: do we have igsets for drupal sites? 02:13:10 anarcat: Funny timing, I just explained an hour ago in #archivebot why we don't. :-P 02:13:48 The ignore depends on the Drupal version as well as the path prefix. So an igset isn't really very possible. 02:14:03 oic 02:14:07 to pabs i bet :p 02:14:25 AB does actually have code to deal with that mess, but it's broken. :-/ 02:15:35 :( 02:16:03 me 🤝 AB's code 02:33:51 Since divested.dev appears to have some sort of time exclusion but not an explicit whole site exclusion, should it go on the WBM exclusion list wiki? 03:44:17 PaulWise edited Deathwatch (+474, Open Collective Foundation shutting down end of…): https://wiki.archiveteam.org/?diff=51805&oldid=51800 04:03:27 JAA: can you shove Mannie's list above (maybe fix the merged URL) into queueh2ibot? then I'll try to do subdomains for them after lad's ones 04:04:04 AB needs some more capacity :) 04:04:18 soon: waterpipe 04:04:20 maybe 04:04:21 lol 04:04:37 :) 04:04:42 =] 04:04:56 * pabs ponders if SWH would fund one 04:05:03 ooh 04:08:59 I'm guessing not, didn't talk to them about it yet 04:09:41 Seems too far away from their mission. 04:10:41 there was a comment in olasd's DebConf talk that made me think they might 04:10:52 oh? 04:11:27 something like we aren't working on archiving websites but we would like to suport that or something 04:11:39 Interesting 04:11:41 ooh 04:12:06 it was towards the end IIRC if you want the exact wording 04:38:16 pabs: So I can't easily stop what queueh2ibot is running through currently. :-/ 04:38:36 ok, no worries 07:54:34 the bot to queue items was killed due to memory usage. too much memory was consumed by other processes on the machine, not the bot 07:54:53 the bot it back up now 10:16:08 how much would a useful chunk of ab capacity roughly be per month? 10:16:25 or does anyone know when the next "black friday style" offers will come around? ^^" 11:38:54 OOh that's one I can answer, (I run the ak-was-here-hel3 + ak-was-here-hel4 boxes), lemme find some numbers 11:39:23 (Pay for them, JA_A does all the hard work for me) 11:40:42 So we've got one AX41-NVMe and one EX42 NVMe, each one is ~€45 per month at the moment 11:42:20 🤔Hel3 is the EX42 which only has 8 cores vs hel4's 12 11:48:38 thanks! i've not been good with financial decisions lately, but in principle i should be able to handle at least one of those (i'd be starting with only one either way) 11:51:47 i'll think this through for a bit. i'm on vacation in a month from now so i would have time to set it up then. 11:51:55 i think that's a plan :) 12:24:58 fwiw, the EX44 seem to be a great deal, even better than the AX41-NVMe wrt. CPU perf 12:25:02 if you can justify the setup fee 12:31:07 they often have promos for free setup. 12:36:18 r 12:36:20 ops 13:42:23 Yeah I'm tempted to look at replacing/upgrading hel3. But the drawdown time on pipelines can be up to 6 months, so I'd basically need to deploy hel8 and then wait 6 months to cancel hel3 which is quite a cost 14:01:58 might be worth setting it up in a container/vm so you can just migrate it next time? 14:03:35 That's probably a JAA question on how easy it would be to pause/migrate a running pipeline. I suspect the answer is "Not at all" 14:04:17 I knew when I offered these servers that I've gotta keep them going for months (And now years) because some of the jobs can take months 14:10:02 as in qemu does live migrates, so the vm won't even notice it got moved 14:16:50 how does a pipeline deal with powerfailure? 14:17:06 or a reboot for other (important) reasons? 15:03:01 It doesn';t 15:05:13 oof. 15:15:01 so no checkpointing / saving of state? 15:19:23 Nope 15:23:25 Here is the former kiskaVultrJP instance: Linux vultr.guest 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux 15:23:48 Its running Debian 9.9 and has: 15:22:36 up 328 days, 4:16, 10 users, load average: 0.32, 0.22, 0.09 15:34:01 we now have a maximum to the number of concurrently running jobs by a queuing bot in a channel 15:34:15 see !help for details on the maximum 16:21:53 Hel3: 16:21:13 up 310 days, 22:45, 13 users, load average: 5.16, 5.51, 5.80 16:21:53 Hel4: 16:21:35 up 593 days, 18:08, 17 users, load average: 8.88, 9.16, 8.89 16:22:01 I win :P 16:53:18 Congrats! :D 17:34:51 c3manu: are you married? these servers tend to have a pretty bad Spouse Acceptance Factor 17:35:51 So you're saying I'm the perfect candidate... 17:36:15 what’s “married”? 17:43:43 nicolas17: no, i’m not ^^ but that also means my tax bracket is called "get fucked" 18:04:12 o3o 18:08:02 couples are taxed at a lower rate? that seems strnage, surely they save money with economies of scale etc? 18:09:21 i think different countries have different implementations of that, but it does seem weird 19:18:29 just reading up on that a bit, apparently i have a wrong understanding on how tax brackets work here. it's different brackets, yes, but that mostly enables a couple to shift things around a bit. but it's not a difference in principle, as i thought until now. 19:20:23 so nevermind what i said 19:23:59 Yeah my understanding is it's (at least in the US) roughly just letting you choose to merge your money with your spouse and scale up the values for 2 people (though you can also choose to file separately in which case the brackets are for one person), but I haven't looked at exactly how it works. Also remember that getting bumped up a tax bracket generally only means you pay 19:24:01 more in tax for money over that point; tax bracket wise there's no situation where earning an additional dollar will result in an increase of taxes over a dollar (i.e. if you make more money, even after taxes you'll make more money) 19:27:34 had we not broken up we would have become "common law" automatically here... 19:27:43 (but also best to continue in -ot) 19:28:40 I am squarely in the "As long as my girlfriend doesn't know exactly how much I spend then I'll be okay" camp 19:33:22 ;) 19:35:15 if it ever comes up just say 'hey at least it's not heroin or whatever' 20:12:48 AK: Rookie numbers 20:13:03 20:12:59 up 2443 days, 1:52 20:14:13 imer: IP will change though, unless you route everything through another box, which may cause pain. 20:15:07 everyone cross your fingers for firepipe not losing power or whatver x3 21:40:08 JAA: yeah, that's the one downside. there is some hackyness you can do (done that before), assuming it handles the network disconnect and having another ip for outgoing stuff just fine, reverse proxy any incoming (http?) connections to the new ip until dns updates 21:40:55 imer: We're talking about AB pipelines, so the outgoing IP is what matters. 21:42:17 Reason being URLs or cookies that are IP-bound, of course. 21:42:19 is it? not familiar how that stuff works, I was assuming it just does a recursive crawl and then hands it off once finished, ip change shouldn't matter for most things there unless a site restricts session to the ip? 21:42:24 beat me to it. 21:42:33 makes sense 21:42:51 It's basically impossible to know how much of a problem it would actually be without trying it out. 21:47:27 Also, we did experiment with containers before. I wanted to use CRIU to resume things after host maintenance, and that worked fine in tests. The resumption never happened though for unrelated reasons. 22:09:00 Does anyone know if archivebot has grabbed https://dcist.com yet? Or how to find out? 22:10:01 Yes, it was: https://archive.fart.website/archivebot/viewer/domain/dcist.com 22:10:42 Oh that's great, thank you! 22:13:12 I know it would take quite some time to set up, (which doesn't seem like something we're ever going to have at this point) but would getting #Y running be better than another AB pipeline or two?