00:04:37 <fireonlive> nitter.net having issues?
00:29:17 <Nulo|m> how does grabsite get lists of urls to scrap apart from crawling the site itself?
00:30:10 <JAA> Sitemaps
00:31:03 <JAA> fireonlive: Not that I'm aware of. I think it was more of a 'this domain is neat' comment.
00:31:38 <Nulo|m> JAA: omg i can't believe i missed that, thanks
00:32:41 <pokechu22> Nulo|m: and more generally (at least assuming the same behavior as archivebot), it gets it both by trying example.com/sitemap.xml and by looking for Sitemap: directives in example.com/robots.txt
00:36:36 <fireonlive> ahh ok :3
09:36:38 <h2ibot> Inti83 edited Argentina (+81, /* Educational Websites */): https://wiki.archiveteam.org/?diff=51435&oldid=51354
09:36:39 <h2ibot> Dylan edited SubRocks (+379, updated URLs, added more information and some…): https://wiki.archiveteam.org/?diff=51436&oldid=50515
09:41:39 <h2ibot> Exorcism edited SubRocks (-4): https://wiki.archiveteam.org/?diff=51437&oldid=51436
13:50:30 <h2ibot> Exorcism edited Arto (+36): https://wiki.archiveteam.org/?diff=51438&oldid=25934
14:15:35 <h2ibot> Exorcism uploaded File:Facebook-screenshot.png: https://wiki.archiveteam.org/?title=File%3AFacebook-screenshot.png
14:15:36 <h2ibot> Exorcism edited Facebook (+33): https://wiki.archiveteam.org/?diff=51440&oldid=49532
14:25:46 <ctag> Does grab-site not stop crawling?
14:26:10 <ctag> I started it against that astronomy website, and for a while the web UI shows "0 crawl" but now it's back
14:26:16 <ctag> Has been running for over a day
14:38:18 <that_lurker> You can check the todo database to see if it is still going. Most sites tend to take a few days
14:58:36 <that_lurker> also its a good idea to monitor the grab every now and then in case its stuck in a loop or some other spam url so you can add them to the ignore
15:37:06 <ctag> Oh, it looks like its stuck crawling the calendar widget
15:37:19 <ctag> This site is tiny, shouldn't take too long to grab all the pages.
15:37:45 <ctag> I'm running it from docker, will try to figure out the todo database, thanks lurker
15:40:53 <joepie91|m> calendars, wrecking site grabs since web 2.0
15:44:06 <ctag> Should I just let it crunch? The calendar widget probably has a finite range, right?
15:44:10 <ctag> .. Right?
15:44:37 <joepie91|m> nope
15:44:43 <ctag> :|
15:44:52 <joepie91|m> most calendar widgets are infinite and generated on-demand
15:45:01 <joepie91|m> they're a famous tarpit for archival efforts
15:45:18 <ctag> It appears to be crawling backwards, I'll let it get to the early era of the site and then cancell it
15:45:24 <ctag> Thanks joepie
15:45:49 <ctag> Oh, I was going to try wget again too
15:46:04 <ctag> And the redirect might work, if anyone with AB can get it for me
15:46:21 <that_lurker> just add the calendar url to the ignores with regex
15:46:57 <that_lurker> so grab-site will ignore it and still grab everyting else
15:53:25 <ctag> Awesome, that got it back out of it, thanks!
17:09:17 <joepie91|m> may have already come up here, but: https://bitbang.social/@Geekman/111666278406724944
17:48:03 <fireonlive> DLoader pls
17:48:28 <fireonlive> ah it was matrix all along
17:57:32 <nulldata> Yay splitting of nets!
17:58:09 <fireonlive> matrix says hello and goodbye and hello and goodbye and hello
17:58:11 <fireonlive> :p
17:58:40 <fireonlive> (i don’t have the scroll back to see the actual number)
18:01:15 <nulldata> I believe someone mentioned a ddos in #hackint, but with the number of joins and leaves I can't see it anymore lol
18:04:25 <fireonlive> ahh boo 😒
18:04:45 <fireonlive> ddosing irc is so 90s
18:07:23 <hexa-> nulldata: clients in 2023 should really support smart filtering of join/parts
18:08:50 <nulldata> hexa- The Lounge collapses them, but they still count toward the history limit loaded in memory
18:09:27 <nulldata> And for some reason my history limit is set low, fixing that now
18:11:32 <hexa-> rough
18:11:52 <hexa-> but yeah, udp ddos against a closed port
18:14:01 <nulldata> Fixed - now I can see that it was you that posted it was a ddos in hackint :P
18:26:43 <fireonlive> ahh lame :(
18:26:58 <fireonlive> and yeah TL is special that way lol
18:42:00 <nicolas17> ugh
18:43:29 <nicolas17> can someone politely tell DLoader (*cough* mode +b) to fix their connection
18:45:53 <fireonlive> b with a redirect to #fixyourshit ?
18:45:59 <fireonlive> :3
18:46:28 <fireonlive> .op
18:46:35 <fireonlive> i tried :D
19:30:00 <nulldata> I guess DLoader isn't *🕶️* DoneLoading
19:35:33 <fireonlive> yeeeeaaaaaahhhhhhhh 🎶
22:33:07 <that_lurker> One of the reasons why znc's flood protection is nice
23:46:04 <katia> i have a by-no-means-complete dump of some37c3 ftps,  468,162 files, 4,753.5 GB
23:46:41 <katia> some of the exclusions here might have been followed: https://github.com/katlol/37c3-ftp-mirror/blob/main/exclude.txt
23:48:43 <JAA> arkiver: ^ Think IA would take that?
23:51:12 <fireonlive> **151.217.62.31** :(
23:51:14 <fireonlive> :P
23:59:26 <hexa-> fireonlive: is this the gay porn ftp ip? :D
23:59:42 <fireonlive> yeah :D
23:59:46 <hexa-> hehe
23:59:53 <fireonlive> :3