02:22:20 <fireonlive> via #nanog: <+rss> Anyone got a contact at OpenAI. They have a spider problem.: Posted by John Levine on Apr 10 As I think I have mentioned before, I have the world's lamest content farm at https://www.web.sp.am/ . Click on a link or two and you'll get the idea. Unfortunately, GPTBot has found it and has not gotten the idea. It has fetched over 3
02:22:20 <fireonlive> million pages today. Before someone tells me to fix my robots.txt, this is a content farm so [...] https://seclists.org/nanog/2024/Apr/63
02:22:26 <fireonlive> let's hope we don't stumble on it :p
02:51:46 <nicolas17> >content farm
02:52:02 <nicolas17> it's easy to solve that issue
02:52:04 <nicolas17> shut it down
02:52:40 <fireonlive> i think it's intended to trap bots
02:52:47 <fireonlive> >IECC ChurnWare 0.3
02:53:00 <nicolas17> well, it worked?
02:53:48 <nicolas17> I don't understand
02:54:12 <fireonlive> hmm maybe he's looking for bad bots? i'm not too sure either
02:54:13 <nicolas17> "I made a set of websites linking to each other to trap bots in a loop following those links, now a bot got trapped in a loop following those links, how do I stop it"
02:55:49 <fireonlive> https://mailman.nanog.org/pipermail/nanog/2024-February/224864.html
02:56:08 <fireonlive> >A few years ago the bingbot got trapped but fortunately I knew someone at Microsoft who could pass the word. He reported back that while he could not go into detail, there was a great deal of animated conversation at the other end of the hall, and shortly after that it stopped.
02:56:11 <fireonlive> lol
02:56:54 <fireonlive> ah found his blog post: https://circleid.com/posts/20120713_silly_bing
02:59:06 <nicolas17> really don't know what he expected
02:59:50 <fireonlive> don't see anything else about it
03:00:31 <fireonlive> yeah not sure lol
03:09:32 <pabs> arkiver: in urls-sources, I note you disable web.archive.org links for a couple of dead blog aggregators that still have working blogs linked from them. should I just copy the blogs URLs into urls-sources instead?
06:04:35 <pabs> arkiver: btw, found a few more FOSS blog aggregators https://github.com/ArchiveTeam/urls-sources/pull/30
06:09:16 <arkiver> pabs: ah that disabling of web.archive.org happens automatically
06:10:14 <fireonlive> https://github.com/ArchiveTeam/urls-sources/blob/master/3600_web3isgoinggreat_com.txt :D
06:10:26 * pabs lol at the filename
06:10:41 <fireonlive> x3
06:11:18 <fireonlive> its rss feed is also pumped into #web3
07:30:50 <datechnoman> arkiver quick status update. Everything has been running pretty smoothly. Only real thing of note is that we have been pumping through a fair bit of porn the last few hours
07:31:19 <datechnoman> Not too sure if we are wanting that here but worth a mention :)
07:32:26 <datechnoman> Queue has been stable all day
07:54:11 <fireonlive> i have a few of those to add :P
08:32:30 <datechnoman> Haha why am I not surprised you piped up fireonlive :P
08:56:22 <arkiver> datechnoman: i'm going to do a round through a recent CDX, will probably come across the porn and see if it is a problem
08:59:21 <imer> can probably restore the stashed data as well, right? no explosions so far
09:04:11 <arkiver> yep coming up!
09:04:54 <arkiver> so this is only news sites from which we now get outlinks, i plan on soon adding political/government/research sites too
09:06:14 <arkiver> datechnoman: imer: todo:secondary stash is moving back in
09:10:28 <datechnoman> Thanks for that arkiver and I also agree that political/government/research is the next step
09:11:15 <arkiver> yeah :)
09:11:23 <arkiver> i see a loop similar to a previous loop in the logs
09:11:34 <arkiver> it has not escalated yet, but will support for it to be killed
09:12:20 <datechnoman> Sounds like a plan! Weed em out!
09:14:42 <arkiver> :)
09:17:07 <arkiver> moving some of these 'share this web page' links to one-time URLs list, so they don't go into the bloom filter
09:24:09 <datechnoman> That makes sense
09:24:43 <datechnoman> Is it just another bloom filter? Eg; multiple bloom filters that get queried for different things?
09:25:04 <arkiver> no, it does not go through a bloom filter at all
09:25:18 <arkiver> these are usually either those pixels with a one time code in the URL
09:25:41 <arkiver> or the "share to facebook/twitter/etc." links that only exist on the web page that would be shared itself.
09:26:03 <arkiver> oh hah, actually they do go through a bloom filter, but we may remove that filter any time :P
09:27:01 <datechnoman> Haha all good! Also good planning for bloom filter hygiene
09:28:26 <datechnoman> No point wasting bloom filter resources on one time links etc
09:32:29 <arkiver> yep
09:32:54 <arkiver> indeed i see more porn stuff than usual
09:33:25 <katia> 👀
09:36:05 <datechnoman> Ohhh yes. There is quite a lot. Like don't get me wrong, I like porn as much as the next man lol
09:36:27 <datechnoman> But the videos were starting to pile up in the GB's :O
09:37:06 <datechnoman> Which is a lot of HTML we could have instead ;)
09:46:40 <arkiver> updates are in!
09:49:44 <arkiver> and forced now as minimum
09:50:49 <datechnoman> Cheers mate! Love your work as always:D
09:51:02 <datechnoman> Thanks for requesting the stash also :)
09:51:09 <arkiver> thanks datechnoman :)
09:52:10 <arkiver> redo stash is now also being fed back in
09:56:12 <datechnoman> Smick! Then once we clean it all up maybe look at those new outlink sources :D
09:56:48 <datechnoman> Get a nice comprehensive archiving solution together!
10:00:22 <arkiver> yeah!
10:00:28 <arkiver> capturing all the stuff on the internet
10:02:47 <arkiver> pabs: merged
10:02:53 <datechnoman> The more important stuff ****
10:02:54 <datechnoman> haha
10:03:01 <datechnoman> Cant get everything ;)
10:03:07 <datechnoman> nor will we ever
11:23:00 <arkiver> very true!
11:23:09 <arkiver> but we're well underway to get the most interesting bits
11:44:51 <imer> doing a lot of gravatar atm (~50%) cant dig where that might be coming from atm
11:49:57 <datechnoman> yeah we are smashing through them. They will push through quite fast
11:53:30 <datechnoman> Going so fast that the websocket it breaking :P
14:29:04 <arkiver> things are looking good to me
14:29:10 <arkiver> we got through the gravatar stuff fast
15:42:31 <fireonlive> datechnoman: :P
15:44:45 <arkiver> i added the government and political sites as well to have outlinks extracted from
15:50:53 <fireonlive> sweet
15:51:03 <fireonlive> juicy pdf links
15:51:07 <fireonlive> 🔗
15:56:32 <arkiver> the amount of porn in the recent WARCs has been going down
16:03:28 <arkiver> it's all holding up really surprisingly well
16:06:37 * fireonlive 😶
17:05:49 <arkiver> queue going up a bit, largely URLs related to research
17:06:00 <arkiver> i'll leave it going while i'm off for the night
17:20:05 <fireonlive> have fun!
20:52:53 <datechnoman> Looks good mate. Just needs time to chew threw all the new urls
20:58:00 <datechnoman> For some odd reason my containers didn't update when you made the change and I they have been idling out of date :( Just kicked them now
20:58:07 <datechnoman> Should speed things right up!
21:06:08 <datechnoman> Now we are zooming again:D
21:24:55 <imer> oh no target -1s, zooming too fast
21:57:36 <datechnoman> Na was just some mega uploads causing some backlog. All cleared up already :)
22:41:26 <datechnoman> Much data, much wow :D