What is LWN using ? [LWN.net]

What is LWN using ?

Posted Feb 12, 2026 21:51 UTC (Thu) by daroc (editor, #160859)
In reply to: What is LWN using ? by Poliorcetics
Parent article: Poisoning scraperbots with iocaine

We've discussed it, but haven't decided whether to deploy it or not yet. Since it would have a low impact on human visitors, I think that we probably will if we start seeing sustained attacks.

As mentioned in the article, we've done some work to make the site faster for everyone, and we've also implemented a "load shedding" mode where, when the site is experiencing a high volume of traffic, requests that look like they come from a bot sometimes get an extra interstitial page that they have to click through to actually access the content.

If you haven't seen those pages, it's because logged-in users never see them. Hopefully that suffices for the moment, but if things get bad (and especially if this policy prevents new readers from reaching us), we might have to go a step farther.

to post comments

What is LWN using ?

Posted Feb 12, 2026 23:00 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (5 responses)

I know one of the things Slashdot did a long time ago was to disable comments on old discussions and then archive the whole thing to static HTML. I think they did that mostly to keep their database a reasonable size, but it would also reduce the load if somebody starts trying to download the entire history of the site. You would obviously want to keep comments open for a lot longer than they do, but it seems like a potentially useful approach. It would probably be some extra work, but you might be able to have a dynamic, database dependent mode for ordinary operation and serve only static web pages when you go to "load shedding" mode.

What is LWN using ?

Posted Feb 13, 2026 2:16 UTC (Fri) by KJ7RRV (subscriber, #153595) [Link] (2 responses)

Would it be possible to generate a static archive of all pages, then have the comment links still go to the usual system, then have that update the static archive after posting the comment? That would increase the burden created by the posting of comments, but certainly that occurs far less than regular page loads, and presumably is never done by AI scrapers.

What is LWN using ?

Posted Feb 13, 2026 11:17 UTC (Fri) by leromarinvit (subscriber, #56850) [Link] (1 responses)

I was going to write a comment about how it's probably cheaper to keep querying the comments from a database for recent articles with active discussions, and that there should be a break-even point somewhere. But then I realized that in that mode, *every* page load must do the same work of generating a static page to serve to that single reader. So the only difference would be saving that page to disk, which the database presumably also does already (even if it only writes new comments and therefore less data).

Therefore, the break-even point is reached as soon as saving somewhat larger generated pages for new comments is less expensive than the DB round-trip and dynamic page generation for every request. My totally scientific gut feeling says it would require a ridiculously high comment load to make generating static pages for every new comment *not* cheaper than dynamic rendering for every request.

Of course, this only works if all/most users get served the exact same content. But I think this should be the case here, and in any case, per-user differences can probably be outsourced to a per-user CSS file (which the scrapers probably wouldn't even fetch, since it provides no useful value for LLM training - so this could be dynamic).

What is LWN using ?

Posted Feb 16, 2026 8:52 UTC (Mon) by taladar (subscriber, #68407) [Link]

The main issue with active, relatively new pages would probably be some form of locking on the page re-generation process. But that could be skipped if you are willing to just serve the old page until the generation of the new version is complete.

What is LWN using ?

Posted Feb 13, 2026 10:40 UTC (Fri) by Wol (subscriber, #4433) [Link]

I've said it before, but I do get a bit annoyed by new comments on old articles. You often don't realise the article is out of date until you've been provoked into responding.

Yup - creating a static page AND turning on the moderator flag after about three months imho would be a good idea.

Cheers,
Wol

Static pages

Posted Feb 13, 2026 14:16 UTC (Fri) by corbet (editor, #1) [Link]

The early LWN site ran on some pretty minimal hardware; as a result, it does a lot of aggressive caching of rendered pages. For anonymous readers, it essentially is static pages, comments included. Logged-in users get freshly rendered pages so that their preferences can be honored, but the cost of that is zero relative to the bot load.

What is LWN using ?

Posted Feb 13, 2026 10:22 UTC (Fri) by paulj (subscriber, #341) [Link] (4 responses)

Please don't adopt anubis. You will probably lock me out of LWN.

I can't access lore.kernel.org and other kernel.org stuff cause of Anubis. I have not been able to figure out why (using LibreWolf).

What is LWN using ?

Posted Feb 13, 2026 10:45 UTC (Fri) by paulj (subscriber, #341) [Link] (3 responses)

Could also be the network I'm on. On another network I can access.

What is LWN using ?

Posted Feb 13, 2026 14:37 UTC (Fri) by daroc (editor, #160859) [Link] (2 responses)

If we do adopt Anubis, which I think is unlikely given that we're keeping up so far using other methods, logging in to your LWN account will exempt you from it. We really want to make sure that readers can access the site — that's really what's bothersome about the bots: they were spiking the load and causing timeouts for real humans trying to access the site before we put our countermeasures in place.

What is LWN using ?

Posted Feb 13, 2026 16:57 UTC (Fri) by leromarinvit (subscriber, #56850) [Link]

The effort is much appreciated, and whatever you're doing seems to be working really well (for users at least). For what it's worth, I never noticed any delays or even timeouts, so either it was resolved *very* quickly, or it never affected a large fraction of requests. LWN has consistently remained one of the fastest-loading websites. Thank you for that!

What is LWN using ?

Posted Feb 16, 2026 12:37 UTC (Mon) by Baughn (subscriber, #124425) [Link]

I’m in the same boat. Anubis causes my iPad to overheat and crash, and that’s the device I use for reading things like LWN.

I know it’s old, but it works fine so long as I keep JavaScript disabled.