What is LWN using ? [LWN.net]

What is LWN using ?

Posted Feb 12, 2026 23:00 UTC (Thu) by rgmoore (✭ supporter ✭, #75)
In reply to: What is LWN using ? by daroc
Parent article: Poisoning scraperbots with iocaine

I know one of the things Slashdot did a long time ago was to disable comments on old discussions and then archive the whole thing to static HTML. I think they did that mostly to keep their database a reasonable size, but it would also reduce the load if somebody starts trying to download the entire history of the site. You would obviously want to keep comments open for a lot longer than they do, but it seems like a potentially useful approach. It would probably be some extra work, but you might be able to have a dynamic, database dependent mode for ordinary operation and serve only static web pages when you go to "load shedding" mode.

to post comments

What is LWN using ?

Posted Feb 13, 2026 2:16 UTC (Fri) by KJ7RRV (subscriber, #153595) [Link] (2 responses)

Would it be possible to generate a static archive of all pages, then have the comment links still go to the usual system, then have that update the static archive after posting the comment? That would increase the burden created by the posting of comments, but certainly that occurs far less than regular page loads, and presumably is never done by AI scrapers.

What is LWN using ?

Posted Feb 13, 2026 11:17 UTC (Fri) by leromarinvit (subscriber, #56850) [Link] (1 responses)

I was going to write a comment about how it's probably cheaper to keep querying the comments from a database for recent articles with active discussions, and that there should be a break-even point somewhere. But then I realized that in that mode, *every* page load must do the same work of generating a static page to serve to that single reader. So the only difference would be saving that page to disk, which the database presumably also does already (even if it only writes new comments and therefore less data).

Therefore, the break-even point is reached as soon as saving somewhat larger generated pages for new comments is less expensive than the DB round-trip and dynamic page generation for every request. My totally scientific gut feeling says it would require a ridiculously high comment load to make generating static pages for every new comment *not* cheaper than dynamic rendering for every request.

Of course, this only works if all/most users get served the exact same content. But I think this should be the case here, and in any case, per-user differences can probably be outsourced to a per-user CSS file (which the scrapers probably wouldn't even fetch, since it provides no useful value for LLM training - so this could be dynamic).

What is LWN using ?

Posted Feb 16, 2026 8:52 UTC (Mon) by taladar (subscriber, #68407) [Link]

The main issue with active, relatively new pages would probably be some form of locking on the page re-generation process. But that could be skipped if you are willing to just serve the old page until the generation of the new version is complete.

What is LWN using ?

Posted Feb 13, 2026 10:40 UTC (Fri) by Wol (subscriber, #4433) [Link]

I've said it before, but I do get a bit annoyed by new comments on old articles. You often don't realise the article is out of date until you've been provoked into responding.

Yup - creating a static page AND turning on the moderator flag after about three months imho would be a good idea.

Cheers,
Wol

Static pages

Posted Feb 13, 2026 14:16 UTC (Fri) by corbet (editor, #1) [Link]

The early LWN site ran on some pretty minimal hardware; as a result, it does a lot of aggressive caching of rendered pages. For anonymous readers, it essentially is static pages, comments included. Logged-in users get freshly rendered pages so that their preferences can be honored, but the cost of that is zero relative to the bot load.