UI automation is necessary for accessibility [LWN.net]

UI automation is necessary for accessibility

Posted Jul 14, 2026 13:11 UTC (Tue) by daroc (editor, #160859)
In reply to: UI automation is necessary for accessibility by taladar
Parent article: An update on the scraper situation

Your comment made me curious about how the numbers being thrown around compare to LWN's average, sustainable traffic. So I went and poked at the logs, and found that LWN.net served 1,380,129 page requests yesterday, which was not a particularly heavy day. When there's an active scraper attack that number can be significantly higher.

Even so, only 722,949 of the requests were for actual pages that we have; 657,180 were either 404s or redirections, mostly from HTTP to HTTPS.

to post comments

UI automation is necessary for accessibility

Posted Jul 14, 2026 15:31 UTC (Tue) by quotemstr (subscriber, #45331) [Link] (8 responses)

Why not get lwn on the HSTS preload list? And set up a HTTPS DNS record while you're at it?

UI automation is necessary for accessibility

Posted Jul 14, 2026 16:37 UTC (Tue) by daroc (editor, #160859) [Link] (7 responses)

Most of the traffic still requesting HTTP is from bots or other automated processes that are unlikely to respect the HSTS preload list. Note that we do serve a HSTS header with a duration of one year, so in theory any client that loads a single page from LWN.net should not try HTTP again for at least a year. In practice, we see plenty of clients that do, in fact, ignore the permanent redirects and HSTS header and continue to request the HTTP version of the site first.

As for the HTTPS DNS record — I didn't know that kind of record existed, so I've learned something new today. It probably doesn't make sense to set it up without getting our DNSSEC situation sorted out, but it's a good idea.

UI automation is necessary for accessibility

Posted Jul 14, 2026 16:47 UTC (Tue) by quotemstr (subscriber, #45331) [Link]

FWIW, given how many scrapers are headless Chromium, I'd be a bit more optimistic about the preload list.

UI automation is necessary for accessibility

Posted Jul 14, 2026 20:21 UTC (Tue) by zdzichu (subscriber, #17118) [Link] (4 responses)

Maybe disable plain HTTP? Browsers default to HTTPS connections if I'm not mistaken. Port 80 is legacy.

UI automation is necessary for accessibility

Posted Jul 14, 2026 20:43 UTC (Tue) by daroc (editor, #160859) [Link] (3 responses)

We've considered it. We only started redirecting HTTP to HTTP around ... two years ago, I think? So we wanted to give it time for people to update their bookmarks, feed readers, and so on. It might be worth taking another look at whether it's really worth continuing to listen on port 80, though.

UI automation is necessary for accessibility

Posted Jul 15, 2026 12:49 UTC (Wed) by daroc (editor, #160859) [Link]

Correction: we've redirected HTTP to HTTPS for many years, but changed how that was accomplished in our backend relatively recently, and it is that change that I was remembering.

UI automation is necessary for accessibility

Posted Jul 15, 2026 13:12 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

Or just serve a page that says "waiting 10 secs to redirect to https ..."

That'll slow down any crawler that waits for previous requests to complete, and will make real humans update their links :-) without inconveniencing said humans *that* much.

Cheers,
Wol

UI automation is necessary for accessibility

Posted Jul 15, 2026 13:16 UTC (Wed) by daroc (editor, #160859) [Link]

Well, there are two problems with that:

1. Scrapers often don't complete requests on real time; they receive the redirection, and the redirected URL goes on a queue that gets processed minutes or hours later, often by a different IP address.

2. I don't think there's an elegant way to redirect after a timeout without using JavaScript (which we try to keep to a minimum) or tying up server resources with an open connection.

Honestly, clients hitting port 80 isn't a huge problem. Those redirects are relatively cheap to serve compared to serving a page with dynamic content on it. It's annoying, clutters the logs, and they should really take the hint, but it isn't the end of the world.

UI automation is necessary for accessibility

Posted Jul 15, 2026 5:25 UTC (Wed) by PhilippWendler (subscriber, #126612) [Link]

AFAIU the HTTPS DNS record is useful for telling browser to use HTTPS (and potentially HTTP2), independently of DNSSEC. It is not like TLSA or SSHFP, which contain certificates and need to be authenticated in order to make sense.