UI automation is necessary for accessibility [LWN.net]

UI automation is necessary for accessibility

Posted Jul 13, 2026 22:00 UTC (Mon) by Cyberax (✭ supporter ✭, #52523)
In reply to: UI automation is necessary for accessibility by farnz
Parent article: An update on the scraper situation

There probably needs to be a daily limit. E.g. a million pages, that's more than a human can reasonably do, yet not really an issue for any real CDN/server.

I'd love for Scroll to be resurrected. There are also cryptocrap versions, but they're DoA as all cryptocrap is.

to post comments

UI automation is necessary for accessibility

Posted Jul 13, 2026 22:04 UTC (Mon) by quotemstr (subscriber, #45331) [Link] (1 responses)

> There probably needs to be a daily limit.

How do you determine the principal? A daily limit *for who*?

> E.g. a million pages

A million pager per, what, person? How is individual personhood established? What prevents a scraping outfit identifying as a small town's worth of people? The whole concept of rate-limiting per-person per-day sounds easy at first but is fiendishly hard if you think about it and care about privacy.

UI automation is necessary for accessibility

Posted Jul 14, 2026 15:35 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> How do you determine the principal? A daily limit *for who*?

A subscriber. Feel free to create multiple ones, you just have to pay (for example) $10 a month for each one.

> What prevents a scraping outfit identifying as a small town's worth of people?

Nothing. And that's the goal. No Kafkaesque attestations, no complicated surveillance state. Each time you request a page, the page's operator gets credited (with obvious sanity checks).

Want to scrape my Git by following all the links recursively? Be my guest. You'll just have to pay for it.

UI automation is necessary for accessibility

Posted Jul 14, 2026 8:06 UTC (Tue) by taladar (subscriber, #68407) [Link] (15 responses)

A million requests is a lot for most servers. Certainly more than even most unreasonable individual scrapers do on most pages in a day and, depending on which rarely called (and thus uncached) pages they dig up to request, possibly more than a server can handle in total in a day.

UI automation is necessary for accessibility

Posted Jul 14, 2026 13:11 UTC (Tue) by daroc (editor, #160859) [Link] (9 responses)

Your comment made me curious about how the numbers being thrown around compare to LWN's average, sustainable traffic. So I went and poked at the logs, and found that LWN.net served 1,380,129 page requests yesterday, which was not a particularly heavy day. When there's an active scraper attack that number can be significantly higher.

Even so, only 722,949 of the requests were for actual pages that we have; 657,180 were either 404s or redirections, mostly from HTTP to HTTPS.

UI automation is necessary for accessibility

Posted Jul 14, 2026 15:31 UTC (Tue) by quotemstr (subscriber, #45331) [Link] (8 responses)

Why not get lwn on the HSTS preload list? And set up a HTTPS DNS record while you're at it?

UI automation is necessary for accessibility

Posted Jul 14, 2026 16:37 UTC (Tue) by daroc (editor, #160859) [Link] (7 responses)

Most of the traffic still requesting HTTP is from bots or other automated processes that are unlikely to respect the HSTS preload list. Note that we do serve a HSTS header with a duration of one year, so in theory any client that loads a single page from LWN.net should not try HTTP again for at least a year. In practice, we see plenty of clients that do, in fact, ignore the permanent redirects and HSTS header and continue to request the HTTP version of the site first.

As for the HTTPS DNS record — I didn't know that kind of record existed, so I've learned something new today. It probably doesn't make sense to set it up without getting our DNSSEC situation sorted out, but it's a good idea.

UI automation is necessary for accessibility

Posted Jul 14, 2026 16:47 UTC (Tue) by quotemstr (subscriber, #45331) [Link]

FWIW, given how many scrapers are headless Chromium, I'd be a bit more optimistic about the preload list.

UI automation is necessary for accessibility

Posted Jul 14, 2026 20:21 UTC (Tue) by zdzichu (subscriber, #17118) [Link] (4 responses)

Maybe disable plain HTTP? Browsers default to HTTPS connections if I'm not mistaken. Port 80 is legacy.

UI automation is necessary for accessibility

Posted Jul 14, 2026 20:43 UTC (Tue) by daroc (editor, #160859) [Link] (3 responses)

We've considered it. We only started redirecting HTTP to HTTP around ... two years ago, I think? So we wanted to give it time for people to update their bookmarks, feed readers, and so on. It might be worth taking another look at whether it's really worth continuing to listen on port 80, though.

UI automation is necessary for accessibility

Posted Jul 15, 2026 12:49 UTC (Wed) by daroc (editor, #160859) [Link]

Correction: we've redirected HTTP to HTTPS for many years, but changed how that was accomplished in our backend relatively recently, and it is that change that I was remembering.

UI automation is necessary for accessibility

Posted Jul 15, 2026 13:12 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

Or just serve a page that says "waiting 10 secs to redirect to https ..."

That'll slow down any crawler that waits for previous requests to complete, and will make real humans update their links :-) without inconveniencing said humans *that* much.

Cheers,
Wol

UI automation is necessary for accessibility

Posted Jul 15, 2026 13:16 UTC (Wed) by daroc (editor, #160859) [Link]

Well, there are two problems with that:

1. Scrapers often don't complete requests on real time; they receive the redirection, and the redirected URL goes on a queue that gets processed minutes or hours later, often by a different IP address.

2. I don't think there's an elegant way to redirect after a timeout without using JavaScript (which we try to keep to a minimum) or tying up server resources with an open connection.

Honestly, clients hitting port 80 isn't a huge problem. Those redirects are relatively cheap to serve compared to serving a page with dynamic content on it. It's annoying, clutters the logs, and they should really take the hint, but it isn't the end of the world.

UI automation is necessary for accessibility

Posted Jul 15, 2026 5:25 UTC (Wed) by PhilippWendler (subscriber, #126612) [Link]

AFAIU the HTTPS DNS record is useful for telling browser to use HTTPS (and potentially HTTP2), independently of DNSSEC. It is not like TLSA or SSHFP, which contain certificates and need to be authenticated in order to make sense.

UI automation is necessary for accessibility

Posted Jul 14, 2026 15:42 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

A million requests is just about 12 requests per second over the course of the day. That's just not a lot even for dynamic content. And what I meant is the total _daily_ limit for a user for all the sites.

Suppose that we have a Scroll-like service for $10 a month, so around 30 cents per day spread proportionally over visited sites. My poor Forgejo was able to serve around 250 requests per second so that 1 million requests would have been exhausted in little more than 1 hour, getting me 30 cents. During that time, my server would consume about 0.6kWh of electricity rendering all that data. So I would get a small profit given my low local prices.

Anyway, the numbers can be tweaked in either direction, with different pricing tiers.

UI automation is necessary for accessibility

Posted Jul 15, 2026 12:14 UTC (Wed) by taladar (subscriber, #68407) [Link] (3 responses)

It is extremely rare that you get the luxury of having the requests a day equally distributed across every second of the day.

PHP on a cloud server likely has a single digit or small double digit number of workers and for something not cached it can easily take a second to generate a page.

Sure, if you only count compiled languages your 250 requests/s might be realistic but the reality is that there are plenty of PHP, NodeJs, Python, Ruby or even Perl based websites that will never reach that and aren't deployed on huge servers because their normal traffic is nowhere near that high.

UI automation is necessary for accessibility

Posted Jul 15, 2026 16:51 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (2 responses)

> It is extremely rare that you get the luxury of having the requests a day equally distributed across every second of the day.

Just to clarify for folks who haven't done this before: If you graph queries-per-second against time, for the vast majority of web-related services, you get a sinusoid with a period of 1 day, a fairly significant amplitude (relative to the average), and in most cases, the graph looks completely different on the weekend. You must provision for the daily peak and not the average, or else you will fall over. Load balancing, caching, and other middlebox technologies can make the graph a bit flatter (at the origin server), but it will still have a daily peak and you will still need to deal with it.

This is not only true for the origin server, but also for most if not all of its backends (and their backends recursively). You really only get flat QPS from batch jobs running continuously, and even those are often spiky in practice.

UI automation is necessary for accessibility

Posted Jul 15, 2026 16:57 UTC (Wed) by paulj (subscriber, #341) [Link] (1 responses)

> You must provision for the daily peak and not the average,

Well, for January 3rd (boxing day / St. Stephen's day sales) / FA Cup final / $INSERT_APPROPRIATE_ANNUAL_EVENT_HERE.

UI automation is necessary for accessibility

Posted Jul 16, 2026 8:16 UTC (Thu) by taladar (subscriber, #68407) [Link]

With most sites you can probably get away with some flakyness on the busiest day of the year (unless that is specifically what the site is designed to serve of course if it is an event related website) but you certainly won't get away with it on every daily peak.