Mozilla on the coming version-100 apocalypse

Posted Feb 17, 2022 7:52 UTC (Thu) by taladar (subscriber, #68407)
In reply to: Mozilla on the coming version-100 apocalypse by nybble41
Parent article: Mozilla on the coming version-100 apocalypse

As a sysadmin I strongly disagree. The UA of bots is very rarely that of an actual, recent desktop browser. Yes, the bad bots do often copy it but they then never update it again so usually they look like severely outdated versions of the browser whose UA-String they copied. Freezing it would just serve to create that situation you think is already there.

Mozilla on the coming version-100 apocalypse

Posted Feb 17, 2022 15:31 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

> The UA of bots is very rarely that of an actual, recent desktop browser. … Freezing it would just serve to create that situation you think is already there.

That isn't actually a disagreement—you're just not seeing a lot of "*really* badly coded web scrapers". I never said that *most* bots did this today. The point was just that you can't rely on a client-selected User-Agent string to filter out bots reliably. It's an easy thing to implement so long as it's not over-used, so scraper authors don't have any reason to work around it, but if identifying as a bot (or an old browser) will get a scraper blocked or throttled then correcting the problem will take a few minutes of the scraper developer's time at best. And in the meantime, for non-scrapers, we ought to be targeting web standards and not implementing workarounds for specific browsers. *That* is the point of freezing the User-Agent string: force sites to serve the same versions of their resources to everyone so that they don't break or degrade when someone comes along with a standards-compliant user agent the site simply can't identify.