No user-agent = 403

Posted Feb 15, 2025 3:08 UTC (Sat) by corbet (editor, #1)
In reply to: No user-agent = 403 by DemiMarie
Parent article: Fighting the AI scraperbot scourge

It's not that there is no user agent... These things just pretend to be a desktop browser.

User-Agent strings

Posted Feb 15, 2025 17:58 UTC (Sat) by DemiMarie (subscriber, #164188) [Link] (2 responses)

Modern or outdated one?

User-Agent strings

Posted Feb 15, 2025 18:02 UTC (Sat) by corbet (editor, #1) [Link] (1 responses)

Take your pick - they pick a user agent that is meant to disguise the bot and blend into the rest of the site's traffic. It is a fully attacker-controlled string that really has nothing of value in it.

User-Agent strings

Posted Feb 16, 2025 7:05 UTC (Sun) by wtarreau (subscriber, #51152) [Link]

Usually they just capture a request from their browser during the test and copy-paste all of that into the program. This is also what makes the device detection engines quite popular: if the advertised browser doesn't match the one you detect, then it's highly suspicious, since something was changed in a way that doesn't match what regular browsers do.