Hutterer: X servers no longer allow byte-swapped clients

Hutterer: X servers no longer allow byte-swapped clients

In this case, it's worth doing because Xorg runs extra code when dealing with an other-endianness client, and that code has been a great source of security bugs over the years. By disabling other-endianness clients, you get to cut out a whole pile of unmaintained code that's also rarely used.

The cases that are well-tested, and maintained, are big endian client connecting to big endian server, and little endian client connecting to little endian server. The cases that are problematic, and will now be disabled by default, are little endian client connecting to big endian server, and big endian client connecting to little endian server; these run "SProcs" - code which swaps endianness inside structures - which are not run at all if you have the same endianness on both sides of the link.

The two fundamental causes of trouble here are:

1. X11 was designed when a malicious client was simply not a thing, so SProcs have to be written for each and every message, and have to understand the semantic meaning of each message so that they can correctly only the fields that need swapping (e.g. a message that's 2 bytes, followed by a uint16_t, followed by a uint64_t needs no byte swapping on the first two bytes, then a 16 bit byte swap, then an 8 byte reversal). This is manual effort that not everyone working on the X server will remember - and if you forget, it won't normally break your test setup (unless you deliberately test cross-endian, which involves remembering that the other endianness to your normal test system is a Thing that you need to care about, and either having more hardware, or setting up QEMU emulation of a wrong endian system).
2. The X11 endianness "negotiation" is too simple: the client tells the server "I am this endianness" as the very first thing it does, and simply assumes that the server will handle endianness swapping. Thus, there's no way to move the pain to the client easily - you want native endianness when going to a server on the same host (for performance), but different endianness over the network.

If X11 had been designed with all the knowledge we acquired in the first 15 years since its release in 1987, it'd have had a true negotiation, and the server could tell the client "I only handle little endian" or "I only handle big endian" (as well as "I handle both endian choices"), making this something we could push to the client over time. But that'd involve a time machine.

Similarly, if someone cared enough to do the work of changing all of the X11 protocol handling from hand-crafted C to machine-generated code that always did the right thing, it'd be solved. But no-one's stepped up to do that since the people working on XCB moved onto working on Wayland instead.