Xorg flaw

I think that there are actually three things the client has to do: get the server to allocate enough server-side, non-shared resources to use up most of the address space and force the remainder somewhere useful; get a shared memory segment so that the client will be able to change an area of the server's address space; and get the server to overflow the stack into the shared memory segment.

The shared memory aspect is not really a flaw to avoid; the flaws to be fixed on the userspace side are really that the server will go overboard allocating resources for clients, rather than applying some limits to protect itself, and that the server's stack can grow into the heap. At some point, the server should refuse to do what the clients are asking in order to protect itself from overloading (which is hard); the kernel should do better at preventing overloading from leading to unexpected aliasing (which they did). The MIT-SHM aspect just makes the exploit comprehensible.

I don't doubt that a sufficiently clever request could get the server to overflow the stack into the area where the response to the request will be written and write a chosen response into a spot that aliases a return address on the stack, causing the server to return to effectively calling system() on a chunk of an image provided by the client.