X.Org security advisory: root hole via rogue hostname

It's worth nothing that OpenBSD uses BSDAuth, not PAM. Unlike PAM, a program doesn't need root permissions to authenticate against the local passwords. PAM uses shared libraries and thus operates with the credentials of the calling process, whereas BSDAuth uses tiny executable commands in /usr/libexec/auth (with set-user-id root, but requiring auth group to execute). OpenBSD is really rigorous about creating a unique user and group for every little service and role. This helps to mitigate privilege escalation attacks.

But that only solves one of the two issues. You still need root to setuid(). But OpenSSH, for example, uses a multiple process model to isolate the privileged process; and it's the unprivileged process that does network I/O. Theoretically XDM could be setup similarly, so that only an unprivileged fork runs all the startup rc code and interacts with the environment, asking the privileged process to create a new user session when necessary. AFAIK the OpenBSD folks haven't hacked XDM to do this, but they use this multi-process model for all of their new services that require preservation of root credentials (excepting simple authentication, which is solved by BSDAuth).