There are good reasons why PA runs as you and not as a systemwide user (though it can). PA can be asked to load modules providing new features at runtime: this is obviously forbidden for systemwide daemons. PA can operate in a zero-copy mode, transferring audio data directly over shared memory: for obvious security reasons this must be avoided if the daemon may serve more than one user, forcing everything to be serialized and deserialized again.
PA itself is plainly necessary: most modern sound hardware can't mix, so the first open()er blocks all the rest. This is more than slightly aggravating to users (the difference between a very long block and a crash is not very large from a user perspective).
PA is really not that bad. Yes, it was buggy once upon a time (mostly because it used features that had never been used by anyone else), but these days it largely Just Works.