Why bother?

"Audio over network" works well with for example PulseAudio, but what
doesn't work is "audio+video+synchronization over network". While you
could tie X+PulseAudio server close enough together to make this feasible,
I suspect the result is going to be quite messy.

As for MPEG-4... the idea would of course be to provide an AVC
decompressor accessible through the X protocol as well -- reencoding at
the client is quite obviously highly undesirable (maybe useful as an
emergency fallback, but that's it).

And last time I checked, there was no media player that could play
through "ssh -X" :)