PulseAudio is portable also. It runs on Alsa/OSS/Windows/OSX/ blah blah blah. But I think for what your asking you'd have better luck with Gstreamer.
If you target Alsa then you can use the 'safe' subset that is supported by PulseAudio then you might be fine. It's possible to port the 'safe' parts of the libasound to other platforms, but what a pain.
You could target SDL and get cross platform compatibility, but that is mostly for game makers.
If you target full Alsa then that is Linux-only. If you OSS then that means it's only useful on some of the BSDs and possibly Solaris.