A Guide Through The Linux Sound API Jungle

[Posted September 25, 2008 by corbet]

Lennart Poettering has put together a guide to Linux sound APIs, with emphasis on helping developers choose the right one. "At the Audio MC at the Linux Plumbers Conference one thing became very clear: it is very difficult for programmers to figure out which audio API to use for which purpose and which API not to use when doing audio programming on Linux. So here's my try to guide you through this jungle."

to post comments

A Guide Through The Linux Sound API Jungle

Posted Sep 25, 2008 15:16 UTC (Thu) by kev009 (guest, #43906) [Link] (20 responses)

Phonon is really glossed over in the article. If you want real cross platform apps for Windows, Mac, and Unix QT4 should probably be the first place you check.

See: http://aseigo.blogspot.com/2008/09/linux-audio-layers.html

A Guide Through The Linux Sound API Jungle

Posted Sep 25, 2008 15:24 UTC (Thu) by xav (guest, #18536) [Link] (3 responses)

Apparently Phonon isn't yet in a fully working state.

A Guide Through The Linux Sound API Jungle

Posted Sep 25, 2008 19:47 UTC (Thu) by bangert (subscriber, #28342) [Link] (2 responses)

http://en.wikipedia.org/wiki/Phonon_(KDE)
http://en.wikipedia.org/wiki/Qt_(toolkit)

Phonon has been shipped with Qt 4.4 - release date May 6, 2008.

> Apparently Phonon isn't yet in a fully working state.

What makes you think so?

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 7:05 UTC (Fri) by xav (guest, #18536) [Link]

Some comments in the blog article, as well as the Wikipedia page describing Phonon you just linked to: look at the list of features, they all start with "Phonon will ...".

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 10:48 UTC (Fri) by macson_g (guest, #12717) [Link]

Phonon shipped with Qt-4.4 is still in experimental state. This is the first time they included externally developed open source code into they codebase, and apparently they followed the "release early" principle.

From Qt-4.4 documentation:
> Phonon and its Qt backends, though fully functional for multimedia playback, are still under development.

A Guide Through The Linux Sound API Jungle

Posted Sep 25, 2008 16:08 UTC (Thu) by kripkenstein (guest, #43281) [Link] (1 responses)

Well, whether Qt is the best choice is debatable, and it's one of the most common recurring debates we have in these parts. So no need to rehash that stuff.

Now, if we're talking about cross-platform sound, then one could argue that SDL is a good choice. It works on more platforms than basically anything else (there is even an iPhone port, that only awaits Apple's blessing).

PortAudio

Posted Sep 25, 2008 20:34 UTC (Thu) by dmarti (subscriber, #11625) [Link]

Don't forget PortAudio. If you're going to judge a tree by its fruits, Audacity is a working application on the Microsoft, Apple, and Linux platforms (good enough that mic manufacturers can put a copy in a box with a USB mic and call it COMPLETE PODCASTING KIT) and it uses PortAudio.

Looks like this could be a useful project for somebody: PortAudio/PulseAudio Integration

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 2:55 UTC (Fri) by obi (guest, #5784) [Link] (11 responses)

First of all, it's true that if you're developing a QT4 app and you should definitely consider Phonon when evaluating sound APIs. It exists, it's out there, and if you're linking to QT4 already, there's no reason to avoid it.

That being said, I've always had a bit of trouble understanding Phonon's raison d'être. From what I can tell - and feel free to correct me -, Phonon was made to:

1) have a clear and simple API that meshes well with QT,
2) make portability to Windows and MacOS a bit easier,
3) make sure that QT/KDE libs ensure ABI compatibility - i.e. avoid the risk of having to maintain an old branch of an unfamiliar multimedia framework (f.e. gstreamer 0.8) on their own for the duration of the QT/KDE 4.xx cycle,
4) avoid the politics of having to pick and bless a multimedia framework for use with QT or KDE.

However, I wonder whether in doing so Phonon users lost more than they gained. Not picking an existing framework and creating a abstraction layer means that:

1) Phonon apps can only use the "simplified" API: they can't f.e. manipulate the gstreamer pipeline the Phonon gst backend creates for them
1b) ... which means apps will be tempted to stick to the functionality Phonon provides; brave koolaid-drinking app developers might try to extend the Phonon API (and the backends), others will just pick a multimedia framework that allows them to express what they need for their slightly more advanced app. But of course, they will all choose a different framework ;-)
2) resorting to an abstraction for portability was unnecessary. IIRC, gstreamer can bridge to and reuse Directshow components on Windows, and Quicktime components on MacOSX.
3) they had to write decent Phonon backends which probably is quite a bit of work and deals with the framework they're not familiar with. To avoid _the possibility_ that the framework might move on and stop maintaining the branch QT or KDE committed to for the duration of the ABI cycle. Um...?
4) ... and as mentioned before, while they avoid the politics of having to bless one multimedia system, the discussion won't really go away. The developers of non-trivial apps will still have to make a choice.

End result: if you have some simple and some advanced multimedia apps running, you'll end up with Phonon + backend + one or more multimedia frameworks in memory. I believe that just picking one would've saved a lot of work and memory.

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 11:49 UTC (Fri) by quintesse (guest, #14569) [Link] (10 responses)

"I believe that just picking one would've saved a lot of work and memory"

They did once and it backfired completely saddling them up with an unmaintainable situation in what was most of the KDE3.x lifetime.

And they specifically say that Phonon is for those applications that have only modest audio/video needs. So it will do relatively simple playback and capture which will be enough for 99% of the applications (which includes full featured media players). Basically the only applications it will not be enough for are those that do audio/video processing/editing. But in that "application space" GStreamer isn't the only one either because a lot of those apps want to support things like JACK as well.

And let's not forget that Linux is about choice, I think in an an environment like KDE/Qt that needs to be binary compatible for MANY years betting on one single audio layer (that you don't control) is just stupid. GStreamer might never go away, true, but maybe in 2 years something even better comes along and we want to use that for our apps. Why not make that a possibility? (Only half a year ago I would always use the Xine back-end for most of my media apps because it just worked better on my system.)

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 15:26 UTC (Fri) by obi (guest, #5784) [Link] (9 responses)

"They did once and it backfired completely saddling them up with an unmaintainable situation in what was most of the KDE3.x lifetime."

Maybe they just chose the wrong framework? From what I can remember, not a lot of people were actually working on aRts, and it didn't actually already have the features needed for a multimedia framework.

I don't want to hold up Gstreamer as the only shining example of everything that's good and proper, but at least Gstreamer has been around a while and has actually been doing video and complicated pipelines for a while. Experience over a few release cycles make me trust it a lot more. Plus it has a lot of mindshare in general, so it would seriously surprise me that it would all of a sudden die off in the middle of a KDE cycle.

For the same reason, I don't think a new framework will pop up from nowhere that supplants Gstreamer - these things take time to mature, and even if one does appear, I doubt it would be ready before KDE 5.xx.

As the adage goes "make sure the simple things are kept simple, and the hard things possible". It's nice (and important) that Phonon makes the simple things easy for 90% of the applications - however the hard things are simply impossible with Phonon, without dragging in the frameworks themselves anyway.

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 17:01 UTC (Fri) by quintesse (guest, #14569) [Link] (5 responses)

Well with the advantage of hindsight it's obvious they made the wrong choice, but you just can't look into the future so it makes sense to "protect" yourself.

And that making a new audio framework takes time is undoubtedly true but if I see it correctly GStreamer was in development for 3 years before it got included in GNOME (I imagine because it was good-enough by then, but maybe even before that?) while the entire lifetime of KDE3 has been 6 years! And that's up to now, KDE3 isn't dead yet. A lot can happen in 6 years.

But it the end it just comes down to what you yourself are saying: make the simple things easy. Well that's what Phonon does, it makes it very easy for you to add media capabilities to your application and it will even let the user/distro decide which framework to use. That's a pretty big advantage! And it still makes the hard things possible. How? Well you are not forced to use Phonon! Just go ahead and use the any underlying framework that you want. And if in the end it's GStreamer that wins out over all the other frameworks (which at this point in time at least seems likely) you don't have to worry about having several frameworks installed on your system either, both your media studio app and Phonon will be using the same one!

A Guide Through The Linux Sound API Jungle

Posted Sep 27, 2008 23:32 UTC (Sat) by obi (guest, #5784) [Link] (4 responses)

As for hindsight - of course; when KDE adopted aRts the multimedia world was a very different place. I don't think they could have foreseen that. But we've come a long way in the mean time, and I think it should've been possible to have a better idea of where things were going in 2006. Yes, you ran the risk of getting burnt again, but the chances were smaller.

I'd also forgotten KDE 3.xx was out for over 6 years. But still, see my reply to pynm0001 - I think that maintaining gst 0.10 would not have been an impossible task; at least the odds you'd get outside help would be bigger - I imagine there's more people familiar with gstreamer's codebase than Phonon's.

A Guide Through The Linux Sound API Jungle

Posted Sep 28, 2008 13:00 UTC (Sun) by quintesse (guest, #14569) [Link] (3 responses)

I don't know Obi, why on earth do you care so much that KDE has made a small, thin wrapper layer around existing audio frameworks?

Because it's obvious in your reply to pynm0001 where you say "weighing the work required to create Phonon and its backend vs the potential work needed to maintain gst 0.10 for a while on your own" that you seem to think these are somehow related jobs in terms of time, effort and knowledge needed to be actually able to do it. They're not! Phonon is a pretty thin layer, no extensive knowledge of audio frameworks is needed neither to maintain it nor to make the backend "drivers", so the KDE devs are pretty confident that this time around, come hell or high water, they WILL be able to take KDE into the future.

Now if you can somehow come up with a reason why NOT using GStreamer directly is a huge mistake there would be something worth talking about, but right now we're just wasting our time.

A Guide Through The Linux Sound API Jungle

Posted Sep 28, 2008 14:25 UTC (Sun) by nix (subscriber, #2304) [Link] (2 responses)

obi's argument appears to be that maintaining phonon is wasted effort
because you have to adapt to changes in gstreamer's API anyway.

But this is erroneous, because if you have to maintain gstreamer yourself,
even if you don't add any features it's like maintaining a hundred
phonons: all those plugins then need to be maintained, and each of *those*
requires tracking changes in the downstream APIs... much more work.

A Guide Through The Linux Sound API Jungle

Posted Sep 28, 2008 23:50 UTC (Sun) by obi (guest, #5784) [Link] (1 responses)

Actually my argument was more along the line of: "I'd rather have a lot of people help maintain one framework (albeit an old branch), than have several libraries maintained by just a a few people each"; However if Phonon really is as thin as you guys say, and maintaining a stable gst 0.10 really would be like a hundred phonons - then I guess I'm just wrong. The point is moot especially considering Phonon is out there; sorry for wasting your time.

A Guide Through The Linux Sound API Jungle

Posted Sep 29, 2008 0:02 UTC (Mon) by nix (subscriber, #2304) [Link]

gstreamer basically consists of two pieces: the core, which provides
plumbing to connect together plugins, and the plugins. The majority of
those plugins are interfaces to other audio/video libraries (everything
from monsters like ffmpeg to little things like libogg or libmad). As
such, gstreamer has to track changes to the APIs of the libraries used by
all those plugins. (Thankfully, most are quite stable, but not all:
pulseaudio has bumped API in the last couple of years, and ffmpeg hardly
ever stops changing...)

A Guide Through The Linux Sound API Jungle

Posted Sep 26, 2008 21:33 UTC (Fri) by pynm0001 (guest, #18379) [Link] (2 responses)

"For the same reason, I don't think a new framework will pop up from
nowhere that supplants Gstreamer - these things take time to mature, and
even if one does appear, I doubt it would be ready before KDE 5.xx."

A new framework isn't the issue. All that is required is to break API
compatibility in gstreamer itself.

This happened no less than two times in my time with JuK in KDE 3. We had
C++ bindings to gst-0.6 which we used in JuK. These bindings were broken
by gst-0.8 and we eventually simply coded gst-0.8 support directly into
JuK. Then gst-0.10 came out and we had to re-do the support again.
gst-0.10 has been out for awhile but do you want to be the one to guarantee
that gst-0.12 won't be out while KDE 4 is still supported?

I've gone over all of these arguments more than 2 years ago:
http://lwn.net/Articles/183462/

A Guide Through The Linux Sound API Jungle

Posted Sep 27, 2008 23:14 UTC (Sat) by obi (guest, #5784) [Link] (1 responses)

I see, what you say makes sense - it's not unlikely there will be a gst 0.12 within the lifetime of KDE 4.x.

However - weighing the work required to create Phonon and its backends vs the potential work needed to maintain gst 0.10 for a while on your own (there will likely be other organisations interested in a stable 0.10 branch too however) if a gst were to happen during 4.xx, I'd probably avoid building something completely new.

However, this is a judgment call, and I trust you have a lot more experience with it than me. You address the problems with supporting 0.10 yourselves in your mail from 2 years ago:

"So if we just rely on gstreamer 0.10, now we're stuck with an abandoned code base, which KDE developers are unfamiliar with. Does this sound familiar to anyone? ;)"

I think this is the crux of the issue.

You've avoided this by creating Phonon. However, I believe 0.10 has quite a bit of mindshare, and I believe that the KDE people would not have found themselves all alone maintaining the abandoned 0.10 (while OTOH they are pretty much on their own maintaining Phonon). Well, I do see how this might seem very similar to the aRts situation, but aRts wasn't used a lot outside of the KDE world. Maintaining Phonon + backends (which have to interface with the codebases you're not familiar with) must take quite a lot of effort too.

A Guide Through The Linux Sound API Jungle

Posted Oct 3, 2008 19:43 UTC (Fri) by jospoortvliet (guest, #33164) [Link]

There won't be much work maintaining Phonon, as it's incredibly small
compared to gstreamer (or any other multimedia framework). After all, it's
only a wrapper. Besides, Qt Software (Nokia) is maintaining it now ;-)

Don't skip over the advantage of transparent support for Mac & Windows
either - and Phonon also allows users to use another multimedia layer like
Xine or VLC. So more choice for users, easier api for developers and
insurance against api changes. I'd say a bargain ;-)

A Guide Through The Linux Sound API Jungle

Posted Oct 3, 2008 4:56 UTC (Fri) by lysse (guest, #3190) [Link] (1 responses)

Bit of a bugger if you're developing a command-line application, though.

A Guide Through The Linux Sound API Jungle

Posted Oct 3, 2008 7:37 UTC (Fri) by nix (subscriber, #2304) [Link]

You can use Qt in command-line applications.

OpenAL omitted?

Posted Sep 25, 2008 17:52 UTC (Thu) by tetromino (guest, #33846) [Link] (4 responses)

Where is OpenAL in Lennart's picture? It's the standard way of doing positional audio in modern games for Linux (SDL is only useful for basic 1980s-1990s style games), but Lennart does not mention it at all.

OpenAL omitted?

Posted Sep 25, 2008 19:19 UTC (Thu) by drag (guest, #31333) [Link] (3 responses)

> Where is OpenAL in Lennart's picture? It's the standard way of doing positional audio in modern games for Linux (SDL is only useful for basic 1980s-1990s style games), but Lennart does not mention it at all.

I wouldn't look down at SDL audio to much. Quake4 uses it, for example. (I think).

But it's probably worth mentioning.

OpenAL omitted?

Posted Sep 28, 2008 7:53 UTC (Sun) by NinjaSeg (guest, #33460) [Link]

Because John Carmack is more than happy to write his own 3D audio engine from scratch. Because he's hardcore like that...

SDL itself only provides a bare minimum callback API, as it should. Unless you're looking to, or already have, re-invented the wheel, or are doing something special like real-time synthesis, it's not very useful.

The SDL_mixer library provides a more practical API for the needs of a typical game, providing multi-channel mixing, panning, resampling, file loading and decoding and even music streaming/synthesis. It appears to even have some basic positional 3D support.

OpenAL of course provides a full-blown 3D audio solution, and is the only mature open source solution I know of. But it's a bit narrower in scope than something like SDL_mixer, it doesn't really cover stuff like music streaming leaving you to either do some amount of wheel-reinvention, or go outside of OpenAL for that.

If you're not doing positional 3D audio then you'll probably find OpenAL 's API more awkward than necessary. But it comes with recent versions of OSX, and has the backing of Creative Labs on Windows, so it's really the way to go for cross-platform 3D audio. Note that Microsoft killed off hardware Directsound3D in Vista so OpenAL is now the only way to get hardware 3d audio in Vista...

For what it's worth I authored the initial OpenAL patch for the Second Life client, that looks like it might actually get merged soon. It's replacing the closed source "fmod" library it currently uses. SL makes use of the internet music streaming that fmod provides, which is outside the scope of OpenAL, so for that we end up going to gstreamer (on Linux), completely outside of OpenAL. Which means you need to have multiple streams working, any sane distribution should have ALSA+dmix or PulseAudio working out of the box by now...

OpenAL omitted?

Posted Sep 28, 2008 8:01 UTC (Sun) by NinjaSeg (guest, #33460) [Link] (1 responses)

Erk, and actually, Carmack likes reinventing the wheel so much, that none of his game engines have ever used SDL. They use X11 and OSS/ALSA directly. Which off course means they end up getting things like fullscreen wrong. Grrrr... (Synergy fails miserably in Openarena in fullscreen. Also media keys don't work, though I think SDL fails that too last I checked, someday I need to write a patch...)

OpenAL omitted?

Posted Oct 2, 2008 5:37 UTC (Thu) by drag (guest, #31333) [Link]

Ya. I am talking about the very latest stuff. Quake3 and other stuff around that era use this funky mmap thing that grew up out of the Doom DOS era.

A Guide Through The Linux Sound API Jungle

Posted Sep 25, 2008 20:10 UTC (Thu) by wertigon (guest, #42963) [Link] (1 responses)

OSS deprecated? WTF? OSS3, yes, but OSS4 definitely isn't.

The reason? It's spelled *BSD, Solaris and virtually *every other UNIX out there*. ALSA is *incompatible* with those, and trying to kill off OSS the way Linux is doing it will only lead to lots of extra pain for people trying to write things that works on the other UNIXes, sadly.

A Guide Through The Linux Sound API Jungle

Posted Sep 25, 2008 20:27 UTC (Thu) by drag (guest, #31333) [Link]

Ya need to relax.

He said it's obsolete, not depreciated. There is a difference. In his opinion it's a awkward and difficult design and requires every application that supports it to have their own resampling mechanisms, among other things. "Obsolete", in this case, is politically correct codeword for "it sucks so don't use it".

Also he went out of his way to mention (since he mentions it multiple times) the fact that most of the libraries he was talking about have OSS support. Even Alsa libs can run on OSS and are quite possibly portable even though nobody is interested in making them run on anything other then Linux. In most every category mentioned he recommends NOT programming in Alsa except for very simple PCM playback and capture and low-level audio plumbing.

His recommendations:
Jack -- has OSS support
SDL -- has OSS support
libcanberra -- has OSS support
"Safe" ALSA -- has OSS backend
PulseAudio -- has OSS support