RealtimeKit and the audio problem [LWN.net]

RealtimeKit and the audio problem

Posted Jul 1, 2009 22:48 UTC (Wed) by drag (subscriber, #31333) [Link]

Audio latency and realtime performance is critical in any sort of audio producing, recording, remixing, or anything like that.

Saying "I don't know what these people are doing when listenning to music" is just about as a ignorant thing as you can possibly say. It's borderline insulting.

Audio processing is a very intensive realtime thing. Very. It's, in fact, absolutely critical for any sort of digital audio workstation environment. It's as critical as having reliable TCP/IP stack for a web server.

> if you then overload the box, what is the system supposed to stop doing?

Disk I/O for starters. I/O to the video card. I/O to the network. In fact it should absolute halt everything non-critical in a effort to keep the important applications working properly and doing what that particular application requires.

What is critical in a file server is decidely uncritical in audio situation.

HOWEVER...

You have to understand that we are not just talking about 'overload'.

Humans can perceive audio latency of about 100ms or so. It requires that a audio professional be able to control audio syncing issues down to about that much, give or take. You have to match beats, match vocals, sync audio and video, etc etc.

--------------------

Take this for example:

I have a MIDI controller. It's a very simple one. Most people would call it a 'keyboard'. But it doesn't play music notes. What it does is that it's a midi controller shaped like a keyboard. I has the traditional piano keys and a few knobs and sliders. It plugs into my Linux box using a USB adapter. It creates a virtual midi device. When I press a key or turn a knob this sends Midi events to the PC.

Midi events are not musical events. They are mearly some sort of 'X happenned' event. Then it's up to various music programs or other midi hardware to interpret that midi event to mean something.

Typically, in Linux, this means using a software synth. A software synth has a number of pre-recorded 'sounds' that play some sort of noise. So my midi keyboard sends that event to the software synth, that synth in turns plays a PCM audio stream that is that particular sound that is then modulated to some sort of note.

Now that noise can be anything. A drum-like noise, a piano, a cat meow, or whatever.

Then I typically route that PCM audio from the software synth through another program that then performs some sort of audio distortion on it. Makes it sound smoother, less digitalized, maybe add some echo or depth or whatever else that I think sounds cool.

This way I can take a noise produced by the software synth and make it sound like, say, a grand piano in a hall or something like that.

Then that modulated PCM audio is then sent to my audio card that then plays the sound out over the speakers.

So the time from when I press the keyboard to the time when I hear the sound will be the total latency:

So in summary that looks like this:

My finger --> Midi keyboard keypress --> USB-Midi event --> Jackd midi routing --> Software synth --> Jackd PCM audio routing --> realtime audio/distortion program --> Jack PCM audio routing ---> sound card --> stereo receiver --> speaker ---> My Ear.

It's suprisingly complicated isn't it? Even with a somewhat fast computer your looking at continious 60-70% CPU load in many cases just to keep the sound card feed and the audio worker happy. Dual/Quad cores can help a lot, obviously.

And that is a SIMPLE setup. Each step along the way will add it's own small amnount of latency. You want the OS to add as little latency as possible for performing tasks that are not immediately critical for the task at hand. Like it doesn't matter if the disk I/O doesn't happen immediately.. the requirements to keep the sound card buffer feed is a much higher priority in this sort of situation.

So as you can imagine a piano player with a half a second of latency from when they press a key to when they hear a sound is completely unacceptable. It's just shit and will screw them up.

I just play around so I am not critical, but it's certainly unbearable to have a big delay. Like I said that most latency below a 100ms will be generally unpercievable. But a small amount preceived audio latency isn't that bad. After all it takes a while for sound to reach your ears. Sound is kinda slow compared to other things.

------------------------------

Now during the normal operation of Linux there are all sorts of things that introduce latency. The kernel blocks when doing lots of different things and all that happy horseshit.

Having high latency is a good thing, often, in a server. When your dealing with network stuff people expect delays and short pauses when doing stuff so that you can take that into advantage to be able to do your processing tasks in a very fast and efficient manner. Having your system continiously interrupting processes every few milliseconds is going to have a very detrimental effect on overal performance.

If you want to look at it like this:
* High latency behavior == efficient processing == good for servers
* Low latency behavior == desirable behavior == good for digital audio workstations.

On early versions of 2.6 they had system latencies... when the entire system would just pause and be crunching on some task or another for 500ms or so. It'll just blank out, go out ot lunch. Make it impossible for any other application to do any work.

And this isn't even under 'high load' or 'moderate load'. It's just normal operating proceedure. (Obviously huge efforts have been made to improve performance)

When your configuring your audio applications to perform in realtime and you get a lag spike of 250ms or 500ms this does not just cause delay it causes huge problems. It causes audio tracks to fall out of sync. Causes video and audio to go out of sync, which causes performance playback problems and bad video performance as the application struggles to keep things in sync. It causes squaks and pops and other audio artifacts in audio playbacks and audio recordings.

----------------------

This is a serious problem in other OSes too.

In Windows XP, even though it is very popular, the Win32 API and audio stack is so miserable and realtime performance is so poor that third parties had to create their own driver model for XP called ASIO.

Even with ASIO, though, you have to be very careful about doing disk I/O in Windows to avoid audio latency and the resulting undesirable audio distortions and artifacts. Typically you have to disable your virus protection and things like that.

RealtimeKit and the audio problem

Posted Jul 1, 2009 23:07 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]

please note that I dis say that there were cases where the system needed to reduce it's latencies.

but the fundamental problem is that no system is going to know that the priorities that you want for your audio processing are the righ priorities.

you say kill disk I/O and network I/O

what if you are using one of those for your audio? how is the system supposed to know?

it's a very different thing to say that the system should prioritize your audio over everything else than to say that the system shouldn't have 500ms pauses. the latter is something that everyone can agree to, the former is not.

linux has gotten much better since the early 2.6 days, and I don't think that anyone is saying that there isn't room for further improvement (they may not know where the problems are, but I don't see anyone arguing that there are definantly no problems)

but if you want to dedicate your box to audio processing, you should tell the system that you want to do so. it has no way of (accurately) knowing.

I am not saying that realtimekit is the right answer (I have no idea if it is or not, and am also not a fan of the idea that D-bus is being used for everything nowdays) but the idea that any desktop/server system should out-of-the-box be suitable for heavy audio processing is just wrong.

it should be good for _light_ audio processing (up to the squishy point where the load on the box causes enough contention for resources that you start having an unacceptable number of problems)

RealtimeKit and the audio problem

Posted Jul 2, 2009 6:04 UTC (Thu) by drag (subscriber, #31333) [Link]

Look. What I am talking about there is not optional, it's not a "nice" thing, its a _requirement_.

> you say kill disk I/O and network I/O

> what if you are using one of those for your audio? how is the system supposed to know?

Probably by PID number. I was under the impression that the Linux I/O scedualer was capable of prioritizing different things.

The disk access can almost always be delayed because of the nature of cache in ram. A system can't prioritize incomming network packets becuase you need to analyze the packet first to determine weither or not it's priority, but processes can be give priority on network access.

This is all stuff that is already in the kernel.

> it's a very different thing to say that the system should prioritize your audio over everything else than to say that the system shouldn't have 500ms pauses. the latter is something that everyone can agree to, the former is not.

Ya... Its more complicated then you think here, I think.

Very low-latency = low performance. A system that has very good low-latency behavior tends to be inefficient. This means reduced performance for web servers, file servers, database servers, encryption, etc etc.

So it's certainly not something everybody can agree on. So it has to be tunable. Which is the point behind having a configurable system were you can assign "realtime" performance to key processes at the detriment for the rest of the system.

Audio is just one of the many times you need to have good "realtime" performance.

Any time your having a computer interacting with the real world.. You can't pause real-life. You can't put events in a buffer and schedual things so that you can have good performance.. many times the computer needs to react to events outside of it's control in a timely manner.

Robot controls, scientific instrumentation, vehicle navigation systems, vehicle control systems, cell phones, VoIP devices, wallstreet stuff blah blah blah.

It'll be nearly impossible to make Linux ever perform "hard realtime" but certainly things can be improved to the point were the utility of the vanilla Linux kernel can be increased massively.

RealtimeKit and the audio problem

Posted Jul 2, 2009 6:20 UTC (Thu) by elanthis (guest, #6227) [Link]

It's NOT SUPPOSED TO KNOW. That's the whole damn point.

The application is real-time. It is in full control, not the OS. When it knows it needs to fill an audio buffer, it does that. When it knows it needs to write a sample to disk, it does that. When it knows it has nothing better to do with its time for a little while, it lets the kernel do some stuff.

You are being OS centric. Let's be perfectly clear here: the OS is unimportant. It doesn't matter, at all. Linux, Windows, Solaris, OS X, they're all totally and absolutely irrelevant. The fact is, a computer is only useful for running applications, be they your eCommerce server solution, your audio recording apps, or telecommunications/GSM phone app. The only reason we even run had an OS at all is for convenience: it's easier to be able to run a couple apps together rather than having to reboot to change apps, it's cheaper to use a single computer to run your browser and email client at the same time rather than needing two computers, it's cheaper to let six students share a single mainframe rather than each needing their own, etc. We put a lot of effort into making these OSes powerful and pleasant to use, but at the end of the day, the OS is only there to run our applications.

When the application developers have to start bending over backwards to appease the OS instead of having the OS bend over backwards to run the application the way it needs to be run, something is wrong. It's like selling your car for gas money: you give up something you need in exchange for something you then suddenly no longer need. Without the applications, the OS is entirely freaking useless.

Therefor, if Linux is incapable of knowing what the audio applications want and can't automagically optimize appropriately, the solution is very simple: let the applications tell Linux exactly what they need and how they need it. Which is exactly what real-time systems in the kernel (and other related facilities) are meant to do.

If you dislike the security implications of those facilities, DON'T USE THEM. The audio folks don't give two shits about good server security or what not, because they're not running servers or multi-user desktops or even Web-based network-centric applications. They're running a dedicated set of audio processing apps and little else, and anything else they are running can quite safely be shoved into the background and starved of any CPU or I/O time while the audio app is doing its thing.

RealtimeKit and the audio problem

Posted Jul 2, 2009 7:47 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]

and if you really want to dedicate the computer to the one application, and risk a bug in it taking out everything and locking the machine up, you can do so. use the root pivilages that you have to give your application realtime priority, they you can keep control of the CPU as much as you want.

just don't consider the result a general purpose system, it's become a dedicated appliance that could be totally useless for anything else.

in fact, other than possibly the need for drivers, you are better off going with something like DOS for your appliance.

that has a place, but it also will have zero effect on the linux distributions, because they are not making single-purpose appliance builds, they are making an OS that will run many different things, frequently at the same time. and while people don't like their music playback to stutter, they usually aren't willing to sacrafice the lots of performance when playing games, or running autocad, etc just so that the background music never skips

RealtimeKit and the audio problem

Posted Jul 1, 2009 23:47 UTC (Wed) by jwoithe (subscriber, #10521) [Link]

A minor detail: in terms of the human ability to detect latency it is often as low as 10 ms although it does depend a bit on what you're doing. Perhaps for some 100 ms is acceptable but in many situations I can "feel" latencies of the order of 20 ms. I know several professional keyboard players who need to go to 10 ms or below before things "feel" right. Certainly for me a system latency of 100 ms is far too large.

In some cases latency compensation (as implemented in Ardour for example) can help out. In others (such as running incoming audio through realtime effects for live work) there is no getting around the need to lower system latency as much as possible. In this latter case the target latency needs to be somewhat lower than 100 ms - probably closer to 10 ms or there abouts.

For those working with low system latencies (of the order of 10 ms) the chances of audio skips and other related problems are *much* higher than they are for those who are able to work with higher latencies.

RealtimeKit and the audio problem

Posted Jul 2, 2009 23:19 UTC (Thu) by Tuxie (guest, #47191) [Link]

I'm nowhere a professional musician, but I have found latencies above 30 ms to be extremely annoying when playing the keyboard, especially when I use electronic zero-attack sounds rather than more natural sounds. When playing the guitar however, anything above 15 ms is almost unusable.