Schaller: Launching Pipewire

Posted Sep 21, 2017 20:03 UTC (Thu) by tuna (guest, #44480)
In reply to: Schaller: Launching Pipewire by smurf
Parent article: Schaller: Launching Pipewire

Audio can come in a lot of different formats than "a 48-kHz stream of bytes".

Schaller: Launching Pipewire

Posted Sep 21, 2017 21:47 UTC (Thu) by smurf (subscriber, #17840) [Link] (2 responses)

and video can come in a lot of formats too. So?

My point is that a stream of n>=0 loosely synced(*) video, audio, and subtitle channels is conceptually a very different thing than a single tightly-synced multi-channel audio stream.

(*) audio vs. video frame rates differ by three orders of magnitude. Also, dropping a video frame may not even be noticeable at 50+fps, while dropping 20msec of audio certainly is.

From the blog entry:

>> designing Pipewire to just be able to do video would be a mistake as a major challenge he was familiar with working on GStreamer was how to ensure perfect audio and video syncronisation.

Yeah, but IMHO that's a result of sub-optimal design choices in GStreamer, which make compensating for divergent latency of audio vs. video playback particularly difficult.

I have no problem with Pipewire carrying audio as well as video, after all you can't get reliable sync if you have entirely unrelated subsystems for each. However, the goal should be to hook into the existing pulseaudio or jack infrastructure at the endpoints where audio is collected or played, instead of re-inventing the wheel yet again.

Schaller: Launching Pipewire

Posted Sep 22, 2017 10:36 UTC (Fri) by farnz (subscriber, #17727) [Link]

PCM, DSD, and other uncompressed audio certainly has a much higher frame rate than video; once you get into passing on compressed audio streams (Dolby Digital/AC3, DTS), you can end up with audio frame rates on the order of 10 to 100 frames per second, much the same as video.

The key to perfect synchronization remains the same regardless of how you achieve it - some buffering, a common clock source for video and audio, and care taken to ensure that timestamps are met - whether that's handled by skewing audio clocks, resampling, doing frame rate conversion on video, or otherwise. Given a common clock, you can do this with PulseAudio's timing information and a separate video daemon handling video timing, or you can do it from within one process.

Schaller: Launching Pipewire

Posted Oct 7, 2017 18:46 UTC (Sat) by tpm (subscriber, #56271) [Link]

What design choices are you thinking of that would make that difficult?