Taylor: Avoiding Jitter in Composited Frame Display [LWN.net]

Taylor: Avoiding Jitter in Composited Frame Display

Posted Dec 1, 2012 19:44 UTC (Sat) by cyanit (guest, #86671)
Parent article: Taylor: Avoiding Jitter in Composited Frame Display

Maybe I'm stupid, but I don't understand the article at all.

In his "jitter" example, the media frame is delayed simply because it arrived too late, and I fail to see why an extra application would have any effect.

Also, I cannot understand why on Earth you would want to do anything "2ms after vblank".

Ignoring the bizarre article, the correct way to do it while minimizing GPU usage it is to estimate an upper bound T for the time needed to draw all windows and flip buffers (in the worst case, assuming all clients send an updated frame), and then on time T before the time the new buffer starts being scanned out, elevate to SCHED_RT (or use an RT deadline scheduler set to that moment if available), grab exclusive access to the GPU, and do the blits and flips.

While the blits are performed, updates to windows not blitted yet should still be accepted, and thus it's a good idea to start blitting the windows that are the least likely to send a further update in the current frame, which probably means starting from all windows that are currently waiting for user input, and then the most recently updated to least recently updated window among the windows that are "being actively animated".

If a client is not capable of informing the compositor whether a window is "being actively animated", the compositor can simply use the heuristic that a window is such if the last update is more recent than 100ms ago.

If the GPU cannot be acquired exclusively, or GPU render time cannot otherwise be reliably bounded, then another option, which is simpler but more wasteful, is to simply use two backbuffers, where one is always displayable, and process each update by simply copying the good backbuffer, plus the update to the other one; a separate thread will then swap the good backbuffer at the swap time.

Also, in general, the display refresh rate should be set to the highest multiple of the least common multiple of the requested rates of each application if possible (or the exact lcm if all windows request an exact frame rate and the mouse cursor is not visible), or to the highest supported rate otherwise.

(Log in to post comments)

Taylor: Avoiding Jitter in Composited Frame Display

Posted Dec 1, 2012 20:45 UTC (Sat) by iabervon (subscriber, #722) [Link]

It's hard to see in the diagram, but the difference between the 2nd and 3rd diagrams is that the compositor doesn't draw in the space between the 2nd brown box and the yellow box. There's plenty of time for it to get a frame drawn which would have the content from the brown box, but it just doesn't bother. The brown application would have been in time, except that the yellow application was early, and the compositor drew the frame early. The diagram is confusing because the bright color makes it look like the complaint is about the yellow application's data taking a long time to appear, but it's actually that one video frame from the brown application is visible for 4 cycles and the next is visible for 1 cycle, instead of 3 and 2, despite the fact that the brown application behaves exactly the same and the compositor isn't overloaded.

Taylor: Avoiding Jitter in Composited Frame Display

Posted Dec 4, 2012 9:29 UTC (Tue) by iive (guest, #59638) [Link]

I'll try to explain the issue.

The desktop have 2 buffers, one is visible while the other is been draw.

Imagine the compositor as drawing a 3D scene where there is single rectangle for each window and its content is rendered from texture. The composer (at the moment) seems to be executed right after buffers are swapped.

The problem seems to arise when the program is sending commands to draw into the texture. Either the texture would be in incomplete state when it is time to compose the desktop (e.g. async drawing) OR the server should wait for the texture update to finish before calling compose and showing the desktop.

There are 4 ways to solve this:
1. Use 2 textures. Draw into new texture. The composer would use the old one until the new one is complete.
Problems: Doubles memory usage. The server doesn't know if it should copy the content of the old texture, so the app could only modify it. As backing store have been removed, I'd say it expects the program to draw the window(texture) in full (but that's slow...). With current composer it also adds latency to output(draw,texture,compose,swap).

2. Use 2 textures, one scratch pad and one for composing. When drawing is done, XSync and XFlush commands would copy the scratchpad into the composing texture. This is indeed variation of #1. The difference is that you have a backing store (preserving the content of previous drawing) for free. As modern toolkits draw everything in ram and just blit it, it makes sense for scratch pad to be in RAM. Of course blit takes time so you may need #4 to control it. Why use this method then? The application may issue a long and complicated drawing instructions. If we have to care only for a single blit, we will know how much time it would take.
Problems: Additional copy makes things slower.

(tylors' methods)
3. Hold the redraw/damage events until after desktop framebuffer is swapped (and new image is composed). When received, these events cause the application to issue new drawing commands. The hold would ensure that redrawing starts when there is still a lot of time. This technique is actually quite useful for non-composite case, where you can have damage from moving one window over the other (overlapping). There are no per window textures only desktop frame buffer(s).

4. Withhold the drawing commands if the time for swap is near. Since X is client-server protocol, the program doesn't have full control when the commands are executed.

There are other tricks that could be used, but they require synchronizing with the video card. Stuff like knowing what is the current line that is been output, when the vblank begins, when it ends. Things that were possible with CGA,EGA and VGA.

And don't get me started on video playback issues...