XDC2012: OpenGL futures [LWN.net]

XDC2012: OpenGL futures

Posted Oct 9, 2012 8:27 UTC (Tue) by renox (guest, #23785)
In reply to: XDC2012: OpenGL futures by anholt
Parent article: XDC2012: OpenGL futures

> It's cheaper to ship pixels over the wire than the commands to render those pixels from other primitives.

Cheaper in what measure? Not in bandwith usage..

to post comments

XDC2012: OpenGL futures

Posted Oct 9, 2012 18:28 UTC (Tue) by dlang (guest, #313) [Link] (22 responses)

If it was cheaper to just shovel pixels around, the GPU graphics cards would never have taken off.

They were built because the internal bus on the motherboard was slow enough that it's faster to send primitives to the processor on the GPU and have it process things than it is to do it all on the CPU.

how can the local bus be too slow, but the network is fast enough?

XDC2012: OpenGL futures

Posted Oct 9, 2012 19:16 UTC (Tue) by rleigh (guest, #14622) [Link]

I would think that the amount of computation and memory bandwidth required to *render* an image into the colour buffer is vastly more than the amount of computation and network bandwidth required to ship the rendered colour buffer over the network. I'm not suggesting that this would necessarily result in better performance than GLX, but I think that two separate issues are being mixed up here.

Regards,
Roger

XDC2012: OpenGL futures

Posted Oct 9, 2012 20:05 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (16 responses)

A modern GPU can render over a billion triangles per second. Do you think it's going to take more bandwidth to transfer a billion triangles per second or 60 frames of HD video?

XDC2012: OpenGL futures

Posted Oct 10, 2012 0:24 UTC (Wed) by dgm (subscriber, #49227) [Link] (3 responses)

Triangles are usually not enough, you want textures too. If you're playing video you are faced with some though choices:
- stream the compressed video (the display has to be able to decode the format).
- transcode the video to a format supported by the display (expensive without dedicated hardware).
- stream uncompressed video (back to 60 frames of HD).
Clearly, the best option is hardware assisted transcoding. But then, why not use that hardware to compress on the fly the HD video, and keep the whole architecture a lot more simple in the first place? Food for thought.

XDC2012: OpenGL futures

Posted Oct 10, 2012 1:43 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (2 responses)

Right, once you're dealing with a significant number of textures then the texture load alone is probably more expensive than just pushing the pixels over the wire. Acceleration primitives make sense when you're doing nothing but drawing rectangles, but once you're beyond that point the reason you're using a GPU is because it's faster at turning your scene into a 2D image than the CPU is, not because you're using less memory bandwidth that way.

XDC2012: OpenGL futures

Posted Oct 11, 2012 10:15 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

Still, doing the rendering at the client side when the client side is networkologically remote means that your X server (which is very likely to have a fast 3D-capable GPU) will see its GPU going unused, while the server (rather less likely to have a GPU) suddenly has to do GPU tasks... probably using llvmpipe. Ah well.

XDC2012: OpenGL futures

Posted Oct 11, 2012 15:17 UTC (Thu) by raven667 (guest, #5198) [Link]

If this is a significant concern, for a large multi-user desktop server, then you can stuff GPUs into it to offload rendering, this is what other VDI solutions support such as VMware.

XDC2012: OpenGL futures

Posted Oct 10, 2012 13:12 UTC (Wed) by renox (guest, #23785) [Link] (11 responses)

If you want to display an office desktop you only need a few of these drawing instructions.
If you want to display a game, that's different, then sending images may reduce the bandwith needed.

XDC2012: OpenGL futures

Posted Oct 10, 2012 14:49 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (10 responses)

Even an office desktop is a 3D composited environment now, with potentially arbitrary levels of complexity depending on application behaviour. Focusing on the simple case just means that you end up limiting UI design for no good reason.

XDC2012: OpenGL futures

Posted Oct 11, 2012 12:09 UTC (Thu) by renox (guest, #23785) [Link] (9 responses)

No good reason? Reducing the bandwith used in many common situations (when you don't run a game or a video) is a *very good* reason.
That said, you do have a point that at some point the bandwith used by the rendering commands(and textures) could be higher than sending a bitmap.
So the "ideal" (but complicated solution) would be to send the rendering commands&texture when their size is smaller that the size of a screen bitmap otherwise send the bitmap.
Of course, this is made more complicated that you can send the diff between two bitmaps or/and compress the bitmap, but this has a latency cost..

XDC2012: OpenGL futures

Posted Oct 11, 2012 15:24 UTC (Thu) by raven667 (guest, #5198) [Link] (8 responses)

My understanding is that sending drawing commands has a major latency cost. The way that was fixed in the X protocol was to run a proxy or local renderer where the client app runs and then ship bitmaps, like NX and xpra. You can still do smart things with caches, icons, prerendered font glyphs, offscreen scroll areas, prerendered buttons, etc. should be cachable at the display side.

XDC2012: OpenGL futures

Posted Oct 11, 2012 15:46 UTC (Thu) by renox (guest, #23785) [Link] (7 responses)

> My understanding is that sending drawing commands has a major latency cost

Huh? Only if you want to re-read the output of these commands (a bad thing to do anyway) or if you do 'cmd1; wait for status; cmd2; wait for status' otherwise why should there be a difference?

> The way that was fixed in the X protocol was to run a proxy or local renderer where the client app runs and then ship bitmaps, like NX and xpra.

Are you sure that this is what NX do?

> You can still do smart things with caches, icons, prerendered font glyphs, offscreen scroll areas, prerendered buttons, etc. should be cachable at the display side.

For X, yes, for Wayland I don't think so as it has no server side drawing commands (short of sending buffer without knowing where they will be displayed).

XDC2012: OpenGL futures

Posted Oct 11, 2012 16:16 UTC (Thu) by raven667 (guest, #5198) [Link] (4 responses)

You may think its a "bad thing" for the app to have access to it's rendered image but this is often required on modern apps (not xterm, xclock, xman and xeyes) which can lead to a perverse result where all the drawing commands and bitmaps are sent from the app to the display, rendered, then shipped back to the app. If you want a simple rendering model for remote apps then HTML/CSS/JS seems a better bet.

Actually think about the rendering needs of a modern web browser, what you can do from CSS and WebGL and what that requires of the underlying graphics subsystem.

Wayland is just the protocol to the output buffer on the display, you have free reign on what the remoting protocol looks like. You can make a virtual wayland framebuffer and send that or do something toolkit specific or use X or make some new standard protocol that the toolkits can output to.

XDC2012: OpenGL futures

Posted Oct 11, 2012 19:55 UTC (Thu) by renox (guest, #23785) [Link] (3 responses)

>You may think its a "bad thing" for the app to have access to it's rendered image but this is often required on modern apps

Then they should do their own rendering locally, not require that the display protocol is limited to local image rendering..

>If you want a simple rendering model for remote apps then HTML/CSS/JS seems a better bet.

That's a totally different use case: the resources used are not comparable..

>Actually think about the rendering needs of a modern web browser, what you can do from CSS and WebGL and what that requires of the underlying graphics subsystem.

WebGL is the same use case as games..

XDC2012: OpenGL futures

Posted Oct 11, 2012 20:20 UTC (Thu) by raven667 (guest, #5198) [Link] (2 responses)

>>You may think its a "bad thing" for the app to have access to it's rendered image but this is often required on modern apps
>Then they should do their own rendering locally, not require that the display protocol is limited to local image rendering..

They can be two different protocols, the protocol for talking to a local display doesn't have to be the same protocol for remoting they are different use cases with different requirements.

>>If you want a simple rendering model for remote apps then HTML/CSS/JS seems a better bet.
>That's a totally different use case: the resources used are not comparable..

Those cases are blending together more and more. There are three ways to deploy an app, native local, native using a remote display protocol or HTML/CSS/JS and the HTML app deployment method is becoming the norm, local apps are becoming niche and remote display apps are a tiny case compared to the other two.

>>Actually think about the rendering needs of a modern web browser, what you can do from CSS and WebGL and what that requires of the underlying graphics subsystem.
>WebGL is the same use case as games..

Not exactly it's the underlying accelerated GUI toolkit functions needed by your standard HTML5 Canvas or WebGL app. You have text and images and 3d objects all composited and styled together, not just an empty buffer that the application manages like a game. Unless your point is that a web browser should just be given a GLES framebuffer and implement it's own GUI toolkit internally to support modern features.

XDC2012: OpenGL futures

Posted Oct 12, 2012 7:28 UTC (Fri) by renox (guest, #23785) [Link] (1 responses)

> They can be two different protocols, the protocol for talking to a local display doesn't have to be the same protocol for remoting they are different use cases with different requirements.

This would be a regression compared to X.
If an application needs to do local rendering then it can renders its images however it likes: I don't see the need to blend this requirement into the display protocol.

As for WebGL, it is a toy until it can be made reasonably secure.

XDC2012: OpenGL futures

Posted Oct 12, 2012 15:44 UTC (Fri) by khim (subscriber, #9252) [Link]

As for WebGL, it is a toy until it can be made reasonably secure.

It's the other way around: nobody will bother to make it reasonably secure till it's more than a toy. You may claim that it's totally wrong way to build "anything secure", but this is how things are done in real world.

XDC2012: OpenGL futures

Posted Oct 11, 2012 19:28 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

my understanding of NX isn't that it does any local rendering, but that it instead shortcuts the roundtrips to the server by intercepting calls and returning the values directly.

Due to it's history, X can't assume much about what's on the other end, however in modern implementations there really is a lot that is always going to be available. With pure X there is a lot of chatter of the type 'do you support feature Y' that can just be short circuited.

In addition, there were drawing primitives that were synchronous (cmd1l result cmd2 result...) because systems were so short of resources that any of the commands could fail due to running out of them. Nowadays the resources are so plentiful that this is almost always a waste of time, so something like NX can return a result immediately (unless the system on the other side has said that it's getting low on resources)

you see the same type of thing in WAN accelerators for windows networking protocols. In both cases the real limit isn't bandwith, it's a chatty protocol that works well on low-latency systems, but can suffer badly when on a high latency system

XDC2012: OpenGL futures

Posted Oct 11, 2012 20:03 UTC (Thu) by raven667 (guest, #5198) [Link]

> my understanding of NX isn't that it does any local rendering, but that it instead shortcuts the roundtrips to the server by intercepting calls and returning the values directly.

I think that's true for older apps that use the core X protocol, xman, xterm, xeyes, xclock are accelerated this way. For newer apps using newer toolkits like Gtk2/3, Qt, etc. I think there is a lot more bitmap shuffling because the client app toolkit depends on being able to do compositing, blending, antialiasing, etc. and shipping the result to the display.

XDC2012: OpenGL futures

Posted Oct 10, 2012 5:06 UTC (Wed) by khim (subscriber, #9252) [Link] (3 responses)

If it was cheaper to just shovel pixels around, the GPU graphics cards would never have taken off.

Sorry, but no. GPU graphics cards have "taken off" as separate, dedicated hardware for 3D. In in about the same era first graphics chipsets (which used the fact that it's cheap enough to just shovel [fully rendered] pixels around) arrived. They went out of favor for a long time because it's hard to build realistic 3D render but it's very easy to attach RAMDAC to anything.

Later it was found that GPUs need fast low-latency pipes and they got them and today they have dedicated connection to CPU or are even totally merged with CPU.

No, if anything GPUs evolution show that you need fast, wide pipe between GPU and CPU and then you can have much slower and narrower pipe which delivers fully-rendered picture.

IOW: X architecture (where 3D was in X Server) is long obsolete, it's time for something like Wayland (where CPU and GPU used are local and only fully processed picture is sent over the wire).

Of course fully-rendered picture without any compression is still too large, so we need to compress it somehow - but this is separate issue.

XDC2012: OpenGL futures

Posted Oct 11, 2012 10:18 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

Of course, you don't need Wayland for that -- as Eric pointed out, you can implement the send-images-over-the-wire thing without changing anything about the current user-visible API/ABI. It's not like the clients care where the rendering is being performed: they just care that it *is* performed.

The answer to everything is not always "Wayland".

XDC2012: OpenGL futures

Posted Oct 12, 2012 15:40 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

That's why I've said something like Wayland. I'm pretty sure it's possible to build system which does rendering locally and is backward-compatible, too. Do we want that? That's different question.

XDC2012: OpenGL futures

Posted Oct 12, 2012 15:59 UTC (Fri) by raven667 (guest, #5198) [Link]

Backward compatible with X? The X server can output to Wayland just fine AFAIK so there should be no trouble continuing to use the X protocol for as long as you wish. It's backwards compatible in much the same way XQuartz provides compatibility.