Weekly Edition
Return to the Development page
| |||||||||||||
XDC2012: OpenGL futuresIan Romanick works on Mesa, an open-source implementation of the OpenGL specification. His presentation on the final day of the 2012 X.Org Developers' Conference looked at what he hoped would be the future of the OpenGL interfaces on Linux. His talk was broken into three broad areas: the current status of the OpenGL interfaces, where they should go in the future, and how to get to that future. The current OpenGL ABI was defined in 2000. It consists of a number of pieces. The first of these is libGL, which implements three components: OpenGL 1.2, GLX up to version 1.2 (the current version of GLX is 1.4), and the ARB multi-texture extensions. Ian highlighted that libGL on its own is not sufficient for any useful applications these days. The remaining pieces of the OpenGL ABI are libEGL, and two separate libraries, libGLES_CM and libGLESv2, for versions 1.2 and 2.0 of OpenGL ES. There are many problems with the current situation. Applications that want to use graphics components beyond OpenGL 1.2 have to "jump through hoops." It's even more work to use GLES with GLX, or to use desktop OpenGL with EGL. The implementation of indirect rendering, a feature that allows OpenGL commands to be encapsulated in the X protocol stream and sent over the wire to the X server, is "completely fail": it performs poorly and supports OpenGL up to only version 1.4—or 1.5 with a lot of effort. The specification requires indirect rendering to be supported, but the number of legitimate use cases is quite small. And the presence of this rarely used feature sometimes creates problems when applications accidentally trigger indirect rendering and force OpenGL back to version 1.4, leading to user "rage" and bug reports. Ian then went through the proposed solution. The first step is to split libGL, so that the OpenGL and GLX components are separated into different libraries, called (say) libOpenGL and libGLX. Andy Ritger has sent out a detailed proposal to the mesa-dev mailing list describing how the split could be accomplished. Splitting the libraries will allow applications to mix and match components as needed, so that, for example, GLES and GLX can be easily used together by linking with the right libraries. Using both OpenGL and EGL together would become similarly straightforward. To maintain backward compatibility for old binaries that look for libGL, it would still be necessary to provide a legacy version of libGL that glues libOpenGL and libGLX together. Among the problems to be solved during the split are how to version the libOpenGL library and "get away from at least some of the GetProcAddress madness." That "madness" exists because the current ABI forces some applications to make calls to various "GetProcAddress" APIs (similar in concept to the GNU dynamic linker's dlsym() API) in order to obtain the addresses of multiple functions in the various libraries that constitute the OpenGL ABI. How the libOpenGL library would be versioned is an open question. Ian noted that the possibilities included ELF library versioning or embedding the version number in the library name, as is done with GLES. He also speculated about whether it would be possible to bump up the minimum OpenGL version supported by the ABI. The current implementation is required to support OpenGL versions as far back as 1.2. However, OpenGL 1.2 is now so old that it is "useless", though Ian still sees occasional bug reports for version 1.3. Once the library is split, Ian would like to see GLX deprecated. In addition to the problems caused by direct rendering, adding new GLX extensions is painful, because support must be added in both the client and the server. This can create problems in the cases where support is not added simultaneously on both sides: the client may end up sending unsupported protocol requests to the server and "they die in a fire." One recent fire-starter was the GLX_ARB_create_context feature: support appeared in the X server only in September, more than a year after client-side support was added to GLX. By contrast, EGL does not have this problem because support needs to be added only on the client side. In other words, getting rid of GLX will allow new features to be shipped to users much more quickly. A prerequisite for deprecating GLX is to have Linux distributors ship EGL by default. Most distributions do provide EGL, but Ian supposed that it is not generally included in the default install. However, Martin Gräßlin said that KDE started optionally depending on EGL about two years ago, so it is now part of the default install in most distributions. Later, Ian noted that encouraging the move to EGL may require the creation of a GLX-to-EGL porting guide; while there is independent documentation for both GLX and EGL, there seems to be none that explains how to port code from one to the other. A lot of the required source code changes can be accomplished with some simple scripting, but there are also a few deeper semantic differences as well as a few GLX features that don't have direct counterparts in EGL. Another important step is to make OpenGL ES part of the OpenGL ABI. Bart Massey's XDC2012 presentation bemoaned the fact that developers are not making applications for X. Ian said that the reason they are not is that they're too busy making applications for mobile platforms. So, by enabling the OpenGL ES ABI that every developer uses on mobile platforms, it becomes possible for developers to use X as a test bed for mobile applications; it also becomes possible to port applications from Android and iOS to X. One final step would be to update the loader/driver interface. This interface defines the way that libGL or libEGL talks to the client-side driver that it loads, and the way that the GLX module in the X server talks to the driver to load and route GL calls to it. This will probably be the hardest step, and it may take some time to resolve the details. As a side note, Ian pointed out that if indirect rendering is dropped, it will probably make the task quite a bit easier, because it won't be necessary to support the loader/driver interface inside the X server. Ian's presentation was followed by some audience discussion of various topics. There were some questions about compatibility with old applications. Ian thinks that compatibility requirements mean that it will probably be necessary to ship a legacy version of libGL for the indefinite future. There was some discussion on how to handle multiple versions of libOpenGL in the future. Some audience members seemed unclear on what options were available, but others were confident that ELF symbol versioning, as used in the GNU C library, would be sufficient. Later, Chad Versace expressed concerns about ensuring that any proposed solution also worked when using the Clang compiler and the gold linker. Ian noted that there will need to be some more investigation of the requirements and build environments before any final decisions are made. Bart Massey expressed concern that indirect rendering seemed to be going away with no replacement in sight. He noted that he'd had students who had been happy users of indirect rendering for use in certain environments with limited hardware. Ian suggested that a VNC "send the pixels across the wire" type of solution might be the way to go. Later, Eric Anholt suggested that the loader could detect that it is communicating over the wire with the X server, open the driver as normal, and then transmit images over the wire with PutImage, with the application being more or less unaware of this happening. There are still many details to be resolved regarding the proposed reworking of the OpenGL ABI. However, there seemed near unanimous agreement that the proposal described by Ian was the right direction, and it seems likely that, once some design details have been resolved, the work will commence quite soon. The X.Org wiki has a pointer to the video of this presentation. (Log in to post comments)
XDC2012: OpenGL futures Posted Oct 5, 2012 22:53 UTC (Fri) by krakensden (subscriber, #72039) [Link]
Wasn't Dave Richards (http://davelargo.blogspot.com/) running a bunch of thin clients with indirect rendering a while ago?
XDC2012: OpenGL futures Posted Oct 6, 2012 0:53 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
Also look at project Caua that Maddog is working on.
In part this is the 'thin client' approach to providing service.
XDC2012: OpenGL futures Posted Oct 8, 2012 19:08 UTC (Mon) by anholt (subscriber, #52292) [Link]
If you look at the desktop environment in his screenshots, it's not using OpenGL anyway.
The important thing to remember with regards to indirect rendering: It's cheaper to ship pixels over the wire than the commands to render those pixels from other primitives. We should see improvements in performance by going to this new over-the-network rendering plan. As a bonus, you'll get improved application compatibility because it will see a sane version of OpenGL instead of the crippled, untested version you get by using GLX.
XDC2012: OpenGL futures Posted Oct 9, 2012 8:27 UTC (Tue) by renox (subscriber, #23785) [Link]
> It's cheaper to ship pixels over the wire than the commands to render those pixels from other primitives.
Cheaper in what measure? Not in bandwith usage..
XDC2012: OpenGL futures Posted Oct 9, 2012 18:28 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
If it was cheaper to just shovel pixels around, the GPU graphics cards would never have taken off.
They were built because the internal bus on the motherboard was slow enough that it's faster to send primitives to the processor on the GPU and have it process things than it is to do it all on the CPU.
how can the local bus be too slow, but the network is fast enough?
XDC2012: OpenGL futures Posted Oct 9, 2012 19:16 UTC (Tue) by rleigh (subscriber, #14622) [Link]
I would think that the amount of computation and memory bandwidth required to *render* an image into the colour buffer is vastly more than the amount of computation and network bandwidth required to ship the rendered colour buffer over the network. I'm not suggesting that this would necessarily result in better performance than GLX, but I think that two separate issues are being mixed up here.
Regards,
XDC2012: OpenGL futures Posted Oct 9, 2012 20:05 UTC (Tue) by mjg59 (subscriber, #23239) [Link]
A modern GPU can render over a billion triangles per second. Do you think it's going to take more bandwidth to transfer a billion triangles per second or 60 frames of HD video?
XDC2012: OpenGL futures Posted Oct 10, 2012 0:24 UTC (Wed) by dgm (subscriber, #49227) [Link]
Triangles are usually not enough, you want textures too. If you're playing video you are faced with some though choices:
- stream the compressed video (the display has to be able to decode the format). - transcode the video to a format supported by the display (expensive without dedicated hardware). - stream uncompressed video (back to 60 frames of HD). Clearly, the best option is hardware assisted transcoding. But then, why not use that hardware to compress on the fly the HD video, and keep the whole architecture a lot more simple in the first place? Food for thought.
XDC2012: OpenGL futures Posted Oct 10, 2012 1:43 UTC (Wed) by mjg59 (subscriber, #23239) [Link]
Right, once you're dealing with a significant number of textures then the texture load alone is probably more expensive than just pushing the pixels over the wire. Acceleration primitives make sense when you're doing nothing but drawing rectangles, but once you're beyond that point the reason you're using a GPU is because it's faster at turning your scene into a 2D image than the CPU is, not because you're using less memory bandwidth that way.
XDC2012: OpenGL futures Posted Oct 11, 2012 10:15 UTC (Thu) by nix (subscriber, #2304) [Link]
Still, doing the rendering at the client side when the client side is networkologically remote means that your X server (which is very likely to have a fast 3D-capable GPU) will see its GPU going unused, while the server (rather less likely to have a GPU) suddenly has to do GPU tasks... probably using llvmpipe. Ah well.
XDC2012: OpenGL futures Posted Oct 11, 2012 15:17 UTC (Thu) by raven667 (subscriber, #5198) [Link]
If this is a significant concern, for a large multi-user desktop server, then you can stuff GPUs into it to offload rendering, this is what other VDI solutions support such as VMware.
XDC2012: OpenGL futures Posted Oct 10, 2012 13:12 UTC (Wed) by renox (subscriber, #23785) [Link]
If you want to display an office desktop you only need a few of these drawing instructions.
If you want to display a game, that's different, then sending images may reduce the bandwith needed.
XDC2012: OpenGL futures Posted Oct 10, 2012 14:49 UTC (Wed) by mjg59 (subscriber, #23239) [Link]
Even an office desktop is a 3D composited environment now, with potentially arbitrary levels of complexity depending on application behaviour. Focusing on the simple case just means that you end up limiting UI design for no good reason.
XDC2012: OpenGL futures Posted Oct 11, 2012 12:09 UTC (Thu) by renox (subscriber, #23785) [Link]
No good reason? Reducing the bandwith used in many common situations (when you don't run a game or a video) is a *very good* reason.
That said, you do have a point that at some point the bandwith used by the rendering commands(and textures) could be higher than sending a bitmap. So the "ideal" (but complicated solution) would be to send the rendering commands&texture when their size is smaller that the size of a screen bitmap otherwise send the bitmap. Of course, this is made more complicated that you can send the diff between two bitmaps or/and compress the bitmap, but this has a latency cost..
XDC2012: OpenGL futures Posted Oct 11, 2012 15:24 UTC (Thu) by raven667 (subscriber, #5198) [Link]
My understanding is that sending drawing commands has a major latency cost. The way that was fixed in the X protocol was to run a proxy or local renderer where the client app runs and then ship bitmaps, like NX and xpra. You can still do smart things with caches, icons, prerendered font glyphs, offscreen scroll areas, prerendered buttons, etc. should be cachable at the display side.
XDC2012: OpenGL futures Posted Oct 11, 2012 15:46 UTC (Thu) by renox (subscriber, #23785) [Link]
> My understanding is that sending drawing commands has a major latency cost
Huh? Only if you want to re-read the output of these commands (a bad thing to do anyway) or if you do 'cmd1; wait for status; cmd2; wait for status' otherwise why should there be a difference?
> The way that was fixed in the X protocol was to run a proxy or local renderer where the client app runs and then ship bitmaps, like NX and xpra.
Are you sure that this is what NX do?
> You can still do smart things with caches, icons, prerendered font glyphs, offscreen scroll areas, prerendered buttons, etc. should be cachable at the display side.
For X, yes, for Wayland I don't think so as it has no server side drawing commands (short of sending buffer without knowing where they will be displayed).
XDC2012: OpenGL futures Posted Oct 11, 2012 16:16 UTC (Thu) by raven667 (subscriber, #5198) [Link]
You may think its a "bad thing" for the app to have access to it's rendered image but this is often required on modern apps (not xterm, xclock, xman and xeyes) which can lead to a perverse result where all the drawing commands and bitmaps are sent from the app to the display, rendered, then shipped back to the app. If you want a simple rendering model for remote apps then HTML/CSS/JS seems a better bet.
Actually think about the rendering needs of a modern web browser, what you can do from CSS and WebGL and what that requires of the underlying graphics subsystem.
Wayland is just the protocol to the output buffer on the display, you have free reign on what the remoting protocol looks like. You can make a virtual wayland framebuffer and send that or do something toolkit specific or use X or make some new standard protocol that the toolkits can output to.
XDC2012: OpenGL futures Posted Oct 11, 2012 19:55 UTC (Thu) by renox (subscriber, #23785) [Link]
>You may think its a "bad thing" for the app to have access to it's rendered image but this is often required on modern apps
Then they should do their own rendering locally, not require that the display protocol is limited to local image rendering..
>If you want a simple rendering model for remote apps then HTML/CSS/JS seems a better bet.
That's a totally different use case: the resources used are not comparable..
>Actually think about the rendering needs of a modern web browser, what you can do from CSS and WebGL and what that requires of the underlying graphics subsystem.
WebGL is the same use case as games..
XDC2012: OpenGL futures Posted Oct 11, 2012 20:20 UTC (Thu) by raven667 (subscriber, #5198) [Link]
>>You may think its a "bad thing" for the app to have access to it's rendered image but this is often required on modern apps
>Then they should do their own rendering locally, not require that the display protocol is limited to local image rendering..
They can be two different protocols, the protocol for talking to a local display doesn't have to be the same protocol for remoting they are different use cases with different requirements.
>>If you want a simple rendering model for remote apps then HTML/CSS/JS seems a better bet.
Those cases are blending together more and more. There are three ways to deploy an app, native local, native using a remote display protocol or HTML/CSS/JS and the HTML app deployment method is becoming the norm, local apps are becoming niche and remote display apps are a tiny case compared to the other two.
>>Actually think about the rendering needs of a modern web browser, what you can do from CSS and WebGL and what that requires of the underlying graphics subsystem.
Not exactly it's the underlying accelerated GUI toolkit functions needed by your standard HTML5 Canvas or WebGL app. You have text and images and 3d objects all composited and styled together, not just an empty buffer that the application manages like a game. Unless your point is that a web browser should just be given a GLES framebuffer and implement it's own GUI toolkit internally to support modern features.
XDC2012: OpenGL futures Posted Oct 12, 2012 7:28 UTC (Fri) by renox (subscriber, #23785) [Link]
> They can be two different protocols, the protocol for talking to a local display doesn't have to be the same protocol for remoting they are different use cases with different requirements.
This would be a regression compared to X.
As for WebGL, it is a toy until it can be made reasonably secure.
XDC2012: OpenGL futures Posted Oct 12, 2012 15:44 UTC (Fri) by khim (subscriber, #9252) [Link] As for WebGL, it is a toy until it can be made reasonably secure. It's the other way around: nobody will bother to make it reasonably secure till it's more than a toy. You may claim that it's totally wrong way to build "anything secure", but this is how things are done in real world.
XDC2012: OpenGL futures Posted Oct 11, 2012 19:28 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
my understanding of NX isn't that it does any local rendering, but that it instead shortcuts the roundtrips to the server by intercepting calls and returning the values directly.
Due to it's history, X can't assume much about what's on the other end, however in modern implementations there really is a lot that is always going to be available. With pure X there is a lot of chatter of the type 'do you support feature Y' that can just be short circuited.
In addition, there were drawing primitives that were synchronous (cmd1l result cmd2 result...) because systems were so short of resources that any of the commands could fail due to running out of them. Nowadays the resources are so plentiful that this is almost always a waste of time, so something like NX can return a result immediately (unless the system on the other side has said that it's getting low on resources)
you see the same type of thing in WAN accelerators for windows networking protocols. In both cases the real limit isn't bandwith, it's a chatty protocol that works well on low-latency systems, but can suffer badly when on a high latency system
XDC2012: OpenGL futures Posted Oct 11, 2012 20:03 UTC (Thu) by raven667 (subscriber, #5198) [Link]
> my understanding of NX isn't that it does any local rendering, but that it instead shortcuts the roundtrips to the server by intercepting calls and returning the values directly.
I think that's true for older apps that use the core X protocol, xman, xterm, xeyes, xclock are accelerated this way. For newer apps using newer toolkits like Gtk2/3, Qt, etc. I think there is a lot more bitmap shuffling because the client app toolkit depends on being able to do compositing, blending, antialiasing, etc. and shipping the result to the display.
XDC2012: OpenGL futures Posted Oct 10, 2012 5:06 UTC (Wed) by khim (subscriber, #9252) [Link] If it was cheaper to just shovel pixels around, the GPU graphics cards would never have taken off. Sorry, but no. GPU graphics cards have "taken off" as separate, dedicated hardware for 3D. In in about the same era first graphics chipsets (which used the fact that it's cheap enough to just shovel [fully rendered] pixels around) arrived. They went out of favor for a long time because it's hard to build realistic 3D render but it's very easy to attach RAMDAC to anything. Later it was found that GPUs need fast low-latency pipes and they got them and today they have dedicated connection to CPU or are even totally merged with CPU. No, if anything GPUs evolution show that you need fast, wide pipe between GPU and CPU and then you can have much slower and narrower pipe which delivers fully-rendered picture. IOW: X architecture (where 3D was in X Server) is long obsolete, it's time for something like Wayland (where CPU and GPU used are local and only fully processed picture is sent over the wire). Of course fully-rendered picture without any compression is still too large, so we need to compress it somehow - but this is separate issue.
XDC2012: OpenGL futures Posted Oct 11, 2012 10:18 UTC (Thu) by nix (subscriber, #2304) [Link]
Of course, you don't need Wayland for that -- as Eric pointed out, you can implement the send-images-over-the-wire thing without changing anything about the current user-visible API/ABI. It's not like the clients care where the rendering is being performed: they just care that it *is* performed.
The answer to everything is not always "Wayland".
XDC2012: OpenGL futures Posted Oct 12, 2012 15:40 UTC (Fri) by khim (subscriber, #9252) [Link] That's why I've said something like Wayland. I'm pretty sure it's possible to build system which does rendering locally and is backward-compatible, too. Do we want that? That's different question.
XDC2012: OpenGL futures Posted Oct 12, 2012 15:59 UTC (Fri) by raven667 (subscriber, #5198) [Link]
Backward compatible with X? The X server can output to Wayland just fine AFAIK so there should be no trouble continuing to use the X protocol for as long as you wish. It's backwards compatible in much the same way XQuartz provides compatibility.
|
|||||||||||||