LWN: Comments on "FOSDEM09: RandR 1.3 and multimedia processing extensions for X"

Acceleration Architectures, state of the art.

daenzer — Sat, 28 Feb 2009 13:10:25 +0000

> is EXA a one-for-one replacement for XAA, or just for some use cases?

EXA is designed for modern compositing environments, whereas XAA is
excessively optimized for every little bit of core X11 rendering operations
and hasn't really evolved much this millenium. So while EXA should be on par
or faster with modern applications (especially with a compositing manager,
even just xcompmgr -a) it'll probably never be able to match XAA in all
cases with older applications, at least not without turning into the same
kind of monster.

Acceleration Architectures, state of the art.

daenzer — Sat, 28 Feb 2009 12:53:44 +0000

> On the other hand, UXA is used by the intel driver to play nice with GEM
> and KMS. It started as a straight clone of EXA, though. I don't know if
> the UXA features can be ported back to EXA in the server. You'd probably
> have to ask keithp or anholt.

And take their answers with a big grain of salt... There's nothing to be
ported back, really. UXA basically just removes the EXA code the intel
driver doesn't need. However, it could 'play nice with GEM and KMS' (i.e.
use buffer objects for pixmap storage) in pretty much the same way with EXA,
and the code that was removed would be inactive.

FOSDEM09: RandR 1.3 and multimedia processing extensions for X

phdm — Fri, 27 Feb 2009 12:51:13 +0000

For the image part, this reminds me of XIE (X11 Image Extension), that 15 years ago allowed X servers to display jpeg images sent uncompressed using the X protocol. I miss XIE, that was dropped 5 years ago by xorg (or was it xfree ?), but nowadays we should have XMME (multimedia), not XIE + audio.

FOSDEM09: RandR 1.3 and multimedia processing extensions for X

Duncan — Thu, 26 Feb 2009 15:10:08 +0000

AFAIK, in theory at least, RandR 1.2 supported not panning, but specifying
the viewport location. At least xrandr had parameters for it. It is/was
thus possible (in theory) to use xrandr (my usage was in a script) to set
the viewport to wherever one wanted it, and one could (in theory)
programatically pan by repeatedly invoking said script or application to
move X and Y pixels.

You'll note the "in theory"s however. Apparently not all video drivers
implemented the settable viewport functionality. The resolution change
and virtual size functions of my xrandr invoking script worked just fine,
but I've never been able to get the viewport parameters to work -- xrandr
takes them -- they just don't do anything. When the resolution is set
below the maximum virtual size, the viewport stays nailed to the top left
corner, no matter /what/ position parameters I give xrandr. This is with
xf86-video-ati, 6.9 and 6.10 at least, and I think I tried it with 6.8 but
that was far enough back IDR for sure, on a Radeon 9200SE AGP with dual
outputs, DVI-I and VGA, formerly running dual 22" CRT analog monitors in
1600x1200 stacked for 1600x2400, now running dual 24" LCDs 1920x1200
stacked for 1920x2400. (Yes, I need to upgrade video cards.)

Thus I could change resolution with the orientation staying correct, but
for the most part it wasn't of much use since all I could see at the lower
resolutions was the top left corner of the screen, no matter where I told
it to put the viewport. Still, I was able to work around that problem to
split-resolution top and bottom to play the only piece of proprietaryware
I have left, Master of Orion original DOS edition (1993 update copyright),
full-screen (single-screen) in DOSBOX at 640x480, while keeping the other
screen normal resolution to run my ksysguard graphs and a music player,
etc. That worked even tho I couldn't pan, because I could use kwin's
absolute positioning options to put dosbox/orion right under the 640x480
viewport of the one screen, while keeping the other at normal resolution.

Why bother?

muwlgr — Thu, 26 Feb 2009 11:24:27 +0000

Exactly. Hardly anyone would want to stream a sequence fo JPEGs over the network. Streaming of MP3/MP4/AVI/OGG/Theora right to the X-server, then decoding and playing it there, would look much more savvy and tasty. I think so.

Why bother?

jlokier — Tue, 24 Feb 2009 03:20:59 +0000

Yet I still have to kill the PulseAudio daemon sometimes before some apps produce sound. I don't believe the hype about PA's ability to imitate everything else, because it plainly fails at it.

Acceleration Architectures, state of the art.

dbnichol — Mon, 23 Feb 2009 17:48:02 +0000

It has definitely been the plan for a long time to get all drivers onto EXA. XAA is only receiving bug fixes in the server. On the other hand, UXA is used by the intel driver to play nice with GEM and KMS. It started as a straight clone of EXA, though. I don't know if the UXA features can be ported back to EXA in the server. You'd probably have to ask keithp or anholt.

Acceleration Architectures, state of the art.

jcristau — Mon, 23 Feb 2009 16:54:32 +0000

Only the old type1 libXfont backend has been removed, support for type1 fonts from the freetype backend is still there.

Why bother?

helge.bahmann — Mon, 23 Feb 2009 12:04:38 +0000

"Audio over network" works well with for example PulseAudio, but what
doesn't work is "audio+video+synchronization over network". While you
could tie X+PulseAudio server close enough together to make this feasible,
I suspect the result is going to be quite messy.

As for MPEG-4... the idea would of course be to provide an AVC
decompressor accessible through the X protocol as well -- reencoding at
the client is quite obviously highly undesirable (maybe useful as an
emergency fallback, but that's it).

And last time I checked, there was no media player that could play
through "ssh -X" :)

FOSDEM09: RandR 1.3 and multimedia processing extensions for X

helge.bahmann — Mon, 23 Feb 2009 11:57:41 +0000

That would work, but it requires quite a bit of "protocol" between the X
server and the decompressor (select which compressed/uncompressed images
to retain or delete inside the decompressor process).

Getting the concurrency benefits is however more tricky -- if a client
calls "Decompress" in the X server, the X server must delegate the
operation to the helper process, suspend the calling client's request
queue (the X server is single-threaded!), receive completion, resume the
client's request queue etc.

One thing I dislike about the "helper process" approach is that I am not
yet sure how the interface should look like -- currently all decompressors
are for efficiency reasons (cache locality) "iterator-based": you pass
compressed data (as well as decoding dependencies) in, and what you get is
an iterator that traverses the image top to bottom and yields
horizontal "bands" of the image. Currently you receive pixels, usually in
some sort of YCrCb format, which must then be converted to RGB etc. (but
the interface should later also allow getting e.g. DCT coeffecients +
motion vectors for hardware-assisted decode). Mapping this model to the
X/decompressor process is probably not going to work due to excessive
context switches.

Acceleration Architectures, state of the art.

roblucid — Sat, 21 Feb 2009 14:00:58 +0000

> Surely 99.999% of graphics implementations will be one of Intel, Nvida,
> AMD/ATI and Via (I've not come across anything else for years now, and
> certainly not one which allows you to use multiple adapters).

Some of us still have Matrox cards, which were a well supported card if you liked Open Source drivers, not so long ago and thus reccomended. I don't know if the dual monitor output still works, but it'd be bad form to drop support of cards that were FOSS favoured, only five years ago.

Why bother?

shapr — Fri, 20 Feb 2009 17:10:52 +0000

The point is that ssh -X would one day be able to forward sound and video across the network, as if you were sitting in front of the computer itself.

Acceleration Architectures, state of the art.

jamesh — Fri, 20 Feb 2009 14:47:07 +0000

LibXfont is pretty much just the core font support library. Any application that has been ported to fontconfig (using Xft or cairo) won't be using it, and should retain Type1 support through the use of freetype.

Looking at the libXfont 1.4.0 release notes, it looks like you were correct about Type1 fonts not working with the new version. That said, I totally understand the desire to reduce the size/complexity of the core font code: the feature is deprecated, but can't easily be removed as it is not an extension to the X protocol.

If they can support Type1 fonts through FreeType with minimal effort, then great. If they can't, it isn't a huge loss.

Why bother?

nix — Fri, 20 Feb 2009 11:03:29 +0000

esd gives you audio over the network. The advantage of pulseaudio over it
is its pluggability, its lower CPU consumption, its lower latencies, its
huge swarms of extra features, the way it can imitate or output to
virtually anything, its active development, and the fact that it generally
doesn't suck.

Acceleration Architectures, state of the art.

nix — Fri, 20 Feb 2009 10:44:51 +0000

It's possible. I'm not entirely sure *where* libXfont is used, to be
honest (something or other to do with core fonts). I used to know; I need
a prosthetic memory.

Acceleration Architectures, state of the art.

jamesh — Fri, 20 Feb 2009 05:05:52 +0000

While the Type1 code is gone from libXfont, it still has the FreeType backend. Given that FreeType can read Type1 fonts, is it possible that they were just removing a superfluous backend and switching to the code base that everyone else uses? (including the apps that use Xft).

Acceleration Architectures, state of the art.

wookey — Fri, 20 Feb 2009 00:57:12 +0000

Those manufacturers may account for nearly all PC/x86 type machines, but that is not the whole X space. Modern ARM chips, as used in things like smartphones, and soon to be seen in netbooks, now have 3D graphics cores in them, such as the powerVR from Imagination in Cortex A8-based CPUs. This looks likely to be a non-trivial fraction of devices in the future, so X shouldn't be ignoring them, although my understanding is that Imagination are about as helpful as nVidia when it comes to free drivers.

Why bother?

NAR — Thu, 19 Feb 2009 20:15:29 +0000

Exactly what's the point of these multimedia extensions? We already have at least one framework for audio over network (I've heard that this is the advantage of pulseaudio over e.g. esd) and I don't really see why would anyone decode e.g. MPEG4, then encode the result picture to JPEG, send over the network, then decode it again. Isn't that one decoding step lossy enough? On the other hand mediaplayers nowadays are usually able to play over network directly.

Acceleration Architectures, state of the art.

kingdon — Thu, 19 Feb 2009 18:59:37 +0000

Those four vendors are about 92% based on the statistics at http://smolt.fedoraproject.org/static/stats/by_class_VIDE... .

Acceleration Architectures, state of the art.

nix — Thu, 19 Feb 2009 12:48:53 +0000

As far as I can tell the intention is for EXA to replace XAA entirely. As of (very) recently it even does core fonts nice and fast, so unless you rely on things like stipple acceleration (!) I'd say it's getting there quite well.

(downside: it looks like Speedo and Type1 core fonts are about to stop working, as the code to handle them has been torn out of libXfont 1.4.0. I doubt anyone misses Speedo, but Type1 may very well be missed, because they *do* still work client-side.)

Acceleration Architectures, state of the art.

ctg — Thu, 19 Feb 2009 10:02:30 +0000

It is a little hard to see where development is going:

is EXA a one-for-one replacement for XAA, or just for some use cases?
Is UXA a replacement for EXA, or a branch of some ideas that will be remerged?

Surely 99.999% of graphics implementations will be one of Intel, Nvida, AMD/ATI and Via (I've not come across anything else for years now, and certainly not one which allows you to use multiple adapters).

Is there a case for streamlining X down to EXA/UXA and drivers which support one of those....?

I've rootled around on xorg.freedesktop.org, but can't really find obvious answers to these sorts of questions.

Users of, say Matrox, would probably be pretty content with the feature set of Xorg 7.current.

FOSDEM09: RandR 1.3 and multimedia processing extensions for X

aleXXX — Thu, 19 Feb 2009 07:20:20 +0000

Are these multimedia extensions similar to MAS ? Does this actually still
exist ?

Regarding RandR: I was quite surprised when I noticed that suddenly
virtual screens (now called panning) didn't work anymore after upgrading
my distro (to Slackware 12.1). The keyword "virtual" is still supported
in xorg.conf, but it does something different now. It sets up a frame
buffer of the given size, but you can't see it, since you can't move to
make it visible :-/
IMO it's not very good style to keep a keyword/function with the same
name but make it do something very different. If they would have just
removed support for "virtual", X would have told me that this is not
longer supported, which would have saved me a lot of time.

Alex

FOSDEM09: RandR 1.3 and multimedia processing extensions for X

daniels — Thu, 19 Feb 2009 02:57:16 +0000

I think our current position on threads could probably be summed up as 'we did it once and it went really badly; doing it again is going to be really, really hard'. It doesn't mean it's a bad idea, though. :)

FOSDEM09: RandR 1.3 and multimedia processing extensions for X

quotemstr — Thu, 19 Feb 2009 02:44:25 +0000

Why not spin the decompression off into a separate process and use some kind of shared-memory ring buffer instead? You get the concurrency benefits without the security problems, since the spun-off process can have vastly reduced privileges.