LWN: Comments on "LPC: Life after X"

LPC: Life after X

Lestibournes — Thu, 09 Dec 2010 22:36:25 +0000

In order for X to successfully deal with Wayland it needs to do the following:
1. Reframe the discussion so that it's not X vs Wayland, but Wayland being a replacement for the layer of X that takes care of local display.
2. Provide a user experience on the local machine that is not noticably infirior to the experience of native Wayland apps.
3. Provide a superior experience for remote apps.
4. Provide an API that is at least as attractive to all developers as native Wayland.

For as long as X manages this, it will remain the standard API for GUIs. But the moment X falters in any one of this aspects, it can be completely gobbled up by Wayland.

Simple is good enough

Lestibournes — Thu, 09 Dec 2010 19:16:22 +0000

Maybe something like this will work:
1. Program A indicates that it is ready to supply data by writing its identifier to clipboard/source.
2. Program B requests the data by writing its identifier to clipboard/destination.
3. Program A writes the data files in clipboard/data.
4. Program A indicates that it finished writing the data by erasing the content of clipboard/source and clipboard/destination.
5. Program B reads the data files from clipboard/data.

If no one requests the data from Program A, then it will still dump the data when it terminates. The only weaknesses I detect are a delay when the Paste operation is performed, and that the data will be lost if Program A crashes. There should be a separate clipboard folder for each session to avoid conflicts such as two users who share an account and override each other's Copy operations.

Simple is good enough

i3839 — Sat, 27 Nov 2010 10:39:52 +0000

> I would really like to be clear that I am not trying to argue for the
> sake of arguing here, but rather because responding to the points you
> make forces me to think things through myself.

Same here, we're trying to figure out if a list of formats is really needed, or if always providing only one and having convertors is sufficient. This choice determines the API, so it's pretty important.

Only having one format and providing convertors is simpler, but less complete. My main concern is that it's not always sufficient, or that it makes implementing copy harder than necessary for some applications, because they have to create one "complete" format and convertors.

Another concern is that you convert from simple->complex->simple, when also supporting a complex type, hoping that the "simple" in the end is the same as what you started with. So the unrelated complex type makes simple types more complex too, with too much room for errors in my opinion. Or in other words, copying simple types is not simple anymore, if you also copy a complex one.

Lastly, I don't really see a way to support multiple types when pasting. It should be the pasting program's decision what type to paste, if it supports multiple types. I don't see another way than supporting a list of types in the pasting API anyway, and then you can as well support lists in the copy API too.

I think you make too many assumptions about what the user or pasting program expects in your line of thinking.

All in all I think the automatic convertion idea is good, but not always sufficient. Combined with today's multiple format support in applications, I think it's best to support multiple formats, but to encourage convertor usage when possible.

Then when someone pastes something the lists are compared, and if they have no common format, a convertor is used.

A list of formats is basically "more of the same", so I think the added complexity, both for the API and implementation, is small enough.

Now we just have to find some time to implement this. I think I'll give it a stab next week. I'll keep you informed (my email address is indan@nul.nu).

Simple is good enough

mjthayer — Fri, 26 Nov 2010 22:26:12 +0000

> Problem is that in the case of html, you generally lose the formatting information because that's not in the part you copied, but higher up or in a css file. So there is no native format, you don't want to copy raw html code into a word processor, nor the plain text, but something that more or less looks like what you copied. Not to mention that usually the selected part is "broken" html because not all tags are closed.

You also have to ask, when an application puts HTML data into the clipboard, what data it is actually putting there. When I select a section of text, pictures and whatever in Firefox and copy I get HTML data in the clipboard. But Firefox can't just put the source of the document from the point where the selection begins to the point where it ends into the clipboard, as it is announcing HTML data, and as you point out, that wouldn't be HTML, it would be broken HTML. So Firefox has no choice but to massage the HTML data anyway, and if it is doing that already, adding the style information inline is no great hardship.

Of course, if you want to reuse that data as is as HTML for some other web page then you are probably out of luck, but if you think of it that makes no sense anyway - if you want to do that you should probably be copying the source of the HTML as plain text. If you select and copy part of a page in Firefox, chances are that what you are actually about to do is to paste it either as plain text (the text visible on the page, not the HTML source) or as formatted text into e.g. OpenOffice.

And if you were copying the data inside some visual HTML editor, it would probably still not make sense for the editor to insert the data as naive HTML - chances are there would be no way to paste the data in any form resembling the source of the page the editor was generating, and in any case, if you were trying to get at the generated source it would make more sense to ask the editor directly than copying and pasting to get at it. In fact I would expect the visual editor to use some internal format which was not valid HTML at all when copying to the clipboard, but which another instance of the editor would know what to do with when pasting it. It might provide a filter to convert it to HTML, but not for the purposes of viewing the source - you don't use the clipboard for that - but rather as a stepping stone for converting it to OOXML or something else.

Hope that made sense, as I am rather short of sleep currently. I would really like to be clear that I am not trying to argue for the sake of arguing here, but rather because responding to the points you make forces me to think things through myself.

Simple is good enough

mjthayer — Thu, 25 Nov 2010 21:56:48 +0000

Just for interest I copied some text in Firefox and ran my clipboard format viewer. Here are the results:

$ ../tmp/viewclipformats
Found clipboard format: TIMESTAMP
Found clipboard format: TARGETS
Found clipboard format: MULTIPLE
Found clipboard format: text/html
Found clipboard format: text/_moz_htmlcontext
Found clipboard format: text/_moz_htmlinfo
Found clipboard format: UTF8_STRING
Found clipboard format: COMPOUND_TEXT
Found clipboard format: TEXT
Found clipboard format: STRING
Found clipboard format: text/x-moz-url-priv

Without knowing, it wouldn't surprise me if one of those contained both the html and the formatting information, which I think should be feasible with my proposal too.

Simple is good enough

i3839 — Thu, 25 Nov 2010 21:44:00 +0000

Problem is that in the case of html, you generally lose the formatting information because that's not in the part you copied, but higher up or in a css file. So there is no native format, you don't want to copy raw html code into a word processor, nor the plain text, but something that more or less looks like what you copied. Not to mention that usually the selected part is "broken" html because not all tags are closed. So it's not that simple and I don't think it's safe to get rid of the list support.

For images and other data formats with an obvious raw format are much easier and better suited for automatic convertion. That can be done automatically without changing the API.

Simple is good enough

mjthayer — Mon, 22 Nov 2010 14:52:29 +0000

> Well, the problem is that for complex types you can't easily convert from one to the other, because only the program doing the copying has all the info.

Actually the idea was that the application doing the copying provided the data in its native/internal format, which by definition should have all the information. It could always define an x- or a vnd. mime format for this and provide whatever converters it wanted to transform that data into other formats (they could probably double up as export filters too).

> Example: If you copy a bit of a webpage, it can be either plain text, the raw HTML code, or formatted text depending on the style etc. Converting to plain text is almost always possible, but anything else doesn't really work. So a single mime type isn't always sufficient.

In this case the native format is presumably "text/html", which should be convertable to either plain text or formatted text without the copying application even having to provide its own converters.

LPC: Life after X

dododge — Mon, 22 Nov 2010 10:56:49 +0000

Yeah NS3 or NS4 sounds about the right timeframe. I mostly remember noticing that it was much slower than it should have been and having a real WTF?! moment when I traced the X connection and saw all those images flying by.

LPC: Life after X

nix — Mon, 22 Nov 2010 08:18:16 +0000

IIRC this was only true in Netscape 4, which is when Netscape decided to give the implementation job to a gang of crack-addled monkeys rather than JWZ. (I think that was his phrase...)

Quality suffered accordingly.

LPC: Life after X

dododge — Mon, 22 Nov 2010 06:31:14 +0000

There are just too many impedance mismatches with the native window system, and you need add the huge X11 server software.

It's better than it used to be. In my current office the workstations are controlled by corporate IT, and so I'm forced to use a Windows XP desktop to do my Linux development. XMing is a small and free native X server for Windows that is very easy to install and run, and can mix remote X11 clients with native Windows apps on the same desktop. Many current X11 applications use client-side text rendering, which greatly reduces the headaches of getting core fonts working well on the Windows side. Microsoft's own TweakUI will enable X-style pointer focus to work across the entire desktop, which helps to smooth things out. By adjusting a registry setting I even have caps lock mapped to work like a second left-control key across all applications, and since it's part of my profile (rather than a system setting) it doesn't cause trouble for other users of the same machine.

It's certainly not perfect, but my desktop is about 3/4 Linux+X11+ssh applications and aside from the native ones not doing middle-mouse pasting it's very easy to forget that there's any Windows stuff on there at all.

LPC: Life after X

dododge — Mon, 22 Nov 2010 05:59:50 +0000

Here's a real example from the 10MB days: early web browsers often had their their application logo at the top corner of the window, and would run a little animation loop when loading a page. I believe in Netscape's case it would send the entire logo image to the X server for every frame of animation. This worked fine on the same machine because it could use things like XSHM to send the image data out-of-band, but when running remotely it meant it was continuously and repeatedly encoding and transmitting each frame of the animation through the X protocol.

If they had designed for network transparency they presumably would instead have cached the frames as a handful of server-side pixmaps and flipped between them. It seems like a little thing, but as I recall this minor oversight made it perform terribly over a LAN.

LPC: Life after X

9000 — Sun, 21 Nov 2010 13:00:49 +0000

There's a project aimed to achieve exactly this: http://telepathy.freedesktop.org/wiki/Xzibit

LPC: Life after X

Wol — Sat, 20 Nov 2010 01:04:16 +0000

Actually, in my house, that's true about 50% of the time.

Soon, probably, to be 33% of the time ... :-)

(I've got a Phenom X3 as my main machine, an Athlon 1050 .75Gb ram machine I use as an x-terminal, and an Acer Aspire One that will probably soon have Smeegol on it and get used as an x-terminal too)

Cheers,
Wol

Simple is good enough

i3839 — Fri, 19 Nov 2010 22:13:00 +0000

> I changed that aspect of the proposal in a previous comment - in the new
> version a file containing a single mime type (with a well-known magic
> number) is saved to disk. To handle conversions, a (large) set of filters
> is installed on the system, and the application reading the clipboard
> must iterate through the installed filters to find ones which convert the
> file to a format it can use. This is roughly what BeOS did, which
> apparently worked well (or so I am told by a colleague who developed for
> BeOS).

Well, the problem is that for complex types you can't easily convert from one to the other, because only the program doing the copying has all the info.

Example: If you copy a bit of a webpage, it can be either plain text, the raw HTML code, or formatted text depending on the style etc. Converting to plain text is almost always possible, but anything else doesn't really work. So a single mime type isn't always sufficient.

There are two sides to a copy and paste system: The program ABI to do the copying and pasting, and the system ABI how it's actually done. I think the latter shouldn't be set in stone, only the former, to keep the implementation flexible. So all copying and pasting should happen through the system installed copy&paste library, or the copy and paste programs (simple front-ends for the lib).

(I still haven't found to start implementing this, hopefully next week.)

LPC: Life after X

tjc — Thu, 18 Nov 2010 18:23:18 +0000

+1 from me as well.

We have graphical toolkits which can implement dynamic themes, so it is no longer necessary to run a separate window manager to impose a theme on the system.

It's not the theme that I'm concerned about, it's consistent window management, and the ability to do unpopular things, like lower windows with the mouse.

LPC: Life after X

pivot — Thu, 18 Nov 2010 14:08:21 +0000

Wow! Seldom is the peanut gallery more active than when it comes to discussions about X11!

You must be joking

renox — Thu, 18 Nov 2010 09:45:10 +0000

>> Really the worst problem is that application developers don't care. They refuse to "pay their taxes" as it has been called, by implementing features that require some work on their part to deliver a better experience for the end user across all applications <<

Which is why, I always look at propositions which want to replace things done by a common server by things done by all the applications with a lot of skepticism..

One of the example is XCB which AFAIK is still not implemented by Qt or GTK even though it was supposed to allow better threading.

A similar button does exist.

gmatht — Thu, 18 Nov 2010 03:52:10 +0000

In Google-Chrome middle-click the (+) new tab button to open a new window with that URL, this also works for searches.

With Firefox, middle-click the icon to the left of the icon bar. However this does not work for searching for non-urls; see the patch to implement middle-clicking on the search icon at: https://bugzilla.mozilla.org/show_bug.cgi?id=414849. I would be particularly interested to know if you would find the functionality implemented by this patch useful.

Smooth transition? From what?

mcrbids — Wed, 17 Nov 2010 07:11:04 +0000

<QUOTE>Most applications today are written for Windows, PS3, Wii, iOS or Android. Not for X. Developers know toolkits (mostly GDI, but sometimes WPF or even Qt) and DirectX/OpenGL. They don't know X and then don't want to know X.</quote>

As an application developer with over 10 years of experience, I can say with certainty that this is not true, at least, not for me.

I don't write applications for Windows, PS3, Linux, Android, or *any* of the platforms listed. I write for the web browser! I write complex, data-driven applications and it's been a very, very long time since I wrote anything that wouldn't easily work on Win/Mac/Lin/Android/Iphone/Xbox and anything else with a reasonable browser.

The browser I most target is Firefox since it seems to be the most "Cross platform" although Chrome is close. I develop on Linux, it runs FF well, I don't worry about viruses and stuff like that, and can offer excellent compatibility with all my clients.

I don't want to replace X - I get the best of all possible worlds by making the specific rendering requirements of my applications something handled by the context of the user. And I use network transparency all the time - I can run several Firefox instances concurrently, on the desktop, as different users, without any danger of cookie or session interaction between browsers. As a web-based, network application developer, this is so incredibly useful!

Simple is good enough

mjthayer — Tue, 16 Nov 2010 22:42:06 +0000

> It depends on the program and context. If you select an image in a browser I wouldn't copy the image. But if it's an image editing program, I would.

This does happen currently though (I tested it earlier today).

> But copying it into a hundred formats isn't something you should do for every selection of any random thing (or ever, but if you do...).

I changed that aspect of the proposal in a previous comment - in the new version a file containing a single mime type (with a well-known magic number) is saved to disk. To handle conversions, a (large) set of filters is installed on the system, and the application reading the clipboard must iterate through the installed filters to find ones which convert the file to a format it can use. This is roughly what BeOS did, which apparently worked well (or so I am told by a colleague who developed for BeOS).

Simple is good enough

i3839 — Tue, 16 Nov 2010 22:26:16 +0000

It depends on the program and context. If you select an image in a browser I wouldn't copy the image. But if it's an image editing program, I would.
But copying it into a hundred formats isn't something you should do for every selection of any random thing (or ever, but if you do...).

Simple is good enough

mjthayer — Tue, 16 Nov 2010 14:11:04 +0000

> So if applications are smart they only copy simple things that are quick to copy when selecting, and only do the slow copy when users explicitly copy something.

Shouldn't applications be doing what the user asks them to rather than being smart? Selecting and middle click pasting an image works now. If the user selects it, should the application really assume that they don't want to paste it? Of course, it might still turn out that users don't select things often enough that the overhead would be a big issue.

Simple is good enough

i3839 — Tue, 16 Nov 2010 13:00:41 +0000

Not really. People probably don't want to copy and paste complex things
like images with select and middle mouse button. So if applications are
smart they only copy simple things that are quick to copy when selecting,
and only do the slow copy when users explicitly copy something.

You must be joking

wookey — Mon, 15 Nov 2010 16:24:42 +0000

ctrl-U used to provide this vital feature and then some eejit decided that that in a browser that should open a window with the page source in it. That was _such_ a painful decision and still enrages me on a daily basis on machines where I haven't persuaded the system to change the keybinding. (where did it come from?).

Ctrl-U can still be made to work (as 'clear line/box'), but it gets harder to find the rune every year. A button to prod for the same function would indeed be a useful alternative.

Like many here I find middle-button paste to be one of the finest things about GNU/Linux, and it's extremely tiresome when you get apps that don't do it right. I really hope it does not get sacrificed as part of the GUI re-architecting that it looks like we are headed for.

I use remote-X-over-ssh for graphical apps fairly regularly and it's extremely useful, but accept the argument that we can achieve much the same effect by other means (SPICE/VNC/NX/whatever). I hope that does indeed come to pass.

Simple is good enough

mjthayer — Mon, 15 Nov 2010 10:30:58 +0000

>> "Why does copying this image freeze the program for 30 seconds? Lol that doesnt happen under Windows."

> If that was also addressed to me, I will make an attempt at defending my proposal (consider the correction I added above as part of it).

Replying to myself. One clear weakness of my proposal would be that it might not work well with select and middle button paste, as selecting is something you do more often than copying (again, that would need testing to be sure). I would give it more chances of working with drag and drop (which I personally prefer over middle button paste, but I greatly fear I am in a minority here with that).

Simple is good enough

mjthayer — Mon, 15 Nov 2010 10:10:22 +0000

> "Why does copying this image freeze the program for 30 seconds? Lol that doesnt happen under Windows."

If that was also addressed to me, I will make an attempt at defending my proposal (consider the correction I added above as part of it).

* Clipboard data is written to a file on disk in order to share it, but due to disk caching that doesn't have to mean that the disk has to be a bottleneck (even disregarding the fact that /var/clipboard could be a tmpfs). This would need real life testing of course.
* Some latency is acceptable (ESR's estimate is 0.7 seconds - http://www.faqs.org/docs/artu/ch10s01.html). In this case we have additional room for manoeuvre, as we also have the time the user needs to switch from the copying to the pasting application.
* I will also point out that this proposal actually removes a potential source of latency (one that does occur in the wild) with the X11 selection protocol - when an application pastes X11 clipboard data it requires several rounds of communication between the two applications via the X server. If the application offering the data is currently busy the application pasting will often freeze until the data can be served. With the scheme I proposed the data will be available at once.

I realise that it might still not be workable despite all that, but I do think that there is a chance it might be.

Simple is good enough

mjthayer — Mon, 15 Nov 2010 09:51:48 +0000

> Exactly, this is more or less the approach I was thinking about (high level, not the details. I don't see the need for tar, for instance).

I have been turning this over in my head since I posted that and have refined it somewhat. Rather than creating a tar file it should be enough to create a standard file in a known mime format with a usable magic number (note that the opendesktop.org mime specification allows for adding custom mime formats complete with magic numbers to a system). This would of course mean that text data would also need to be preceded by a mime tag, which I recognise makes the system slightly uglier, but also more reliable. Conversions could be handled in the BeOS way, which is to register conversion filters with the system (shared objects or just executable filters - in the end they need to be loaded from disk one way or another, so it might not make much difference). An application pasting clipboard data of a mime type it couldn't handle could enumerate the available filters to see if one of them helped.

An X11 application using this clipboard should of course start a proxy application - preferably a well known singleton application - which would keep the clipboard in sync with the traditional X11 one. This would allow for painless transition (I say transition, but there is no reason this sort of compatibility shouldn't be maintained for ever). Perhaps (although unlikely) I will try to get this working some time. I already maintain an X11 clipboard proxy tool that could be adapted to do the job.

Rethinking remote display

alon — Sun, 14 Nov 2010 23:23:00 +0000

It is tied to qemu (not really to kvm, although we don't run it on anything else), but should be possible to make run without qemu - although it would require some changes to the design (right now it is built on the trio guest-host-client, if you remove the host the structure changes a little). bandwidth efficiency wise it should be the same.

Simple is good enough

quotemstr — Sun, 14 Nov 2010 21:00:11 +0000

"Why does copying this image freeze the program for 30 seconds? Lol that doesnt happen under Windows."

That's what users will begin saying if your scheme is implemented and we don't have application-to-application copying. What X11 jacobins like you don't realize is that current design choices were made *for a reason* and shouldn't be lightly abandoned.

Simple is good enough

i3839 — Sun, 14 Nov 2010 17:06:50 +0000

Exactly, this is more or less the approach I was thinking about (high level, not the details. I don't see the need for tar, for instance). Have a simple copy and paste system which can be easily used by all applications, including network or X proxies and non-graphical programs.

Simplicity is more important than efficiency. The alternative would be to have some format negotiation and directly copying between programs when needed (for non-text formats), like X has. This is not worth the complexity.

All that's needed is a way to store mutiple formats and retrieve them, as wel as serialisation and notifications when things get copied (for proxies).

You must be joking

bronson — Sun, 14 Nov 2010 06:10:35 +0000

In FF just middle-click the favicon to the left of the URL. Instead of taking 2 clicks, it takes a single click.

Seems a rather weak reason to hate a browser.

You must be joking

tnoo — Sun, 14 Nov 2010 04:57:58 +0000

... which is exactly why I hate using FF. Konqueror has a "delete" button to rub out the old URL.

LPC: Life after X

dlang — Fri, 12 Nov 2010 19:39:16 +0000

you are right, I am assuming that good network transparency (as opposed to what VNC etc provide) it is going to require some consideration in the design of the windowing system, and since the people working on the windowing system are taking the attitude 'nobody needs network transparency', such consideration is unlikely.

LPC: Life after X

bronson — Fri, 12 Nov 2010 19:14:21 +0000

Your reply implies that it's impossible to layer network transparency on top of Wayland. I doubt that's true.

LPC: Life after X

dlang — Fri, 12 Nov 2010 10:36:51 +0000

if all the new applications are written to run on X then wayland has no native apps and does no good.

if all the new applications are written to run on wayland, then they cannot be clients for X and the fact that there is still an X server you can run on top of wayland does no good (except for obsolete apps that pre-date wayland)

do you see why people who need network transparency may be opposed to the common development going in a direction that doesn't support it?

LPC: Life after X

mjthayer — Fri, 12 Nov 2010 09:04:40 +0000

>> what about [...] embedding a small web server

> http://www.transmissionbt.com/images/screenshots/Clutch-L...

That is still going over a socket though, or so it looks to me, not stdin and stdout forwarded by ssh.

LPC: Life after X

mjthayer — Fri, 12 Nov 2010 08:56:45 +0000

>> Given that the X server is still going to be around...

>That won't help you much if the application is no longer able to act as an X client.

Why shouldn't it be able to? If the X server is around people can write new X clients if it makes sense (although they will probably find other ways to do network forwarding when they start to think about it). If it uses Gtk+ for X it can even blend in seamlessly with the non-X clients.

LPC: Life after X

jch — Fri, 12 Nov 2010 06:04:17 +0000

> what about [...] embedding a small web server

http://www.transmissionbt.com/images/screenshots/Clutch-L...

LPC: Life after X

jch — Fri, 12 Nov 2010 06:02:22 +0000

> Given that the X server is still going to be around...

That won't help you much if the application is no longer able to act as an X client.

WHOM is right :-)

jmorris42 — Fri, 12 Nov 2010 00:40:52 +0000

> Proof: more then 50 million Android phones were sold till now.

Who cares. Seriously, counting Android as Linux users is as crazy as counting OS X users as UNIX, in both cases the *NIX underpinnings are only used as a hardware abstraction to host an alien environment to the typical Linux/GNU/X we typically call "Linux" as a shorthand. More power to em and all, but lets not make Apples to Oranges comparisions.

> "Traditional Linux users" who care about network transparency
> and other such things are fast becoming minority

Well take yer asses off and do your clone of all the bad ideas the existing userbase fled screaming from. It is Free Software after all. Here is a clue: Most of us came to *NIX because we saw the benefits and could also see the horrid mess the DOS/Windows world was. Why do we now want to toss one of the most wonderful ideas in computing to become more like the mass market drones? The network is the computer, the computer is the network. It isn't just a marketing slogan for some of us.

Besides, I never understood why suddenly everything needs to be rendered client side as bitmaps. Extend X to give it the mechanisms to support things it can't currently do and make sure they can run over the wire. Why can't X do the modern font tricks, compositing or whatever over the wire? Why isn't the default scalable SVG artwork? Isn't it normally sufficient to note that Apple does something enough to end an argument? Well they do Display Postscript (ok, they had to modernize the terminology to Display PDF) on OS X so obviously the idea is still mainstream, right?

Is it good that Linux is finally getting the last chunk of device support and taking over mode setting and general driving of the video hardware? Of course. Is it good that makes it easier for upstarts to experiment with new display systems? Of course! Might this someday lead to replacement for X? Perhaps. But any replacement has to be able to do the things X does if the target is a general computing environment instead of embedded. We have decades of existing apps and little desire to toss em all and start over. If we wanted to do that we could all just ditch the whole stack and go run Haiku or something.