Wayland support (and more) for Emacs

There were a number of good things to be found in the 27.1 release, which was a "huge jump" in functionality. Perhaps at the top of the list is support for the HarfBuzz library, which brings improved text-rendering support in multiple languages — and the support for color emoji that no self-respecting 2020s application can be without. Portable dumping was finally added, leading to faster startup and less system-dependent code. Emacs also now supports a tab-based interface, something that "had been asked-for forever", Walsh said.

After that release, though, "the navel-gazing set in" with lots of questions about how Emacs could be restored to its one-time popularity. This is something that happens every year, he said, but the discussion was perhaps deeper this time around. The problem with this kind of discussion in the Emacs community is that any resulting action can take years to come to fruition; there tends to be a lot of resistance to change from "old hands" (in whose company Walsh counted himself). But if people present changes in the form of code, they can eventually gain traction.

Some ideas that were discussed are not going to happen; these include replacing elisp with JavaScript or some other "more modern" language. Similarly, porting Emacs's C code to Rust is not something to hold one's breath for. Other desired changes are more "socially easy", but are hard technically; these include making elisp faster so that more of Emacs could be written in Lisp rather than C. Another is rewriting the Unix graphical interface, which is fragile; small changes can break obscure platforms in surprising ways.

Taking a step back, Walsh asked why people bother to use Emacs in the first place. In his own case, it's simply too late to change — and he has tried a couple of times. It is his toolbox at this point. In any case, whenever some other editor demonstrates an interesting feature, it gets ported to Emacs eventually. He also likes the fact that, unlike other editors, Emacs treats C code as C, rather than as some poor-developer's C++.

It is one thing to ask why a developer like Walsh would use Emacs, but it is another to ask why he would want Emacs to work with Wayland, and why he would want to work on that project. The answer was that Wayland works well on his box; he finds it to be "less clunky" than X at this point. Emacs was the only application he had that still used X; fixing that was his opportunity to get the X libraries off his system entirely.

The Wayland port

Emacs, he said, is not really a text editor; it's a Lisp environment that has been ported to Unix, which just happens to have a useful editor built into it. Without Lisp, he said, there is no Emacs. Everything that happens in Emacs goes through Lisp; for example, if one types the letter "x", Emacs will (usually) run:

    (self-insert-command "x")

to insert that letter into the current buffer. Everything can be reconfigured at almost all levels. Among other things, that means that Emacs has no concept of a "plugin" because none is needed; add-on features are equivalent to the core of Emacs. If one looks in the source repository, 60% of what's there is Lisp, while only 15% is C. Amusingly, 16% is changelog containing 35 years of development history. The C code includes the Lisp interpreter, the Athena toolkit, and the core "redisplay" code that manages the screen.

That redisplay code, he said, is C code that was written by Lisp developers while many of the core C standards were still being developed. It is characterized by "heavy macro use" that is trying to hide a lot of history. It is efficient code, but also complex; it's one of the trickiest parts of the entire code base. But this is where changes need to be made to bring Emacs to Wayland.

The problem is complicated by the fact that Emacs was never designed to be a graphical user interface; it has always been a terminal application. To work with the X window system, Emacs pretends that an X window is, in fact, a terminal. As is its usual practice, Emacs deals with X at the lowest level, processing as much X interaction through Lisp code as possible.

At some point, the X redisplay code was ported to the GTK toolkit, but that port is a sort of false front; windows are created with GTK, but then all events in that window are forwarded directly to the Lisp engine. Even with GTK3, Emacs is still using much of its old code, handling drawing and events at the X level. Later, Cairo support was added to help with printing; it has been "periodically maintained" since.

There has been interest in Wayland support for some time; after all, Emacs now uses GTK and Cairo, both of which support Wayland, so how hard can it be? A rudimentary port was done in 2014 using Cairo for drawing; it was missing a number of features but was a useful learning experience. That exercise did lead to the adoption of Cairo as the core rendering library in 2015, though Cairo did not become the default until the 27.1 release in January 2020. Nonetheless, it was one of the first signs that Emacs may be moving away from X.

In 2019, Stefan Monnier pointed out an in-progress Wayland port and asked if anybody had looked at it. That was, it turns out, a pure GTK port that migrates all low-level X code to its GTK3 equivalents. All drawing is done through Cairo and GTK, which should be wonderful for Emacs going forward, in that the maintenance burden for that code rests with the GTK developers.

It works by creating a new terminal type that uses GTK to get a window up on the screen and handles GTK events. There is a new Emacs widget that serves as the endpoint for those events and implements the Emacs display. In the future, this widget could conceivably be used in any program that wants to be able to put up an Emacs window. This widget can hold others, things like GTK scrollbars seem to work well within it, for example.

Getting all of this working has required dealing with a number of challenges, many of which are based in the fact that Emacs is both huge and old. The redisplay code is about 20,000 lines, full of #ifdefs and macros; it's hard to work with. There are a lot old workarounds in this code that may not be needed anymore, but it can be hard to tell.

The X code in Emacs "wants to know everything" and handle everything itself; this approach is wired deeply into Emacs as a whole. GTK, instead, wants to abstract things and deal with as much as possible itself. So, for example, Emacs wants to see and respond to raw X events, but that's not what GTK provides. Emacs is designed around having all events be handled through Lisp code, but that may not be a model that can be supported going forward.

In Wayland, all windows are independent, even when they are nested inside each other. Windows in X, instead, are interrelated and applications can use global coordinates. That leads to confusion in the Wayland switch because it's not always clear where the origin should be; that leads to annoying problems where popups appear in the wrong place. Walsh lost three weeks of his life figuring this out, he said.

The pure-GTK work, in the end, came down to about 100 commits touching 58 files and adding about 16,000 lines of code. Does it work? Yes, it does. It was developed on Arch Linux and Fedora, but has been tested on FreeBSD as well; he has been using it daily for the last year. He does not see a lot of differences between the X and Wayland ports, but prefers the Wayland version. It is "bug-compatible" with X Emacs, though there are a few rendering differences that one might notice. He thinks it renders more quickly, but had no measurements to prove that.

There are a few things that will not be supported in the Wayland version, though. Xsettings has been replaced with GSettings, for example. Wayland doesn't support self-positioning of windows — that is left to the compositor — so Emacs is unable to support options that fix the position of windows.

This code currently lives in a feature branch in the Emacs Savannah repository; he would like to see more people testing it. He also said that, if there are developers who could provide GTK help, that would be appreciated.

Looking forward

Now that there is a pure GTK3 port of Emacs, GTK4 has naturally been released. There is some experimental work going on to port Emacs to GTK4, he said, but there may not be a lot of benefits to be had from that port. GTK4 is an even further departure from X, making the port harder yet. There has also been interest in using features like OpenGL, Vulkan, and DirectX, but he advised patience; it took twelve years to get the Cairo port in, for example. Beyond that, text rendering is relatively hard on a GPU, so it's not clear what Emacs would gain from this work.

What else can be expected from Emacs in general? It probably will not break away from its "ugly look", he predicted. The GTK port was not meant to change the editor's appearance. The default key bindings will stay the same; there are too many old hands who would resist changing them. Those old hands are the core Emacs user base, so there is no desire to make them unhappy. Elisp will not be replaced anytime soon either — except, maybe, with a different flavor of Lisp. Whenever that conversation comes up, though, people start arguing about which flavor, with lots of bikeshedding going on.

What is changing can be seen in the ongoing push toward common libraries rather than doing everything by hand within Emacs itself. The adoption of libraries like HarfBuzz, Cairo, and Jansson are examples; he thinks this trend could continue.

A big thing to look forward to is native compilation of Lisp code; that will have "a huge ripple effect" in the Emacs community. Andrea Corallo first asked about this capability in 2019, and delivered a prototype implementation one month later. It uses libgccjit as the back-end for the byte compiler and supports a wide range of platforms. That is important: by using GNU tools for the compilation, it meets the "GNU political requirements" and can actually be included in Emacs. This work will allow writing more of Emacs in Lisp, migrating some code back from C. That will allow for more customization; in the future, customized Emacs "distributions" may take on a bigger role as a result.

He finished by crediting Yuuki Harano as the main developer behind the pure GTK work in Emacs.

Your editor asked afterward about how interested users could try out this port. One way is to check out the branch from Savannah and build it, but there are also a number of repositories out there with builds for various distributions now. Arch Linux packages it as part of the distribution "because it's Arch". This article was written using a build from the Fedora Copr repository. The xeyes test shows that it is indeed running natively on Wayland, and it seems to work well, with the one strange exception that mouse-wheel scrolling happens at half speed.