The future of Emacs, Guile, and Emacs Lisp

GNU Emacs is one of the longest continuously developed applications in the free-software world; at just over 30 years old it just qualifies, by some definitions, as a multi-generational project. But such longevity brings its own challenges. Many in the Emacs community have concluded that it may soon be time to replace the editor's internal Lisp interpreter with a faster and more modern alternative. But replacing the underlying virtual machine on which so much of Emacs runs has far-reaching implications—including the impact it would have on Emacs's own flavor of Lisp.

Although the subject has come up before, it was raised most recently on September 11, when Chris Webber wrote to the Emacs development list asking about the status of Robin Templeton's work on Guile-Emacs. Like the name implies, Guile-Emacs replaces the internal Emacs Lisp engine with the interpreter from GNU Guile. The Guile interpreter was originally written to support the Scheme language (itself a dialect of Lisp), but today it supports multiple other languages as well, including Emacs Lisp.

Guile's engine is reported to be faster than Emacs's internal Lisp interpreter, but it also offers several other valuable features, like concurrency and the prospect of supporting Emacs extensions that are written in other Guile-supported languages. Templeton, who has been working on Guile-Emacs for the past five years in a series of Google Summer of Code projects, listed quite a few other benefits to rebasing Emacs on the Guile engine, including "a full numeric tower, structure types, CLOS-based object orientation, a foreign function interface, delimited continuations, a module system, hygienic macros, multiple values, and threads." As of now, Templeton reports that the vast majority of modules in GNU Emacs, as well as a significant set of popular external extensions, run reliably on Guile-Emacs.

When it comes to formally grafting Emacs onto the Guile interpreter, however, there are much stricter requirements to consider: users expect all of their Emacs extensions—including both third-party extensions and their own personal code—to continue to run without incident. That is a tall order indeed. And, as Eli Zaretskii pointed out, Guile is not as stable on systems lacking GNU utilities. Furthermore, while Guile is the official extension language for GNU projects, at the moment it is a considerably smaller project than Emacs itself. Suddenly imposing the larger project's needs on the Guile developers (not to mention imposing the bug reports of the entire Emacs community) might place a considerable strain on Guile project resources. David Kastrup, among others, worried whether the Guile team would be able to handle the load. Suddenly gaining a large set of new users could be good for the Guile project, of course, and attract new contributors, but that outcome is not guaranteed.

Moreover, there is the question of how tightly Guile and Emacs should be coupled. For one thing, the two projects currently use different internal string representations, which means that text must be decoded and encoded every time it passes in or out of the Guile interpreter. That inefficiency is certainly not ideal, but as Kastrup noted, attempting to unify the string representations is risky. Since Emacs is primarily a text editor, historically it has been forgiving about incorrectly encoded characters, in the interest of letting users get work done—it will happily convert invalid sequences into raw bytes for display purposes, then write them back as-is when saving a file.

But Guile has other use cases to worry about, such as executing programs which ought to raise an error when an invalid character sequence is encountered. Guile developer Mark H. Weaver cited passing strings into an SQL query as an example situation in which preserving "raw byte" code points could be exploited detrimentally. Weaver also expressed a desire to change Guile's internal string representation to UTF-8, as Emacs uses, but listed several unresolved sticking points that he said warranted further thought before proceeding.

Finding a common string representation might be doable, then, but it is not the only hurdle to integrating Emacs and Guile. Some, like Stefan Monnier, floated the possibility that Emacs Lisp should be intentionally "evolved" into another, more standardized Lisp dialect. Such an evolution would not be easy in any case, but especially not toward Guile's native language Scheme. It might be more feasible to adapt Emacs Lisp into Common Lisp, but it would still be far from trivial. While there are many areas of compatibility between the two languages, even a few incompatibilities can result in a world of pain for developers. Templeton, for example, noted that Common Lisp has no feature that corresponds to Emacs's buffer-local variables, which are often used in Emacs extensions. But Monnier pointed out an even thornier problem—while Emacs Lisp regards a boolean "false" and an empty list as being equal, other Lisp dialects do not:

Redefining such a fundamental language construct would have far-reaching ramifications, of course—for the decades' worth of existing code and for the community of Emacs developers. But even if Guile's Emacs Lisp interpreter were solid, Monnier added, its Scheme underpinnings would likely appear in inconvenient places. "I suspect some errors signals coming from Guile's runtime will end up using Scheme-style data and will end up spilling into the Elisp side if we're not extra careful."

A related question is whether or not Emacs Lisp should remain the sole language for developing Emacs. As long as Guile supports other languages, including Scheme as well as entirely unrelated options like JavaScript, there is a case to be made that a Guile-based Emacs should support a true foreign function interface and allow users to write Emacs extensions in their language of choice. Opinion on that topic seemed to be sharply divided. Richard Stallman expressed support for the idea of supporting Scheme extensions in Emacs, while several others (including Phillip Lord) are open to unrelated languages like JavaScript as well. Others, such as Monnier, doubted whether real integration with other languages would be possible. Some, like Daniel Colascione, argued that attempting to support outside languages popular at any given moment would be a waste of effort.

In the short-and-medium term, adding support for Emacs extensions written in languages other than Emacs Lisp does not seem to have much traction. There are too many more pressing issues to consider, especially if the project decides to pursue rebasing Emacs on the Guile interpreter. Alternative Lisp interpreters have been discussed, such as Kristian Nygaard Jensen's suggestion of Embeddable Common-Lisp. At best, though, the other alternatives come with the same integration challenges as Guile; there are simply not all that many GPL3-compatible, actively developed Lisp interpreters to choose from. Guile, at least, is an official GNU project, and several members of its development team have expressed support for the possibility of working with Emacs.

The way things stand now, there has not been a formal consensus as to the eventual replacement of Emacs's internal Lisp interpreter with Guile. Templeton's Guile-Emacs work has amassed quite a few admirers in the Emacs development community, but it will likely need further testing, followed by some firm decisions on the part of key Guile and Emacs maintainers about exactly what integration would look like. Given that those conversations (such as the string-handling discussion) are happening now, there is reason to be hopeful—but expecting any sort of timeline or roadmap remains premature.

Note: you can avoid this step in the future by logging into your LWN account.