The Emacs dumper dispute

As covered here in January, the Emacs dumping (and "undumping") mechanism has long depended on some low-level hooks in the GNU C Library's memory allocation subsystem. The Glibc developers would like to modernize and improve this code, improving the library overall but removing the hooks that Emacs depends upon. At the end of the January discussions, the Emacs developers had decided to move to a workaround implementation that allowed the dumper to continue to work in the absence of Glibc support.

"Unstable" is the sort of behavior that users of text editors normally go well out of their way to avoid; it's also the sort of thing that could give vi a definitive advantage in the interminable editor wars. So something clearly needs to be done to make the Emacs dumping facility more stable and, preferably, more maintainable going forward. What that "something" would be is unclear, and the posting of a possible solution appears to have simply muddied the waters further.

That solution comes in the form of the "portable dumper" patch from Daniel Colascione. This patch is not small; it adds over 4,500 lines of code to Emacs and it is not yet complete. Rather than try to capture the state of the C library's memory-allocation subsystem, it simply marshals and saves the set of Elisp objects known to the editor. The file format is designed for performance and, in some settings at least, Emacs can start by simply mapping the file into memory and initializing a set of pointers.

Colascione describes the result this way:

It also, he says, matches the startup performance of the current "unexec" system to within 100ms, and he has not yet had the time to collect a bunch of low-hanging optimization fruit. In other words, it seems like an interesting solution to the problem, but a patch of this size is always going to generate some discussion.

Some of that discussion focused on how this dumper works when address-space layout randomization (ASLR) is in use. Current Emacs binaries must disable ASLR entirely, thus losing the security benefits that ASLR is meant to provide. The new dumper does not require disabling ASLR, but it does contain an optimization that can be applied if the dump file can be successfully mapped at a specific address: most of the data therein can be used directly from the mapped image, without the need to allocate storage for and copy it. That should speed the startup process considerably, at the cost of always mapping the dump image at the same location.

Paul Eggert worried about the potential security implications of losing ASLR protection for the bulk of the editor's data. Colascione responded that, since no part of the data image is marked executable, there is little risk of attackers running code from there. But, as Eggert pointed out, that view overlooks an important detail: that memory is full of Elisp bytecode that is executed in the editor itself, and which can do just about anything an attacker might want. So, if this approach is adopted, the fixed-location mapping might have to be turned off, at least by default.

There is, however, a bigger disagreement involving Zaretskii, who described this work as "a wrong direction." His objection, in short, is that this patch adds a lot of low-level complexity, implemented in C, that will be a maintenance burden going forward. That is, he said, a threat to the future of the project:

It makes sense to put thought into the maintainability of the code base and how it can be evolved to attract more developers. It is not entirely clear, though, that C programmers are actually a dying breed — or that the long-term supply of Elisp developers is more certain. In any case, the Emacs community needs to fix the startup problem; those who oppose the portable dumper solution presumably have something else in mind.

Zaretskii's preferred solution would be to make the Elisp loader faster, to the point that it can be used to read Elisp code directly at startup time. That is a solution that others might like to see as well, but it has one significant shortcoming: no code toward that goal exists, and there are no signs that anybody is working in that area. Colascione's solution, instead, does exist and has an interested developer behind it. In almost any development project, working code and ongoing maintenance carries a lot of weight.

Zaretskii feels strongly enough about this issue that he has threatened to resign as co-maintainer if the portable dumper is adopted. He appears to be nearly alone in this stance, though. Colascione has said repeatedly that he sees no other way to get the required performance. Richard Stallman is guardedly favorable to this solution, noting that it will be far easier to maintain than the current unexec code. John Wiegley, the other Emacs co-maintainer, also favors going with the portable dumper code.

The wind thus appears to be blowing in the direction of adopting the portable dumper patch. Nobody seems to want to see Zaretskii relinquish the co-maintainer role (a role he only accepted last July), so, if the portable dumper is merged, the community can only hope that he will change his mind. Any large development project will occasionally make decisions that are opposed by some of its developers, even when those developers are maintainers. But the venerable Emacs editor will still be there, and will still have no end of other problems to solve.