|
|
Subscribe / Log in / New account

Development

Removing support for Emacs unexec from Glibc

By Jake Edge
January 27, 2016

The Emacs editor requires a lot of Lisp code and program state before it can start doing its job. That led Emacs developers to add the "unexec" feature to quickly load all of that at startup, but unexec has always been something of a hack. It employs a fairly ugly (and intrusive) mechanism to do its job. Some non-standard extensions to the GNU C library (Glibc) are required, so a plan to eventually eliminate those extensions was met with some dismay in the Emacs community.

As part of the Emacs build process, a simpler version of the editor, called "temacs", is built. That program consists of all of the C files in Emacs, which comprise the Emacs Lisp interpreter and not much else. It is then run to load the standard Lisp startup files and to dump a copy of the running program. That dump is then used as the binary that users invoke when they want to run Emacs.

The mechanism used is in the Emacs unexec() function that converts the running program into a new executable. To do that, it needs to handle memory that was allocated by malloc(), which also requires that the Glibc internal tracking and housekeeping data structures for memory allocation be preserved. The dumping (and restoring) mechanism uses malloc_get_state() and malloc_set_state() to do that, which are the extensions that Glibc developers would like to eliminate.

In mid-January, Florian Weimer posted a heads-up message about the change to the emacs-devel mailing list. He said that it was likely coming this year and was being done to allow changes to the "heap layout between glibc releases, for standards conformance, performance and security improvements". The intent is that existing Emacs binaries will still continue to work, but that at some point the Emacs build would have to change, he continued. He also noted that supporting existing binaries "causes a significant ongoing maintenance overhead for glibc upstream, basically maintaining a separate malloc implementation indefinitely".

Emacs maintainer John Wiegley voiced his concerns that the alterations needed to Emacs would be "a rather significant change to low-level code that has been functioning for a very long time". He suggested extending the timeline for the change and discussing ways to provide the same or similar functionality. Weimer, though, believes that it is "time to tackle it at the root" by fixing Emacs, which is seen as the only user of the interfaces:

If you cannot make the changes yourself, we'll have to make sure that you have a replacement available when the need arises. Even if we have to do the development, it will pay off fairly soon on the toolchain side.

A change "forced" on Emacs by another GNU project was always likely to get the attention of Richard Stallman. He emailed the private—and largely unused—mailing list for the Glibc steering committee (glibc-sc), which Mark Brown reposted to the normal Glibc mailing list (libc-alpha). The steering committee does not really exist in the form of that list anymore, which led Carlos O'Donell to suggest that the mailing list be closed. In the reposted message, Stallman largely echoed Wiegley:

When you consider making a change that will break other GNU packages in a way that is hard to fix, you must not decide unilaterally. All the more so, when the package is as important as Emacs. We need package maintainers to cooperate.

Please have a discussion with the Emacs developers and decide together which course of action is best for the GNU system, and what time scale for that action fits the release schedules best.

Stallman also mentioned that he had contacted the Glibc maintainers in the emacs-devel thread. That led Weimer to wonder why it was appropriate to move the discussion to a private list: "This doesn't really match how glibc development proceeds today." Stallman, though, thought that it would be better discussed in private: "This a sensitive issue; it is best to discuss it without an audience."

That particular cat was out of the bag at that point, however. The conversation in emacs-devel proceeded by looking at the dump/load functionality, whether it is still needed, and ways to implement it that don't require intimate knowledge of the internals of Glibc's memory-allocation techniques.

Ali Bahrami, who works on the Solaris linker, wondered if it even made sense to continue to support the unexec functionality. Computers have gotten a lot faster since that optimization was added, so it might make sense to simply leave it behind:

Before you fight to to save unexec, I'd encourage you to measure the impact, and see if it still matters. If it does, then it would be worthwhile to consider other means for getting those bytes into memory quickly that don't involve second guessing object layout, memory allocation, and process layout. Speaking as a linker guy, linking is only going to get more dynamic, and more complex, going forward. You might be glad, down the road, to be out of that game.

So Bahrami and others set out to measure the difference between starting Emacs and starting temacs (which must load all of the different startup files). It became clear that there is still a substantial difference; a half second versus more than five seconds, though that depends on various factors. Different tricks were tried, with some success, but the main problem remained.

David Caldwell suggested one possible approach: taking the compiled Lisp files (.elc files) that temacs loads and adding them into the binary.

What if one were to smash all the .elc files that temacs loads as part of the undump process and then put them into the temacs object, maybe by relinking them in as a giant array (portable), or by shoving them into their own section with objcopy (not really portable)?

Stallman seemed interested in that idea, "but the real test is in implementing it".

According to Paul Eggert, making unexec more portable has been on the to-do list for a while, "and this will light more of a fire under it". Concerns that Emacs might not build using a new Glibc API (which has not even been written yet) that came up earlier in the thread are not a problem, he said. "Emacs should still build and run even if the glibc API is changed, as Emacs ./configure probes for the glibc malloc-related API and falls back on its own malloc implementation otherwise."

In another message, Eggert outlined how Emacs uses unexec and how it might be able to get along without it: "Emacs could live without the current unexec in a semi-portable way by doing what XEmacs does, which is to write out data and mmap it in later". He does not know the details of the XEmacs approach, though, and suggested other possibilities as well.

The discussion continued, looking at various ways for Emacs to accomplish its goals without requiring Glibc to maintain the same heap layout forever. If there was a need to consider the matter privately, it certainly wasn't apparent in the thread. As with many changes of this sort, developers on both sides of the change simply worked things out. Perhaps there will be glitches down the road, but there was plenty of notice and it seems like a direction that the Emacs developers wanted to go in anyway, so it is hard to see any major potholes in the road ahead.

Comments (41 posted)

Brief items

Quotes of the week

I live in Mountain View and the Google Car passes me regularly when I'm rollerblading or cycling, so I have a lot of time to think about whether I want to pull out in front of the Google Car to get around another cyclist. So far I've decided not to.
— Alison Chaiken at SCALE 14x, on the perils of autonomous vehicles.

Unikernels are entirely undebuggable. There are no processes, so of course there is no ps, no htop, no strace — but there is also no netstat, no tcpdump, no ping! And these are just the crude, decades-old tools. There is certainly nothing modern like DTrace or MDB. From a debugging perspective, to say this is primitive understates it: this isn't paleolithic — it is precambrian.
— Bryan Cantrill, on the perils on unikernels.

Another way to put it: a team full of hammers will go around looking for nails. A team that’s a whole toolbox might figure out what really needs doing.
— Havoc Pennington

DDSlider is from 2003 and not indicative of my current indenting. I look forward to critiquing your early work thirty years in the future.
— Nic Collins (hat tip to Hanno Zulla).

Comments (31 posted)

Rust 1.6 released

Version 1.6 of the Rust programming language has been released. "The largest new feature in 1.6 is that libcore is now stable! Rust’s standard library is two-tiered: there’s a small core library, libcore, and the full standard library, libstd, that builds on top of it. libcore is completely platform agnostic, and requires only a handful of external symbols to be defined. Rust’s libstd builds on top of libcore, adding support for memory allocation, I/O, and concurrency. Applications using Rust in the embedded space, as well as those writing operating systems, often eschew libstd, using only libcore. libcore being stabilized is a major step towards being able to write the lowest levels of software using stable Rust."

Comments (5 posted)

PulseAudio 8.0 released

Version 8.0 of the PulseAudio framework is available. New features include systemd journal logging for PulseAudio clients, more flexible configuration-file handling, an interface for balancing low-frequency effects (LFE) channels, and improvements to the D-Bus interface.

Full Story (comments: none)

A change of maintainership for Mercurial

Matt Mackall, the creator of the Mercurial source-code management system, has announced that he is ready to move on to a new project. "So over the course of this year, I'm going to gradually remove myself from daily involvement in the project. As lots of people and companies have a lot invested in Mercurial, I'm doing this over a long period of time to make sure it goes smoothly."

Comments (37 posted)

The Linux Test Project has been released for January 2016

The Linux Test Project test suite stable release for January 2016 is available. There were 191 patches by 29 authors merged since the previous release. Some notable changes include rewritten and new cgroup tests for cpuacct and pids controllers, rewritten basic cgroup functional and stress tests, new userns07 test for user namespaces, new syscall tests, and more.

Full Story (comments: none)

Firefox 44 released

Firefox 44.0 has been released. With this version Firefox can get push notifications from your favorite sites. This release also features improved warning pages for certificate errors and untrusted connections, H.264 is enabled if the system decoder is available, if MP4/H.264 are not supported WebM/VP9 video support is enabled, the brotli compression format via HTTPS content-encoding is supported, and more. See the release notes for details.

Comments (48 posted)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

Hutterer: Is Wayland ready yet?

On his blog, Peter Hutterer answers the perennial "is Wayland ready yet?" question by pointing out that it really is not the right question. "The protocol is stable and has been for a while. But not every compositor and/or toolkit/application speak Wayland yet, so it may not be sufficient for your use-case. So rather than asking 'Is Wayland ready yet', you should be asking: 'Can I run GNOME/KDE/Enlightenment/etc. under Wayland?' That is the right question to ask, and the answer is generally 'It depends what you expect to work flawlessly.' This also means 'people working on Wayland' is often better stated as 'people working on Wayland support in ....'. "

Comments (213 posted)

AMD: It's time to open up the GPU

AMD has launched "gpuopen.com" to support open graphics development (on AMD GPUs, naturally). "The second is a commitment to open source software. The game and graphics development community is an active hub of enthusiastic individuals who believe in the value of sharing knowledge. Full and flexible access to the source of tools, libraries and effects is a key pillar of the GPUOpen philosophy. Only through open source access are developers able to modify, optimize, fix, port and learn from software. The goal? Encouraging innovation and the development of amazing graphics techniques and optimizations in PC games."

Comments (12 posted)

Page editor: Nathan Willis
Next page: Announcements>>


Copyright © 2016, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds