LWN: Comments on "Removing support for Emacs unexec from Glibc"

Removing support for Emacs unexec from Glibc

nix — Wed, 15 Sep 2021 12:33:24 +0000

Well, yes. Isn't this true for all Emacs users? :)

(actually, because a lot of my work happens in containers and VMs and the like, some of the more useful features like language server support are things I just don't use because the environment in which stuff is compiled is not the same as the environment in which my Emacs is running. So I spend a lot of time in terminal emulators too. I should fix that!)

Removing support for Emacs unexec from Glibc

sdalley — Sat, 11 Sep 2021 17:05:21 +0000

That's... wow. Kinda feels like being shifted into an alternate universe! Emacs is clearly your operating system, and the Linux kernel a minor appendage. More of a boot loader, really...

Removing support for Emacs unexec from Glibc

mirabilos — Fri, 10 Sep 2021 14:42:12 +0000

This would totally defeat stack carnaries, ASLR, etc. though. Even PRNG state.

Don’t.

Nuke this thing.

Removing support for Emacs unexec from Glibc

mirabilos — Fri, 10 Sep 2021 14:41:15 +0000

Good riddance. This has never been functioning well (except MAYBE on glibc) and been a hindrance in porting.

Removing support for Emacs unexec from Glibc

bjartur — Mon, 05 Dec 2016 11:18:12 +0000

The only checkpointing implementation I knew of is CRIU. Today I found a tiny draft of a reimplementation, CryoPID. When I first heard about CRIU I was surprised that the concept was feasible enough to be seriously pursued. Did you write an article about your 15yo project? It would make an interesting read, if you can find it on your computer or on the Internet Archive.

Removing support for Emacs unexec from Glibc

nix — Tue, 01 Mar 2016 19:45:21 +0000

One of them is running Gnus with a bunch of very large newsgroups that are never expired, plus all the non-mailing-list email I have received since 1996 and a bunch of massively backlogged mailing lists I must catch up on all open at once; the other is running CEDET, which has a habit of pulling source into memory to parse it and then not closing the buffers again, so right now it's got what feels like half the Linux kernel and glibc in a couple of thousand open buffers. I clean them up every few months.

Both are also running several hundred packages off ELPA via John Wiegley's use-package, including the awesome but... distinctly large Icicles.

So, yeah... I got a server with 24GiB RAM back in 2009 to run various crucial virtual machines and to run Emacs. It turns out that quite a lot of that memory goes to Emacs. :)

(And it's still fast! GC time is imperceptible.)

Removing support for Emacs unexec from Glibc

nye — Wed, 17 Feb 2016 17:34:28 +0000

>The two copies of emacs I have running right now (combined RSS, getting on for 3GiB)

What the actual fuck?

Removing support for Emacs unexec from Glibc

nye — Wed, 17 Feb 2016 17:32:17 +0000

>Almost all of the load time increase in an undumped Emacs comes from the way string allocation works in this mode: It performs a linear search over the heap to determine if the requested string has been allocated before and there is an allocation which can be reused (so it does string interning, like Lua, but without a hash table)

That's... disgusting. I remember doing something similarly nasty in, like, my second C program ever, and being pretty ashamed about it.

Removing support for Emacs unexec from Glibc

fw — Tue, 16 Feb 2016 20:34:47 +0000

Minor correction: Almost all of the load time increase in an undumped Emacs comes from the way string allocation works in this mode: It performs a linear search over the heap to determine if the requested string has been allocated before and there is an allocation which can be reused (so it does string interning, like Lua, but without a hash table). Once you disable that, load time goes down to a more bearable 0.4 second (on my nearly four-year-old laptop).

I mentioned this in the emacs-devel thread, but it was pretty out of control at that point.

Removing support for Emacs unexec from Glibc

nix — Fri, 05 Feb 2016 23:28:19 +0000

So yes, Emacs contains not just its own memory allocator but two or depending on how you define it even three memory allocators

Sorry, four: I forgot src/sheap.c. (Though that's just a brk().)

Removing support for Emacs unexec from Glibc

nix — Fri, 05 Feb 2016 23:23:32 +0000

That would be pointless complexity. Having read the thread, in particular <https://lists.gnu.org/archive/html/emacs-devel/2016-01/ms...>, Emacs can just load all the Lisp it needs at startup: no dumper is needed. It'll slow down Emacs startup time by a whole half a second, and in exchange Emacs gets to not have to maintain its own malloc or horrible unlinker any more, and Sun can ditch the massive pile of overengineering that is dldump().

Removing support for Emacs unexec from Glibc

meyert — Fri, 05 Feb 2016 10:12:48 +0000

if there were an option on lwn.net I would "like" your comment!

Removing support for Emacs unexec from Glibc

nix — Thu, 04 Feb 2016 23:31:25 +0000

XEmacs's 'portable undumper' did something fairly similar, serializing its heap into a lump plus relocation information, then mmap()ing it and relocating it on load. This did, of course, mean modifying it, so the text was no longer shared between implementations -- but with modern memory sizes this is about as insignificant a consideration as can be imagined. Oh no, Emacs's memory usage has risen by 5MiB! The two copies of emacs I have running right now (combined RSS, getting on for 3GiB) will truly buckle under the strain.

Removing support for Emacs unexec from Glibc

nix — Thu, 04 Feb 2016 23:28:40 +0000

Because Emacs runs on a lot of platforms and used to run on a lot more. Many of these platforms had terrible malloc()s, and Emacs calls malloc() a *lot*, with a really annoying allocation pattern that intersperses lots of tiny allocations, some very long-lived, with huge allocations (for buffers) with similarly divergent lifetimes, it really is a malloc() torture tester. Given that some platforms had malloc()s that were so bad that they used to provoke Emacs crashes...

(It is possible to use a separate allocator just for buffers, either direct mmap() or a 'relocating allocator' -- that one's notably useful under DOS. Yes, Emacs still supports DOS, specifically in conjunction with DJGPP.)

So yes, Emacs contains not just its own memory allocator but two or depending on how you define it even three memory allocators, and of course at one point had two garbage collector implementations as well (now, it only has one: GCPRO is finally dead.)

Removing support for Emacs unexec from Glibc

njs — Thu, 04 Feb 2016 10:47:52 +0000

> preloading it at boot time and keeping it in memory

this is actually a very-well supported configuration. I recommend it highly. (Well, not preloading at boot time, but loading at first use and then keeping that process around to make future loads fast.) All you have to do is

alias emacs="emacsclient --alternate-editor='' -c"

and you're good to go.

Removing support for Emacs unexec from Glibc

mathstuf — Thu, 04 Feb 2016 00:09:52 +0000

CPython has the concept of "frozen" modules where modules are byte-compiled and then put into a string -> length-byte array and then that is used as a lookup for modules. Don't see why that would be impossible with elisp.

Removing support for Emacs unexec from Glibc

massimiliano — Wed, 03 Feb 2016 06:37:20 +0000

Just to provide some info on how other VMs are handling this: the V8 Javascript VM can prepare (and dump to file) a heap snapshot, and the trick is that the heap contains objects and *functions* (JITted functions, with their machine code).

The VM can load such a snapshot at startup, and start from there. Essentially it is like if an initial preamble of Javascript code had just been executed.

What Chrome does is to prepare a snapshot at build time, containing the pre-jitted Javascript standard library.
I bet also nodejs does the same for its own set of libraries, and recently the interface for doing so has been cleaned up and made easily usable for every embedder.

Maybe the elisp VM could implement a similar feature?
Probably the trick is that in V8 the GC can relocate objects, so pointer values can be rewritten at will. Maybe this is something that elisp cannot do?

Removing support for Emacs unexec from Glibc

flussence — Mon, 01 Feb 2016 23:17:59 +0000

> One level removed from saying "The kernel has special code to allow vi to clear the screen without leaving framebuffer glyph artifacts."

Wouldn't that be the fabled “TTY layer” I've heard horror stories about? I've heard it eats developers...

Removing support for Emacs unexec from Glibc

smcv — Mon, 01 Feb 2016 20:31:17 +0000

Perhaps this was a use of Acronym Letter Sequence Randomization?

What about making it generic?

ksandstr — Sun, 31 Jan 2016 11:00:25 +0000

A couple of observations. The background (IIUC) is that Emacs facilitates a quick-start feature by dumping the malloc heap using an opaque state pointer, which is then restored by recreating the process image shape with the kernel, reloading the image, and putting malloc back in the proper state by passing the state structure back in. The problems are that first, the interface is nigh-inexcusably ugly, and second, it's effectively private between Emacs and glibc ensuring that no other users will appear[0].

It appears that Emacs' function should, at its simplest, require just two pieces of information: a cookie of some kind (root pointer?) that the allocator can use to recover its state, and a list of mmap() calls that would leave the allocator able to remove mappings and/or shrink the heap as though it hadn't been restarted at all.

Is there a significant reason why this couldn't be given a stable interface? I could see this being applicable to other language runtimes as well.

[0] besides those that don't mind being subject to a lock-step update cycle whenever a gnu farts within 8000 miles

Removing support for Emacs unexec from Glibc

kevinm — Sun, 31 Jan 2016 05:04:45 +0000

I wonder if Emacs could use CRIU ( http://criu.org/Main_Page ) instead?

Removing support for Emacs unexec from Glibc

giraffedata — Sun, 31 Jan 2016 04:43:05 +0000

it seems shocking that Glibc has code that exists primarily to support building Emacs.

Running Emacs, not building it. Every time someone invokes Emacs, it is fast because of its call to glibc's malloc_set_state().

And I'm sure it wasn't put there just for Emacs; the expectation was that other programs could exploit it the same way. It's generic enough, and the problem (programs taking a long time to start up because they have to load a bunch of stuff) common enough, that that would have been reasonable.

So I think the dog wagged the tail.

Extremely weird

tabefactus — Sun, 31 Jan 2016 03:09:39 +0000

It sounds like emacs already does this on other platforms, as this line from from Paul Eggert in the article says:

"Emacs should still build and run even if the glibc API is changed, as Emacs ./configure probes for the glibc malloc-related API and falls back on its own malloc implementation otherwise."

So emacs should be fine without any changes, as long as their API detection and malloc fallback work correctly.

Removing support for Emacs unexec from Glibc

aggelos — Sat, 30 Jan 2016 18:16:38 +0000

In the face of pervasive information leak vulnerabilities, ASLR doesn't seem to help much anyway, so...

Removing support for Emacs unexec from Glibc

scottt — Sat, 30 Jan 2016 17:00:34 +0000

You meant to say ASLR (Address Space Layout Randomization) instead of ALSR.

Extremely weird

ncm — Sat, 30 Jan 2016 12:46:09 +0000

If emacs needs a malloc layout, why hasn't it long since switched to its own malloc in place of libc's? Memory allocation is not a black art. Indeed, there are many separate projects providing alternatives to libc malloc for programs to use alongside glibc's.

Removing support for Emacs unexec from Glibc

meyert — Fri, 29 Jan 2016 23:13:22 +0000

1.) WAT?
2.) "on its own malloc implementation otherwise." - WAT? WHY?

Removing support for Emacs unexec from Glibc

andresfreund — Fri, 29 Jan 2016 22:00:19 +0000

> Android's Java VM seems to take a less scary approach: it creates a zygote process which loads the VM and a bunch of standard libraries and some data resources, and then every time you launch a new app it will fork the zygote and load the app's code into the new process. That means the VM initialisation only needs to happen once per boot, and any memory that isn't modified after initialisation will be shared between all the apps, and it doesn't rely on any special OS/library support beyond what's already needed for forking. And it can also preload complicated things like OpenGL drivers, since file descriptors that tie the userspace driver to the kernel driver will get cloned automatically.

On the other hand, it essentially makes things like ALSR useless...

Removing support for Emacs unexec from Glibc

excors — Fri, 29 Jan 2016 21:27:15 +0000

I remember excitedly discovering the dump() function in Perl, which was meant to dump the program state (e.g. after all the expensive module loading and initialisation) so you could run an undump program to convert it to an executable and repeatedly resume it later. I thought it'd be great for speeding up all my CGI scripts. (This was some years ago.)

Sadly I never found an undump program for Linux. But at least there's an undump for HP-UX from 1994, so at least some people had it working once.

Android's Java VM seems to take a less scary approach: it creates a zygote process which loads the VM and a bunch of standard libraries and some data resources, and then every time you launch a new app it will fork the zygote and load the app's code into the new process. That means the VM initialisation only needs to happen once per boot, and any memory that isn't modified after initialisation will be shared between all the apps, and it doesn't rely on any special OS/library support beyond what's already needed for forking. And it can also preload complicated things like OpenGL drivers, since file descriptors that tie the userspace driver to the kernel driver will get cloned automatically.

Maybe that's only a sensible tradeoff since Android can be certain it's going to be launching lots of Java apps - most people probably don't launch Emacs frequently enough to justify preloading it at boot time and keeping it in memory. But perhaps it could be an option for those who really want it.

Removing support for Emacs unexec from Glibc

jtaylor — Fri, 29 Jan 2016 17:32:03 +0000

one thing that could be fixed when emacs removes its use of the malloc state is the wrong alignment glibc provides on powerpc:
https://sourceware.org/bugzilla/show_bug.cgi?id=6527

Removing support for Emacs unexec from Glibc

kamil — Fri, 29 Jan 2016 15:51:25 +0000

> I don't quite understand how one can implement unexec() and still being able to relink with a new version after exec(). A new version of a library may have different struct layout. As I see this is the main complain of glibc developers, but the same applies to any other library. Developers are not allowed to change layout of structs that are supposed to survive unexec().

I'm guessing Emacs is careful to only preserve the malloc() structures, nothing else. But yeah, it must be a hassle, no doubt about it.

> In my optinion unexec() is quite a general method and probably it can be used by other software too. Probably even ability to suspend to ELF/resume from ELF for arbitrary processes would be useful. How to deal with opened file descriptors is an open question though.

You are describing the functionality of a checkpoint/restart system. I've worked on one in the past as a research project, implemented in user space, in the dynamic loader, that worked exactly that way (generating a new ELF binary on checkpoint). I wanted to put a link here but that was 15 years ago and the pages seem to be gone :-(. Tracking open files was indeed a hassle, and dealing with anything more complicated (pipes, sockets, shared memory, GUIs) was basically out of the question.

Removing support for Emacs unexec from Glibc

sorokin — Fri, 29 Jan 2016 09:56:19 +0000

> I'm guessing it could work if, as you say, all the shared objects were dumped as well -- importantly, including glibc).

Yes, exactly. That was how I would implement it. I would stress that all shared objects and stacks should be loaded to exactly the same place they were before unexec(), because objects in heap could point to functions in shared objects and to other objects in a stack.

> That would, however, be akin to static linking, which is a practice frowned upon these days. Glibc doesn't even do proper static linking anymore as far as I know, still requiring some shared objects (NSS) to be loaded dynamically.

I don't quite understand how one can implement unexec() and still being able to relink with a new version after exec(). A new version of a library may have different struct layout. As I see this is the main complain of glibc developers, but the same applies to any other library. Developers are not allowed to change layout of structs that are supposed to survive unexec().

> I think glibc also performs some runtime checks at init time (kernel version, CPU hardware features) and uses those to select between different versions of some functions, so such a dump would not be as portable as proper dynamically linked binary.

Good point. In case of emacs one could check if CPU is the same as the one that was present before unexec() and fallback to normal startup if it is not. One could include checking of a checksum of shared objects if one doesn't want keeping their code in an unexec'ed image.

In my optinion unexec() is quite a general method and probably it can be used by other software too. Probably even ability to suspend to ELF/resume from ELF for arbitrary processes would be useful. How to deal with opened file descriptors is an open question though.

Removing support for Emacs unexec from Glibc

andresfreund — Fri, 29 Jan 2016 08:19:25 +0000

On the other hand, these arrays cause other major problems. Fun like "TeX capacity exceeded, sorry [number of strings=245828]" etc...

Removing support for Emacs unexec from Glibc

iabervon — Fri, 29 Jan 2016 08:08:44 +0000

The TeX implementation is much like the proposed approach of making a big data table out of what has been loaded in temacs, although that's less special in Pascal (what TeX is written in) than C. In Pascal, you can't have storage which can be cast between pointers and non-pointers, so TeX pretty much doesn't use pointers and instead has indexes into a large integer array; dumping that array and loading it into a different process isn't nearly as messy as trying to save a C heap with pointers or transforming a C heap into an array without pointers.

Removing support for Emacs unexec from Glibc

neilbrown — Fri, 29 Jan 2016 04:31:04 +0000

Back then there weren't many slow-starting applications - computers were slower but we expected them to be faster! (Today they are extremely fast be we have become used to their inevitable sluggishness).

There were only two applications that were useful enough to get away with being bloated: emacs and TeX. Both used the same "unexec" trick, though probably different implementations.

It's been sour grapes ever since...

pr1268 — Fri, 29 Jan 2016 03:38:39 +0000

"The kernel has special code to allow vi to clear the screen without leaving framebuffer glyph artifacts."

And ever since the vi devs got the kernel folks to accommodate their special request, the emacs team has been jealously crying foul. How unfair that vi gets special support from the kernel and emacs has to settle for just glibc!

Just kidding. And being facetious. ;-)

Seriously, though, while I know glibc has its origins in the work of Roland McGrath and Ulrich Drepper, I'm convinced that RMS made lots of demands/requests etc. with regards to its development, seeing how emacs was/is his longtime pet project.

Removing support for Emacs unexec from Glibc

NightMonkey — Fri, 29 Jan 2016 02:53:43 +0000

I know I'm ignorant of the reasoning and history here, but, somehow, it seems shocking that Glibc has code that exists primarily to support building Emacs. Seems very "wag the dog"! One level removed from saying "The kernel has special code to allow vi to clear the screen without leaving framebuffer glyph artifacts."

Don't get me wrong - I know all significant software likely has a sordid story in its history. :)

Removing support for Emacs unexec from Glibc

JanC_ — Fri, 29 Jan 2016 01:13:40 +0000

Ha! Now if all the other developers of slow-starting applications would have known there is a magic trick to solve that!

;-)

Removing support for Emacs unexec from Glibc

kamil — Thu, 28 Jan 2016 20:23:32 +0000

I'm guessing it could work if, as you say, all the shared objects were dumped as well -- importantly, including glibc). That would, however, be akin to static linking, which is a practice frowned upon these days. Glibc doesn't even do proper static linking anymore as far as I know, still requiring some shared objects (NSS) to be loaded dynamically.

I think glibc also performs some runtime checks at init time (kernel version, CPU hardware features) and uses those to select between different versions of some functions, so such a dump would not be as portable as proper dynamically linked binary.

Removing support for Emacs unexec from Glibc

sorokin — Thu, 28 Jan 2016 11:59:39 +0000

Could somebody elaborate why the implementation of unexec() should require any support from memory allocator in glibc at all?

Couldn't a program just dump all its memory (including memory allocator data structures and code of all shared objects) on disk?