Microsoft Research: A fork() in the road

<3

Windows CreateProcess() takes 10 or so direct parameters, of which some are structures of yet more parameters again. Yuck.

Some of the complaints in the paper seem misplaced too (fork is slow, doesn't scale, forces memory overcommit). fork() has a certain purpose and trying to use it for performance is something which we already know doesn't work that well - that's why webservers went through design iterations of pre-forking, using threads, async IO or some combination of techniques.

Suggesting fork() is insecure possibly has some truth as inheritting most things by default (except other threads) is the opposite of what would be safest, but it's too late to change. Perhaps a new part() call that inherits little could be made, but the knock on is then making APIs for everything to opt into inheritance.

The real fork drawback is that it does not have sane semantics in multi-threaded semantics and using it with threads with shared memory do more harm then good . But fork in single threaded applications that uses it for computational workers works nicely and may even leads to better CPU cache utilization.

Or in other words... if you gonna use functions from multiple abstraction layers (in this case libc vs thin-libc syscall wrappers) then it's on you to properly manage them.

I'm not sure fork() is an inspired design, however the assertions at the beginning of the paper don't fill me with confidence about the authors

It's also six characters long and good enough for the rest of the world. If this also-ran open-core company can't/won't build something more compelling, then all of this is just intellectual onanism and whining over something they don't have the brains to implement efficiently.

I can only assume Microsoft wants fork() to be removed from the kernel so its easier for them to support Linux apps on Windows. Otherwise why ask for its removal? Why not just educate people on the alternatives?

A very confusing paper, and very much an editorial rather than a research document.

Vlastimil Babka pointed me to https://lwn.net/Articles/717950/ which I understood to mean that it's not at all straightforward to implement and some larger refactorings would be necessary, and that's certainly beyond my ability. So I'm wondering if we're any closer to being able to add such functionality today, and whether others think my idea of reflink on tmpfs is good or bad.

[1]: https://blogs.vmware.com/consulting/2016/09/anatomy-insta...
[2]: http://www.cs.toronto.edu/~brudno/public/pdf/lagar2011sno...
[3]: http://www.cs.toronto.edu/~sahil/suneja-hotcloud14.pdf

This neatly avoids all the complications of forking and memory overcommit.

That's what Fuchsia does, btw.

"""
...found 1304 Ubuntu packages (7.2% of the total) calling fork, compared to only 41 uses of the more modern posix_spawn(). Fork is used by almost every Unix shell, major web and database servers (e.g., Apache, PostgreSQL, and Oracle), Google Chrome, the Redis key-value store, and even Node.js.
"""

That's outright manipulation of the available facts, good enough to be included in propaganda textbooks. For starters, fork() is used not only in a way immediately followed by exec(). E.g., Redis uses fork() as a method to obtain a consistent snapshot of the database in memory, without running a separate executable. While indeed not every use of fork() is justified, the authors could at least not mix examples convertible and not convertible to posix_spawn().

I've always found fork() much better than CreateProcess() because it's simply untenable for a single function call to set up every aspect of the child process, and the authors acknowledge this in section 6. Their suggestion to get around that is to have system calls that let you change the state of the child from the parent. Unfortunately that creates its own problems --- richer system call API surface with more potential races and security issues (though those races also exist for "modify own process" system calls in the presence of threads).

Another possible approach that they don't mention would be to replace fork() with a spawn() function that can handle common cases, but still support exec(). Then for complicated cases not handled by spawn(), you would spawn() a helper binary that communicates with the parent to complete setting up the child before exec()ing the real binary. Then again, Linux execve() is *also* a big problem, requiring all kernel resources to specify what happens when an execve() occurs, and also having problems with multithreaded processes. (The section of the ptrace() man page on execve() of multithreaded processes is very scary.) So maybe the way to go is to eliminate both fork() and exec() and have spawn() start execution in a standard userspace stub like ld.so, which supports a standard protocol for communicating with the parent to set up the child process environment before entering the real binary.

The paper would have been stronger if they had also discussed issues with execve() and talked about the benefits of eliminating *both* fork() and execve() in favour of spawn().

One thing that worries me about marginalizing or removing fork() is that a COW memory snapshot system call is still very needed to implement rr replay and other things. fork() being that syscall is great for us because it's so commonly used, it's guaranteed to work efficiently and well. Then again, a dedicated COW snapshot call could potentially eliminate some of the problems with using fork() for this, e.g. the fact that shared memory segments aren't copied.

There's no OS police that will come knock down your door and arrest you if you don't implement fork in your research system. People create all kinds of research OSes that make all kinds of oddball decisions all the time, unikernels being a specific example of a system that (generally) doesn't implement fork (or certainly doesn't implement fork the same way as everyone else). Sure if you want a POSIX compatibility layer then you need to implement fork because fork is a required part of POSIX but if your hope is to change that then don't be surprised if people tell you to go to hell for wanting to inconvenience the roughly 20M software developers and 4B software users who rely on fork semantics on a daily basis to reduce the amount of work that maybe 10K of OS researchers will have to do in those rarish circumstances where they need to.

The authors' criticisms seem to revolve mostly around "fork plays badly with $modern_os_feature", but usually the fault lies with $modern_os_feature (threading being the most obvious example).

The snark about goto in section 7 demonstrates that the authors have exactly the kind of ivory-tower attitude to which Unix has always been opposed; a sensible programming course would start bottom-up, with assembly, to impart the crucial concepts of a computer's execution model; and assembly does indeed begin with goto (although it calls it jmp).

fork()/exec() is an example of brilliant taste by the original inventors of Unix; and taste is like jazz: if you have to ask what it is, you ain't never gonna know.

I started off with Tenex (1.34) and moved to UNIX (V6 w/ Univ Ill NCP (Arpanet)) so I've seen a bit. I had to dig out my old TENEX docs to refresh my memory... It has been a while since I wrote a JSYS FORK. But I'm not bragging my age but rather, there needs to be some longer perspective in this discussion.

As the authors hinted, TENEX did have a FORK that would would do everything in one go; create the fork, populate it with an image, and start it. There were other variations on the theme such as an equivalent of vfork() but it was not all that great. It was pretty advanced for its day but in the end, it didn't really do much other than introduce a real VM with functioning pagefaults. Its filesystem wasn't much better than its progeny VFAT. For example, with its rigid process architecture, there was no way to do "foocmd < bits.in >stuff.out&". Things like threading were also awkward at best. The UNIX model of fork() + exec() solved a lot of those problems. It worked for two reasons. First, a pid is a global object. I can create a process, tell someone else its pid and they can play with it. A fork was and still is cheap(er) and the model of orphaning a pid to the init proc made backgrounding trivial. That does not seem like much but it is when you try to make a network daemon or batch system and don't have it. Second, splitting the two has real power - and with real power there is also risks (and bugs).

A TENEX fork, just like CreateProcess was a single shot. Once it is gone into the system it is gone. Sure, you can manipulate it some but now you have one proc fiddling with another with all the race conditions it implies as shown by the complexity of ptrace(). CreateProcess solves this problem with its mountain of API args. However...

Back with V6, fork() and exec() were pretty simple. But they already had some things that TENEX didn't have. That power was and still is what goes on between the child's return of fork() and its subsequent exec(). In those days, we didn't have too much to do other than some close() and open() calls, usually redirecting one or more of 0,1,2 and the closing of random other files (very rare but possible), and maybe setuid(), setgid(). There were no capabilities or shared anything to clean up. Since those days the number of things to be policed from the old environment before the new one got launched has grown. For example, execve() popped up because we got this thing called "environment" in V7 which sometimes required a scrubbing of the env vector. It only got better and worse. The authors rightfully note this growth of features and the resource costs that go with them. The costs are there no matter what the model used for creating/managing/destroying them. You will have to do those actions somewhere. The only question is where.

They also criticize all the close-on-exec stuff scattered about in the various kernel subsystems that use an fd. Point well taken. Then again, this is still territory where one really needs to know more than just garden variety algorithms. And, where else would you handle things such as this? The kernel doesn't know the significance of one open fd from another and how would you construct an API extension for clone() to handle such an open ended requirement? This is something that only the app has knowledge of the full context. Therefore, it is the app's responsibility. You have a choice to either do it somewhere in the app (the Linux choice) or in a system API somewhere. You cannot fob it off somewhere else.

Consider the following issues on allocated resources, most of them open "files". There was a time when a proc could only have 16 open files. BSD moved that to 20. They then implemented the select() call. Their argument list included bit masks for the fd's of interest in the API. It seemed like a good idea at the time. Besides, what app would have more than 10-12 files open at any one time? Well, guess what. In the early 90's, having only 4k open files in a database or pthread app was a limit. This forced the new select2() syscall because the API had to change to handle a variable length bitmask but the old select() was cast in concrete. The AltaVista webserver, which I maintained for a while back then, blew that number bigtime. The select() syscall was no longer a good idea and select2 with huge holes in the bitmap was only marginally better. Hint, even if all you kept open most of the time were 0,1,2 and a few fds that you had hanging around after a flurry of file stuff, you still could have an fd > 4096... Bit masks were no longer a good idea. Eventually poll/epoll and friends took over. The lesson here is that systems must evolve to address the continuing stream of new requirements.

This is where CreateProcess() and its API comes in. All those arguments are there for a purpose and are fixed for all time unless you are really into pain. But is that all that will be needed/wanted? History shows that no, it isn't. The things that must be manipulated when a process/fork gets created will grow in size. There are three choices for this, the system remains static for all time, you hack the API one more time or have an interim period in the child's code where all this can happen prior to launching the new image with exec(). UNIX chose the last and we benefit from that choice.

The creation of a process/thread in any OS is always tricky. There are timing/race conditions to consider, security vulnerabilities to deal with, etc., etc. This is not code for kiddies. But just as the original exec() -> execve() and fork() -> vfork() issues, new capabilities stretch the limits of the process model and, from experience, adding a new syscall, painful as it is, is better than extending an existing API into uncharted territory. Therefore, having that interim period in the child's initialization has real value. This is where one deals with the future, right there after the child's return from fork(). Only the app knows what privs and resources should be freed before letting the next image take over after exec. Only the app knows that whether a particular resource that it needs must be closed on exec. So have the app do it. The kernel doesn't know enough to do the right thing even if it cared. This is one place in the app codebase where such things matter. It has to be carefully written and debugged. It takes skill to do it. It also has to be done someplace. Launching a proc in any system involves two phases; first, clean up inherited "stuff" and, second, initialize a new, safe new environment for the child before it invokes main(). There are three places to do it. You can do it in this interim code before exec(); you can do it in crt0 or ld.so; or you can do it in the kernel. Take your pick. The interim code is safe in that it is no worse than the rest of the app and fits the need. Doing it in crt0 or ld.so adds special and unique app requirements to a standard/common bit of system runtime. Don't even think about the kernel. Once in the kernel ABI, always in the kernel ABI, and for very good reasons. Hint, they bring pitchforks to these change proposal meetings...

The problems with threads and the mixing of threads with process creation (fork+exec) are real. They are two very different beasts and don't get along all that well. We argued about that back in the cde_threads days and things did not get easier just because we changed the name to pthreads. But pthreads itself, and any of its offshoots such as LWP are not much better. Pthreads does a reasonable job at forking a process but one successfully does such things by abiding by a set of design rules just a little less complex than a EULA. And that is so for a reason. I look at pthread and its mix of mutex+condition vars as being a little more safe than writing the whole thing in ASM, which I did on TENEX, i.e. there is nothing to enforce any of those mostly documented rules and design patterns. There is little help and no guard rails in this model which is why, 30 years later, there are not all that many of us who can really grok this stuff and even then, getting critical sections and lock optimization right is hard work. A pthread call is, after all, just another function call... Coverity et al have to do some real back breaking work (magic) to make sense of the rubbish shoved into it enough to report a reasonable error. Java doesn't offer much in this space either even if it has some threading "primitives" in the language (more or less). C++ is worse, little better than C + pthreads. The closest I've seen in modern languages is the golang model, mainly because it has concurrency (and the constraints necessary to keep it "safe") built into the language itself where the compiler and analysis tools can see what is going on. Also note that they use a "concurrency" model, not a multi-threading model (See Pike's numerous blogs). All the magic is in semantic pass(es) of the compiler and the runtime well out of the reach of the app programmer.

If we look at the fork() implementation in the Linux kernel, we find that fork and vfork are just wrappers around the full blown clone call, all of them calling _do_fork(). The pthread_create() lib call uses clone directly. This is also why pthread_spawn() is faster in their graph. It is a properly clamped down clone() followed by quick lib resources cleanup followed by exec(). This was a smart move when NPTL entered the kernel in 2.6. The kernel doesn't care if a task is a thread or a proc; it just does its scheduling and resources thing. Only the app cares and the library does a respectable job with the rest. Note that its options to COW or share a restricted set of objects is limited to just the things that user code can't manage. Hint: why don't they use clone() instead in their runtime?

The authors made some comments on how, without fork+exec, they could do really cool stuff like load and relocate another process in the same address space. Why would anyone want to do such a thing, other than fool around with some academic notion? Memory management, even the pre-VM segment management in the PDP-11, is a very good thing. The reason is simple. If you can't address the object (in memory or the kernel) you can't piddle all over it. It is bad enough when a pthread goes rogue and stomps on things. Why would you import an unknown quantity like an arbitrary executable into your address space? That is an attack surface bigger than the flight deck of the Carl Vinson. One can escape any language runtime into ASM and once there, all bets are off. In other words, so what if you an load and randomly relocate multiple copies of a DLL/SO. I submit that is a feature in search of a problem to solve. If you want to do such things, use a VM or container and let the hypervisor keep you out of mischief. If you don't want fork, use a unikernel in a VM and get on with it. The realtime gadget people do it all the time with bare iron things like Arduinos and MIPS SOCs.

One last point. Removing fork+exec from UNIX (really Linux these days) is a fools errand. There is one very big reason why anyone would care, other than an academic exercise in woulda-coulda-been semantics. There is a massive amount of code out there that runs inside a UNIX model and it does so for a very simple engineering and operational reason. As bad as it is, it is still, on the whole better than all the OS models it displaced. I mentioned at the top that my first system was TENEX. I also worked on the DECSystem-20, a commercialized version of that OS before I moved exclusively to UNIX/Linux. Those were good systems that did cool things but most of us who left them behind had good reasons to move on. Those systems and all the other "proprietary" systems are now but memories to talk about over beers with other retired hackers. Anyone remember the DG Eclipse? AOS/VS had some really cool features, such as a built in threading model, that were way ahead of their time. But where is DG, or DEC or even Sun now? Most of the UNIX systems are gone leaving only {Free,Net,Open}BSD still chugging away. All those systems have been replaced by a standard system that does its job very well and it happens to be UNIX. Linux has evolved over the years but the core similarities and model are still closer to UNIX V6 than any of the other long gone OS designs. It is the standard OS just like the electrical outlets you get down at Home Depot are standard. Imagine the chaos that would return to metal fabrication if instead of using metric or "English" sizes, one chose their own arbitrary dimensions for thread sizes for fasteners. Having two complete set of tools, one metric and one SAE is pain enough which is why every country (other than the USA) is now almost completely metric. The same applies to current OS ABI/API standards. That massive amount of code only really happened when those of us who had to build real systems stopped arguing and accepted the one system we could all agree (at least in principle) could do the job and we could share in common without a lot of legal/financial friction. The world converged even more so on Linux because of the same reason. People who want to build big, complex systems or who want to build handheld things like smartphones by the billion just want something on top of the iron that they could depend on rather than re-invent. Even Microsoft has figured this out. There is no money in maintaining a proprietary OS anymore other than to support an Office suite that is, itself moving off the desktop and into "the Cloud". Unlike Linux where the development model scales to fill the staffing requirement because everyone and anyone who needs it can contribute their expertise, all of the Windows system specialists who really understand how the guts of the thing works are proprietary need-to-know box on the Microsoft payroll which is why Windows/N, N=1->inf is really in maintenance mode. That group is a "cost center" that can't grow because it would eat the engineering budget alive while providing little more than a support layer underneath their Office products (the real cash cow). Their next new thing, where their dev money is being spent these days, is Azure which is a service whose profitability is based on simple usage scaling not feature development. And yes, most of the VMs and containers they run have fork() somewhere in the runtime.

I don't mean to dump all over the authors. But this piece is an opinion piece, if not a gripe session, not a research report. Cygwin under their research provides a crappy fork() performance, primarily because of the impedance mis-match between the UNIX model and their model running over Windows. So what else is new. My son has solved that problem. He's given up on using things like git on Windows and is tired of the self inflicted incompatibilities in Mac/OS (old python et al). He now has a Windows/10 machine for company stuff like Outlook and runs Fedora 29 in a VM to do his development work which does the deed just fine. When the authors and the users of their OS paradigm have enough code to double the size of github and Sourceforge, maybe then their argument would make sense. Otherwise, this is much about nothing and wishing for unicorns. (Lots of) code that works beats elegant designs that don't (yet) every time.

Sorry for being a grumpy old hacker.

Memory overcommit on fork is indeed sensible but that's a Linux innovation. The default behaviour of 7th edition emulation forks when not enough swap space can be reserved could be described as "suicide out of fear of death": No one knows how much of the inherited address space will need to be copied, this entirlely arbitrary limit thus prefers "guaranteed failure now" over "possible success in future", despite "guaranteed failure now"-mode obviously cannot guarantee that neither of the two forked process will end up failing due to an out of memory situation encountered in a future memory allocation.