Recently posted comments

Once all the boilerplate code is in place it's seems like alot of unnecessary complexity compared to having started with something like a signalfd or eventfd. OTOH, for some things, such as catching SIGSEGV to extend mmap'd stack data structures, or to interrupt and switch program flow without tight, language-level integration of components, the old semantics are indispensable.

Threading semantics compound the headaches. But as for cancelations in particular--I don't think anybody defends the idea any longer. It's almost a strawman at this point as few if applications actually use them and implementations effectively disclaim liability. I'm surprised there's no movement to remove cancellations from implementations, POSIX, or both.

Is that conjecture or can you back up that claim with data? That might make for an interesting research project.

> A special checkpoint/restore functionality for people who NEED it explicitly (see: Redis) would be much better.

That's an interesting idea, is there an implementation of that in any OS? The problem is as long as you don't have a critical mass of systems implementing the new semantics you'll still have to use fork() and then the question quickly becomes whether or not it's worthwhile to cover other cases.

> After all, why should this very niche use dictate the design of the OS?

Compatibility with existing software.

Isn't that how teaching works, at least in the beginning?

A special checkpoint/restore functionality for people who NEED it explicitly (see: Redis) would be much better.

After all, why should this very niche use dictate the design of the OS?

So after fork(), when there is only one thread in the child, it just creates its own new locks and synchronisation primatives and off it goes. No reinitialisation or destroying is needed in the child.

One thing this can break is Valgrind, which creates some high numbered descriptors above the normal ulimit(). By testing the ulimit you can avoid closing these. Other libraries and tools may not be so lucky though, so an O_NOCLOEXEC maybe better, and it's actually a O_NOCLOFORK that would be best.

There's no OS police that will come knock down your door and arrest you if you don't implement fork in your research system. People create all kinds of research OSes that make all kinds of oddball decisions all the time, unikernels being a specific example of a system that (generally) doesn't implement fork (or certainly doesn't implement fork the same way as everyone else). Sure if you want a POSIX compatibility layer then you need to implement fork because fork is a required part of POSIX but if your hope is to change that then don't be surprised if people tell you to go to hell for wanting to inconvenience the roughly 20M software developers and 4B software users who rely on fork semantics on a daily basis to reduce the amount of work that maybe 10K of OS researchers will have to do in those rarish circumstances where they need to.

I think the fact that the arg names are exposed by default is essentially a good accident that forces interfaces to be a bit more usable than they are by default.

Amusingly I went to a good deal of trouble in some code I made to make the args keyword-only without making the implementation of the callsites harder to read. I had explicit documentation of how calls were required to be made, but used inspection to enforce it. I could used def func(**kwargs) but manually coding all the dispatch would have been really unfortunate.

https://lwn.net/Articles/652125/
https://lwn.net/Articles/675433/

To be fair, it did survive longer than I predicted.

Everything else can probably be expressed much simpler with a newer sane process API.

I think the authors would disagree with you. They claim (rightly) that CreateProcess and pthread_spawn are fundamentally incapable of the expressiveness necessary of a core primitive.

Read fairly their claim is that both CreateProcess and fork+exec suck. The fork+exec model is more expressive and powerful, CreateProcess less of a burden on the kernel and more performant. Their preferred alternative is cross-process operations, though as I mention elsethread the pros and cons basically mirror the debate regarding microkernels, IMO, and unsurprisingly (from the perspective of operating system researchers busily writing experimental kernels) substantially shifts the complexity burden to user space software.

But again, this needs an API that has a process handle as a first-class object.

But Unix *has* cross-process operations in ptrace. Nobody is really clamoring to use that interface to build a better fork replacement because fork *already* represents a compromise between how much complexity to put into the kernel and how much complexity to put into userspace, and additionally how costly (mostly in complexity, not performance) the implementation must be. It shouldn't be shocking that those responsible for the kernel side are complaining about having to put in so much work; nor should it be shocking that they heavily discount the cost to user space by shifting the remainder of the burden to them.

Taken to its logical end the paper's argument basically mirrors the same arguments as for microkernels. And while I think microkernels are great and am eagerly waiting an excuse to put seL4 to some use, fork+exec is sufficiently flexible and performant to have ushered in the age of containers and other more complex process management strategies.

While cross-process operations would be more powerful we can't underestimate the cost necessary in building the stack of software that would be necessary to bring the promise to reality. It's the same inconvenient truth as with microkernels. fork+exec is just too good enough, whether by accident or design.

Still not enough, as "attempting to destroy a locked mutex results in undefined behaviour" (from http://pubs.opengroup.org/onlinepubs/007908799/xsh/pthrea...).

*This* is a killer, because it means you are constrained to whatever process-setup code the people who specified the replacement (posix_spawn(), say) happened to think of, and you can't add to it because in a system without fork() you cannot implement your own spawn replacement, but have to rely on whatever limitations the one in the OS happened to provide. The fact that the posix_spawn() API family is already a horrible tentacular monster and is *still growing* and that it is trivial to generate scenarios it cannot handle suggests that this is a rather serious limitation, and a limitation that bites real code. (Even the scenarios it can handle are really hard to read because it has to handle so many cases that it really wants to be a programming language but isn't.)

Even better, have the spawn() syscall specify an arbitrary executable to do the job of ld.so, and then pass the real executable to it as one of the parameters you send over IPC. Then you can do whatever you want to set up the process even if the standard component doesn't support it.

Of course the elephant in the room is that even if everyone in the world agrees that fork() should go, getting rid of it in the application software people care about is a very long-term project whose benefits would take a long time to be realised. Perhaps all the more reason to start working on a transition now, initially by designing, implementing and deploying the replacement APIs.

NB: I'm not going to read a paper presenting decades-old VMS 'design [irr]rationales' as 'new reseach'.

Why is it ever actually necessary to prevent the use of keyword arguments? The article mentions "what if the API changes", but to me that's all the more reason to use keyword arguments. At least you'd detect the problem. Forcing positional arguments sounds dangerous to me.

I've always found fork() much better than CreateProcess() because it's simply untenable for a single function call to set up every aspect of the child process, and the authors acknowledge this in section 6. Their suggestion to get around that is to have system calls that let you change the state of the child from the parent. Unfortunately that creates its own problems --- richer system call API surface with more potential races and security issues (though those races also exist for "modify own process" system calls in the presence of threads).

Another possible approach that they don't mention would be to replace fork() with a spawn() function that can handle common cases, but still support exec(). Then for complicated cases not handled by spawn(), you would spawn() a helper binary that communicates with the parent to complete setting up the child before exec()ing the real binary. Then again, Linux execve() is *also* a big problem, requiring all kernel resources to specify what happens when an execve() occurs, and also having problems with multithreaded processes. (The section of the ptrace() man page on execve() of multithreaded processes is very scary.) So maybe the way to go is to eliminate both fork() and exec() and have spawn() start execution in a standard userspace stub like ld.so, which supports a standard protocol for communicating with the parent to set up the child process environment before entering the real binary.

The paper would have been stronger if they had also discussed issues with execve() and talked about the benefits of eliminating *both* fork() and execve() in favour of spawn().

One thing that worries me about marginalizing or removing fork() is that a COW memory snapshot system call is still very needed to implement rr replay and other things. fork() being that syscall is great for us because it's so commonly used, it's guaranteed to work efficiently and well. Then again, a dedicated COW snapshot call could potentially eliminate some of the problems with using fork() for this, e.g. the fact that shared memory segments aren't copied.

It's not so much a reinitialise mutexes in the child, but more of a 'create new mutexes for the new process and replace any references to mutexes from the parent' that needs to happen in the child() call.

Sorry if I wrote reinitialise and threw you previously.

"""
...found 1304 Ubuntu packages (7.2% of the total) calling fork, compared to only 41 uses of the more modern posix_spawn(). Fork is used by almost every Unix shell, major web and database servers (e.g., Apache, PostgreSQL, and Oracle), Google Chrome, the Redis key-value store, and even Node.js.
"""

That's outright manipulation of the available facts, good enough to be included in propaganda textbooks. For starters, fork() is used not only in a way immediately followed by exec(). E.g., Redis uses fork() as a method to obtain a consistent snapshot of the database in memory, without running a separate executable. While indeed not every use of fork() is justified, the authors could at least not mix examples convertible and not convertible to posix_spawn().

This neatly avoids all the complications of forking and memory overcommit.

That's what Fuchsia does, btw.

Three of the (four) authors are not from Microsoft Research. You're going to be surprised when you find out what it is the majority of MSR employees do (hint: it's research, and a lot of it uses Linux.)

> Personally I think fork is a nice enough call and actually an advantage to have. For example, Git rather suffers in performance on Window because of the lack of that call (amongst other things).

This is just a flippant response, the paper's entire argument has nothing to do with whether it's "nice enough" or "easy to use", you can use `posix_spawn` for that. It's arguing that the contract implementations of `fork()` must provide is a large burden on the design of new systems that is worth reconsidering, e.g. the section that outlines the design of K42, an object-capability system, had its design significantly burdened by general `fork()` semantics, because suddenly you have to start talking about what state objects can be in after they are cloned, rather than when they are created fresh. Similarly, thanks to the semantics of `CreateProcess` actually spawning separate process (vs fork), it can work in environments that have single-address spaces -- for example, by porting their APIs to different runtimes, you can support single-address-space multi-process SGX enclaves with the same API, same with Unikernels, etc (Section 5). If you actually treat the *process* as the object of work (the object which your vocabulary is designed around) then it's easy to see how this works vs fork. Even if fork is "nice" to use for users, it has significant design ramifications throughout the system.

Also `CreateProcess` on Windows by itself isn't "expensive" because it's expensive to map in new address spaces and run main(), and Windows is just weirdly mysteriously terrible at it and they just never cared, it's expensive because actually initializing user contexts in a Windows process is expensive (that requires kernel locks, context switches to system services, thread/stack initialization, etc) but this is a wholly separate design issue. There's no reason to believe a well-implemented spawn mechanism can't have excellent performance because Windows has technical debt (and given the chance to start fresh, if you wanted a high-level spawning API, you'd almost certainly try to *ensure* it has good performance from the get-go.)

The paper never even makes the argument "`CreateProcess` in Windows is better and easier to use than `fork`" (which some other people here have read somewhere, and something that would be pretty hard to argue anyway), merely that the actual underlying design distinction -- an API revolving around spawning actual processes as a kind of first class object in the system without imposing semantic requirements on the memory subsystem -- is perhaps a better one, moving forward. The arguments seem pretty good, to me. I could live without `fork()` if it meant I got something in return, and it seems I do get some things in return.

It also comes across as a good example of where higher level, domain-specific APIs for users give more flexibility than a low level one when considering the overall system design. A higher level API necessarily gives more degrees of freedom in the implementation which, in turn, helps isolate users from underlying implementation details. You can of course go too far here, but if you capture the domain properly, then you can have the advantages of implementation freedom combined with the exact control you need.

Here's a similar discussion I had recently: if you want to write an efficient C program (energy efficient, time efficient, whatever), compared to some existing one, it will never be enough to just throw a new compiler or better optimizations at said existing program -- you cannot choose better instructions or register allocate your way out of it, etc. That will be peanuts compared to real gains you can have. Real optimization comes from choosing a different design, different algorithms and different memory layout -- the kind of decisions that are impossible for the C compiler to make for you while preserving semantics. But tiling, memory locality optimizations are much more common and practical in more restricted settings if you take away a few degrees of freedom from the user, and give it back to the computer -- the Halide image compiler is an example of this. So it is not a problem, or even a "failure", of individual *technical tools* that you are using (calling it a "failure" is not based on any technical understanding of the problem, but on the deep, social, human desire to assign blame, in order to rationalize and reduce complex interactive failure into singular causes.) It is an impedance mismatch in the *abstract language* you are using to communicate with the machine, in turn, restricting the degrees of freedom the computer has for response. Language problem, not a technical one.

Generally I enjoy LWN but most people here have (IMO) failed to level any actual substantial criticisms of the paper (beyond made up ones in their head) before dismissing it offhand, which is sad because it's pretty well written and easy to approach. You don't have to have .patch files in hand every time you want to take some basic idea and run with it and see where it might lead; and in fact, such a demand hampers actual progress more than anything, but that's another discussion...

From the paper:

> Ironically, the NT kernel natively supports fork; only theWin32 API on which Cygwin depends does not

The paper gives several motivations, but I reckon the primary one comes from the authors' work as OS researchers interested in making new research operating systems. Currently, fork() usage is so prevalent in UNIX software that they are faced with implementing fork() (which they claim - I see no reason to doubt their experience in this area - infects the entire OS design) or have an OS that can't run huge amounts of existing software out there (removing a valuable testing resource and possible adoption path for the OS).

It's notable that S7 only suggests the OS might be rewritten to not have fork() as a core syscall after the most important software (however you want to define that, I guess) has been rewritten to avoid it. They're a little vague on how either part of that process would happen, but the purpose of the paper is just to convince people that it should be done, not set out a plan for achieving it.

Vlastimil Babka pointed me to https://lwn.net/Articles/717950/ which I understood to mean that it's not at all straightforward to implement and some larger refactorings would be necessary, and that's certainly beyond my ability. So I'm wondering if we're any closer to being able to add such functionality today, and whether others think my idea of reflink on tmpfs is good or bad.

[1]: https://blogs.vmware.com/consulting/2016/09/anatomy-insta...
[2]: http://www.cs.toronto.edu/~brudno/public/pdf/lagar2011sno...
[3]: http://www.cs.toronto.edu/~sahil/suneja-hotcloud14.pdf

I can only assume Microsoft wants fork() to be removed from the kernel so its easier for them to support Linux apps on Windows. Otherwise why ask for its removal? Why not just educate people on the alternatives?

A very confusing paper, and very much an editorial rather than a research document.

These researchers happen to be right. fork requires overcommit, and overcommit is the enemy of guaranteed forward progress.

iterating over containers of containers:

for a, b, _ in [(1, 'a', []), (2, 'b', [])]: ...

for functions that need to fulfill an interface:

def f1(f: str, *, opt: bool = False):
def f2(f: str, **_):

myval = [f(myval, opt=True) for f in [f1, f2]]

with the python3 partial container explosion syntax:

a, b, *_ = (1, 2, 3, 4)

and a bunch of other cases.

int pthread_atfork(void (*prepare)(void), void (*parent)(void), void (*child)(void));

prepare() should do something like take locks for exclusive access on the malloc() area (potentially blocking until exclusive access is guaranteed), then returning to allow the fork() to proceed. parent() can drop the locks again in the original process & thread, while child() can replace any locks with new ones specific to the child.

Of course, glibc implements both malloc() and pthread_atfork() so can use internal mechanisms to achieve the same, but it's still there for others if needed on other resources and you really have a design that calls for fork() and threads.

It's also six characters long and good enough for the rest of the world. If this also-ran open-core company can't/won't build something more compelling, then all of this is just intellectual onanism and whining over something they don't have the brains to implement efficiently.

* Thread A calls malloc()
* Thread A acquires malloc() lock
* Thread B calls fork()
* Thread A releases malloc() lock

Now the child of the Thread B ends up with malloc locked for eternity and any attempt to allocate memory will end up with a deadlock there.

It's only easier to teach if you ignore the hard parts.

Or in other words... if you gonna use functions from multiple abstraction layers (in this case libc vs thin-libc syscall wrappers) then it's on you to properly manage them.

I'm not sure fork() is an inspired design, however the assertions at the beginning of the paper don't fill me with confidence about the authors

It would have been a bold decision to make new sub-systems implicitly set CLOEXEC by default, but it perhaps is only now more obvious with hindsight that such a could would have been saner, but it's not the fault of fork() that it came first.

And still there is no better suggestion of a replacement or upgrade.

But that is the point the paper makes. Because of the fork() design *everything* else has to work around the limitations.

The problem is that we when you get to threaded-screws, the hammer no longer works well, and you end up with splintered walls and bashed fingers. So you upgrade your toolbox with better hammers, and maybe some screwdrivers. You might even go with a toolbox with no hammer in it (aka Windows). You quickly find that all of the remaining tools still have enough rusty bits to give every program still gives you a bad case of tentanus, gangrene and blood poisoning while trying to deal with threads.

In the end, papers do not fix things.. especially papers which do not give code which clearly shows a better solution. They may provoke people to think about building better tools.. but even then it takes multiple generations of people who are happy with their rocks to retire before you even get claw hammers or phillips head screwdrivers.. and even then you will find that someone stuck a nice sharp rusty point on the #1 phillipshead and the fix was to keep wrapping it in indirection duct-tape until it only pokes you now and then.

Actually, I have uMatrix set up to block cross-domain redirects by default. It is only really a problem for shopping carts and OAuth stuff which bounce the browser around, but this is a known situation and I can reasonably predict when such stuff is actually going on to know when to relax the behavior.

For me personally, Skype is the only thing now holding me back from switching to Wayland.

Fork() is absurdly hard to use in multithreaded cases because of the implicit forking of resource handles of various kinds. It's also really hard to do error handling around it correctly(!). In languages like C it's certainly *possible* to use it correctly though it's rare to see people get the edge cases right, but try asking any language runtime implementors how much fun they had having to work around its nightmarish semantics.

There's nothing elegant about fork() at all -- it overconstrains implementations by way of its semantics for a use case which is almost always going to be "start a new subprocess /usr/bin/something with arguments X, Y, Z". It's a complete mess because of its implicitness. Explicit over implicit all the way!

The real fork drawback is that it does not have sane semantics in multi-threaded semantics and using it with threads with shared memory do more harm then good . But fork in single threaded applications that uses it for computational workers works nicely and may even leads to better CPU cache utilization.

Windows CreateProcess() takes 10 or so direct parameters, of which some are structures of yet more parameters again. Yuck.

Some of the complaints in the paper seem misplaced too (fork is slow, doesn't scale, forces memory overcommit). fork() has a certain purpose and trying to use it for performance is something which we already know doesn't work that well - that's why webservers went through design iterations of pre-forking, using threads, async IO or some combination of techniques.

Suggesting fork() is insecure possibly has some truth as inheritting most things by default (except other threads) is the opposite of what would be safest, but it's too late to change. Perhaps a new part() call that inherits little could be made, but the knock on is then making APIs for everything to opt into inheritance.

>>> def fun(name, *_):
>>> print(_)
...
>>> fun("A", "B", "C")
('B', 'C')

<3

Personally I think fork is a nice enough call and actually an advantage to have. For example, Git rather suffers in performance on Window because of the lack of that call (amongst other things).

def fun(name, *_):
...

This is a good argument in favour of <a ping> actually; it allows sites to stop obfuscating outbound links.

Since JS was invented, scripts have been able to navigate the current page to some arbitrary URL with or without any user action.

> Back in the days they were always recommending to check the preview of the link to know if it was safe to click or not

That advice makes sense if you trust the page containing the link (e.g. results from a search engine you trust).

If you don't trust the page containing the link, there's no point in checking the link preview because the page can send you wherever it wants at any time, and that's been true for 20 years.

I get the advantages of BPF over e.g. a new LSM hook in this particular case.
What I'm uncomfortable with is, I guess, the fact that it's one more step towards making BPF a mandatory skill for sysadmins trying to correctly enforce their security policy.

But IMO it is a grave security issue from browsers, because the preview of the link shows an URL, but then copying it or clicking it actually takes me to a completely different URL.

Back in the days they were always recommending to check the preview of the link to know if it was safe to click or not, and now browser vendors have made that unsafe to trust.

Maybe you meant that all _required_ arguments should have been made positional only? But, then what about optional arguments? Those must allow keyword calls?

Of course such a change would be quite backwards incompatible. If only they had used the one chance they got to make such changes for actually fixing the language...

Instead of reimplementing WebAssembly badly, just go ahead and support it in the kernel.

On Linux unlike BSD, the binary mechanism is an emulation layer of /proc/sys. While at one point the binary layer was almost equal, that is no longer the case.

The binary layer kept having conflicting paths added, was not used, was never tested, and had security holes the /proc/sys path did not. So a while ago I just reduced it to an emulation layer so we could forget about it.

Which is a long way of saying about the only thing the BSD and Linux sysctl implementations have in common is their name.