Microsoft research: A fork() in the road

Posted Apr 10, 2019 13:32 UTC (Wed) by ibukanov (subscriber, #3942)
Parent article: Microsoft Research: A fork() in the road

The paper argues that fork encourages overcommit of memory, as it is a feature that one should avoid. But overcommit is pretty much a necessity in VM environment or on a modern desktops with memory compression and cannot be used as an argument against it.

The real fork drawback is that it does not have sane semantics in multi-threaded semantics and using it with threads with shared memory do more harm then good . But fork in single threaded applications that uses it for computational workers works nicely and may even leads to better CPU cache utilization.

Microsoft research: A fork() in the road

Posted Apr 10, 2019 21:00 UTC (Wed) by rweikusat2 (subscriber, #117920) [Link] (12 responses)

Ehh ... sorry but that's a pthreads drawback. The people who designed that came from the "fork useless step in front of exec!!1" camp and imagined their baby supplanting all other uses of fork. That's an ancient holy war clotted into a specification.

NB: I'm not going to read a paper presenting decades-old VMS 'design [irr]rationales' as 'new reseach'.

Microsoft research: A fork() in the road

Posted Apr 11, 2019 7:23 UTC (Thu) by ibukanov (subscriber, #3942) [Link] (11 responses)

How a multi-threaded application can do a fork in a sane manner? I mean the real fork that does not follow almost immediately by exec when suspending all other threads is OK.

Microsoft research: A fork() in the road

Posted Apr 11, 2019 10:31 UTC (Thu) by dufkaf (guest, #10358) [Link] (10 responses)

And why should multithreaded app do it if not for the exec? For all other cases it can just create new thread?

Microsoft research: A fork() in the road

Posted Apr 11, 2019 15:33 UTC (Thu) by ibukanov (subscriber, #3942) [Link] (2 responses)

But if the only point of fork is to call exec, then why should it exist in the first place? What is necessary is API to create a new suspended process, setup it in any necessary way and start it. CreateProcess and posix_spawn tried to provide a single function both to setup and start the process. The result was bad and awkward. But it does not mean that fork is necessary.

But my point is that for single-threaded applications fork has clear semantic. For example, to spawn a computation, prepare in the parent process all data in memory, fork, compute and send the results back using a pipe or shared memory. Works nicely.

Microsoft research: A fork() in the road

Posted Apr 11, 2019 17:04 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link]

> But if the only point of fork is to call exec, then why should it exist in the first place?

Even assuming this was true (and it isn't), the original fork use-case would come to mind here: Execute a command in a background process instead of in the current one.

A use-case for 'exec in same process': In-place update of a running program. The currently running instance serializes and records its current state somehow and then execs itself, causing the updated program file to be loaded. The new instance then restores the serialized state and continues where the previous one left off.

Microsoft research: A fork() in the road

Posted Apr 12, 2019 19:09 UTC (Fri) by dufkaf (guest, #10358) [Link]

I was anwering to "How a multi-threaded application can do a fork in a sane manner?". One can fork as much as needed to create separate child processes first and then possibly create threads in each of those (which I guess is not a problem?) but why forking already multithreaded process (instead of creating additional thread) if not for the exec?

Microsoft research: A fork() in the road

Posted Apr 11, 2019 16:58 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link] (6 responses)

Simple example: I have program here which is supposed to capture stdout and stderr output of another program and forward that to syslog. In order to guarantee a sensible process hierarchy (the other command should be the child of the parent process of the program, not the log forwarder), this program forks, executes the other command in the original process and provides the capture-and-forward via two threads running in the forked process (because this was easier to implement than using I/O multiplexing).

Microsoft research: A fork() in the road

Posted Apr 11, 2019 17:18 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

Why shouldn't the log forwarded be a child?

Microsoft research: A fork() in the road

Posted Apr 11, 2019 17:35 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link] (4 responses)

Simple answer: Because the original parent might want the exit status of the payload command and not that of the log forwarder.

More complicated extension: For my usual use-case, the parent of the log forwarder will be a program which monitors another program, restarts that if it terminates unexpectedly and provides facilities for reliable termination of the other program and for reliably sending signals to it (feels like wrong grammar ...). For this to work, it needs to be the parent of the payload process.

Microsoft research: A fork() in the road

Posted Apr 11, 2019 17:51 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Uhm, then why fork()?

In your wrapper just spawn a log process, passing your stdin/stdout to it. Then exec() the payload.

No fork() required.

Microsoft research: A fork() in the road

Posted Apr 11, 2019 18:03 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link] (2 responses)

There's no "wrapper" here. The example I used is a command supposed to log stdout and stderr of another command. And the code of this command has to run in some process, just not in the original one. Besides "this could be implemented in another way" is not an argument. Everything can always be implemented in another way.

Microsoft research: A fork() in the road

Posted Apr 11, 2019 18:07 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

So your code will be cleaner with spawn(), it will work faster (no VM cloning), have less overhead (no need for overcommit) but fork+exec() is better?

Microsoft research: A fork() in the road

Posted Apr 11, 2019 19:07 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link]

I have absolutely no idea what you're writing about.