Poettering: systemd for Administrators, Part VIII

Posted Apr 21, 2011 20:30 UTC (Thu) by wahern (subscriber, #37304)
In reply to: Poettering: systemd for Administrators, Part VIII by alvieboy
Parent article: Poettering: systemd for Administrators, Part VIII

We're talking about reading configuration files, not writing a web server.

One of my fantasy projects is to write a new shell. The core design idea is to forgo forking. Not because forking is slow per se, but so there can be other optimizations. Instead of forking commands, any time a unique PID is required a slave process from a cached pool will be used to direct signals to the executing process. (Or, perhaps, a kernel extension written for the creation of light-weight PID-only processes.)

With most commands running as coroutines in the same process, you can start adding new features and commands to the shell language not feasible when running a collection of separate processes. Similarly, routines like system() and popen() could be run inside the same process.

The tentative name will be "Project Everything Shell".

It'll compile programs for a simple virtual machine, eventually sporting JIT compilation. Like Java the opcodes will be tailored for shell features, with opcodes for psuedo-forking, etc. In a sense it'll be a tiny Unix-oriented virtual machine.

It won't include interactive features, but because it will be written as a library it will be easy to build an interactive shell. That's a whole 'nother fantasy, though.

I'm not even joking. I'm going to make this happen. Eventually....

Poettering: systemd for Administrators, Part VIII

Posted Apr 21, 2011 22:11 UTC (Thu) by HelloWorld (guest, #56129) [Link] (2 responses)

So what does your "new shell" do that several high-level programming languages don't already do today?

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 0:40 UTC (Fri) by wahern (subscriber, #37304) [Link] (1 responses)

Adhere to POSIX shell syntax and semantics so that it can be a drop-in replacement for existing shell scripts.

The shell mirrors the Unix process model very closely, and few other languages do that. That's what makes it useful. The closest example I can think of is Tcl, but it's no more POSIX-shell compatible than any other language.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 3:21 UTC (Fri) by HelloWorld (guest, #56129) [Link]

> Adhere to POSIX shell syntax and semantics so that it can be a drop-in replacement for existing shell scripts.
In that case, I'm not even remotely interested. The traditional Unix Shell model is utterly broken by design, and it's impossible to fix it without incompatible changes.

Poettering: systemd for Administrators, Part VIII

Posted Apr 21, 2011 22:33 UTC (Thu) by cmccabe (guest, #60281) [Link] (10 responses)

> One of my fantasy projects is to write a new shell. The core design idea
> is to forgo forking. Not because forking is slow per se, but so there can
> be other optimizations.

What other optimizations were you thinking of?

You can already share memory between processes. You can already pass messages between processes without too much overhead. You can already control one process with another process.

Cramming unrelated stuff into a single process makes the kernel scheduler's job harder (in some cases impossible to do well, like on multicore machines). It gets rid of the safety net that a separate address space gives you. Now a bug in your application can mutate the shell's memory and look like a shell bug. Fun fun!

If anything, we need to be moving in the direction of more parallelization and more isolation between components. UNIX got it right the first time around-- and using hardware we wouldn't put in a toaster oven nowadays.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 0:32 UTC (Fri) by wahern (subscriber, #37304) [Link] (8 responses)

What other optimizations? Optimizations which mitigate the performance hit that people are complaining about. For example, exec costs like dynamic linking (static linking of /bin binaries never caught on). Also, you could add speculative loading of shell script commands. I'd want to add useability features like exposing file locking and PID management primitives in a way that makes them as convenient to use as in C. Basically, the idea is to solve all the little issues that make people stop using the shell for trivial system management functions which *should* be using the shell.

The shell's raison d'etre was process and file management. Don't you find it odd that all of sudden people are successfully arguing that the shell sucks at process and file management?

I agree we should be moving in the direction of more process isolation. But projects like systemd are moving in the *opposite* direction. init(1) shouldn't be a complex process, and now at least on Linux it's becoming a behemonth. At some point systemd will sprout a scripting language and history will repeat itself. So if we're going to go crazy, I want to join the party.

My idea wouldn't run all system scripts on a single process. There's no mixing of user privileges so security isn't as much of an issue. Concerns regarding bugs apply equally or more so to systemd. And the behavior of shell commands naturally fit a cooperative concurrency model for which coroutines would be a good fit. Plus, the type of shell I'm talking about could fix the issues with control over long pipelines that current shell designs don't or can't easily fix, such as access to exit status.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 1:35 UTC (Fri) by cmccabe (guest, #60281) [Link] (6 responses)

> Optimizations which mitigate the performance hit that people are
> complaining about. For example, exec costs like dynamic linking (static
> linking of /bin binaries never caught on).

I'm pretty sure stuff like preload(8) has this covered.

> I'd want to add useability features like exposing file locking and PID
> management primitives in a way that makes them as convenient to use as in
> C.

Bash already exposes file locking. man flock(1).

PID file management is going to be integrated with systemd, rather than exposed in shell scripts like it is now.

> Plus, the type of shell I'm talking about could fix the issues with
> control over long pipelines that current shell designs don't or can't
> easily fix, such as access to exit status.

Bash has $PIPESTATUS.

> I agree we should be moving in the direction of more process isolation.
> But projects like systemd are moving in the *opposite* direction. init(1)
> shouldn't be a complex process, and now at least on Linux it's becoming a
> behemonth. At some point systemd will sprout a scripting language and
> history will repeat itself. So if we're going to go crazy, I want to join
> the party.

No matter what you think about systemd, you can't deny that it runs daemons in a separate process. And that is a very Good Thing.

Personally, I think systemd will work out well in practice. But I haven't had a chance to try it out yet.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 2:11 UTC (Fri) by wahern (subscriber, #37304) [Link] (5 responses)

systemd will work well in the short run. But give me an example where moving a hundred different tasks into a single, monolithic C application with a descriptive or declarative configuration format has worked out in the long run.

Everything that systemd replaces is the result of endless experimentation made possible by the flexibility that scripting languages--particularly and especially shell scripting--allows. Systemd will now destroy this process. Systemd won't stop the inevitable hacks that people will implement in the future; it's only cleaned up the past hacks. Future hacks are going to be uglier, more obscure, and more frustrating because they're going to include dreadful workarounds to the rigidity of systemd. That is, unless and until systemd reinvents the wheel regarding scripting support.

Come back to this thread in 10 years, tell me I was wrong, and I'll buy you a beer.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 4:39 UTC (Fri) by josh (subscriber, #17465) [Link]

You can always patch systemd if you don't like its behavior. Or you can install a systemd file that invokes a shell script if you really want to.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 10:13 UTC (Fri) by mpr22 (subscriber, #60784) [Link]

Tell me one thing you would want to do which (a) would need to be managed by one of the things systemd replaces and (b) would be impossible or ugly with systemd. Here's the man pages.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 12:45 UTC (Fri) by mezcalero (subscriber, #45103) [Link]

You are very confused.

The early boot stuff systemd covers is not done from a single process. It's all neatly separated in individual binaries. Nice, modular, simple and parallelizable. All those binaries are simple C programs with no deps but libc. So it's not monolithic at all.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 17:25 UTC (Fri) by martinfick (subscriber, #4455) [Link]

Perhaps you should think of systemd as a "little language"? I do.

http://c2.com/cgi/wiki?LittleLanguage

And, from that perspective, systemd is something well overdo. Unix has many little languages to make simple things simple, and my impression of systemd is that it makes many more simple things simpler.

I love the power of shells, but I cannot endorse what standard linux startup scripts have become. They have always seemed to me to be a place ripe for the introduction of a little language, and I am shocked that it has taken so long! Now that someone has finally done so, and improved on system isolation and dependency management, without taking away the power of using a shell, I am delighted.

I feel that systemd embodies the unix philosophy way better than some tangled mess of shell scripts do (and way better than other startup systems on other OSes). I hope that Lennart continues to push solutions to fill the linux gaps that he sees, particularly the ones which the oldbeards can no longer see because they are so used to overlooking them. Thank you Lennart!

Poettering: systemd for Administrators, Part VIII

Posted Apr 28, 2011 15:56 UTC (Thu) by stevem (subscriber, #1512) [Link]

Absolutely: loss of flexibility, loss of experimentation, loss of tweakability. Ick.

Poettering: systemd for Administrators, Part VIII

Posted Apr 22, 2011 15:58 UTC (Fri) by vonbrand (subscriber, #4458) [Link]

All the system calls, and a good load of "system" libary functions, are available (almost) directly in Perl (and I'd guess in other such languages)...

Poettering: systemd for Administrators, Part VIII

Posted Apr 28, 2011 23:45 UTC (Thu) by steven.blake (subscriber, #13731) [Link]

We could probably use the original Unix hardware as a toaster oven.

Poettering: systemd for Administrators, Part VIII

Posted Apr 21, 2011 22:42 UTC (Thu) by cmccabe (guest, #60281) [Link]

P.S. Sorry to be so negative. I think there are a lot of cool improvements that you could probably make to the shells that are out there; just not the one you described!

Poettering: systemd for Administrators, Part VIII

Posted Apr 27, 2011 12:34 UTC (Wed) by bluss (guest, #47454) [Link]

Is it worth it over just improving busybox?

I'm not saying that busybox should include yet more utilities, but the ones it has provide a good basis for a non-forking shell.. for the important parts.

But if you could help, the POSIX compliance of busybox' shell needs some improvement to be perfect.