LWN: Comments on "Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)"

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

drdabbles — Mon, 03 May 2010 21:46:34 +0000

The amount of vitriol, misunderstanding, and desire for status quo is unbelievable at times lately. The Ubuntu distribution is leading the charge (along with several other distros) to modernize the boot process. Some features make it past the chopping block, and others don't. It's that simple.

You can write software for years and it still won't address every corner case a user or developer could possibly throw at it, so you write the best software to solve the core problem.

Most of the "problems" with Upstart come from people not reading the documentation. Upstart allows you to configure the way you start or stop a script at boot by listing the events that should kick it off. If you want to prevent a service from starting on the next boot, comment out the line that tells the script what events should signal it to start. No need to create or delete symlinks. And the symlinks were there as a work around for other issues to begin with. SysV startup has been a nightmare for distributions for a long time, and it has left quite a lot to be desired. Upstart brings a framework to each startup event being created that has to be written in SysV scripts (either by inheriting functions and variables from a script kicked off at the beginning, or some other kludgey work around).

The bug report was a complete joke after the first few posts. FAR too many people trying to add their own worthless opinion about a sequence of events they have never even seen before. The root cause for the bug being filed was a user blindly running a script from a location users would almost certainly never run scripts from by hand. And so, instead of reading the script to see what it does, he found out the hard way. Don't supply it with a variable to tell it where to start a `find` process, and it'll default to the current directory. Wouldn't have been too hard to spot that little gem. Should the script be cleaned up? Certainly! Not performing sanity checks on variables is a poor design choice by anybody's standards. But the author decided to push the bug up a level so that the framework would be fixed instead of this one script. Fixing the framework means any OTHER scripts with similar issues will be fixed as well. This is a positive move on his behalf! But still people berate him.

Finally, using a bug report to inform a single maintainer that you will not be using a particular distribution or that you prefer they make a packaging change so Upstart is optional is, to be blunt, pathetic. Try that on a Debian or a Gnome bug tracker and see where it gets you. Bug reports are NOT the place to launch a political campaign against a maintainer, developer, or distribution. It's that simple. If you have a concern for the direction of a package, project, or distribution it should be taken to the appropriate forums or IRC channels. It can be better dealt with there, and is open to general community input. Sometimes you'll find that you are the vast minority in your opinion, and other times you will find a large body of support. But if you don't bother using the appropriate channels, you'll never find out.

The long and short of it is this- Upstart is being used now by several distributions as the main system init. suite. The distributions didn't just use it because it was a new fad to jump on. Everyone has had times where SysV didn't provide enough functionality, and everyone has had times where SysV didn't add a feature for a LONG time due to heritage and compatibility with generic *nix. That's their choice. Upstart brings them the features they wanted while allowing fallback to SysV-style startup. If you find a feature missing, or you find a bug in the way Upstart works, stop bitching about it and submit a bug report. If you think you can do a better job than the maintainer, submit a patch! Community EFFORT is what makes Open Source work. Community bitching and moaning is what makes Open Source fail.

upstart vs. runit

jhohm — Thu, 22 Apr 2010 16:21:05 +0000

One notable runit feature, lacking in upstart AFAICT, is guaranteed logging of job output via a "child" log job, which is not actually a child and will survive the job being killed or dying and not lose any log output.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

liljencrantz — Sat, 17 Apr 2010 09:00:55 +0000

Am I misreading you, or are you actually advocating that using a single shellscript file to stuff your entire boot sequence in is preferable to sysV-init or upstart, because there are so many different files to keep track of? That sounds like a completely insane argument from where I'm standing.

First of all, keeping everything in a single shellscript file is a nightmare for packaging and scripting. For example, if you make a service start up at boot time by appending a few lines to rc.conf, how do you automatically uninstall said service?

Do you sed out the exact line you added? What if the user has edited the file and the editor decided to add a bit of extra whitespace to your line?

Do you try to make a fuzzy match with a clever regexp and pray that it works?

Do you conclude that «starting and stopping a services at boot is way to hard a problem for a computer to solve, let the user deal with it»?

Do you fail to provide any form of working uninstall script?

In Debian, the rule is that everything that should be possible to enable or disable using a script lives in it's own file in a directory named something appropriate, like «modules-available». You enable and disable things by creating a symlink to the file from some other directory, e.g. «modules-enabled». Usually, there is a simple script to do it for you, e.g. «a2enmod». Sure, this involves more files than stuffing your entire configuration into a single file, but since this method is used pretty consistently, it's not harder to figure out for even a slightly experienced sysadmin, and the benefits for packagers, sysadmins and just about anybody how likes to automate things is enormous.

Secondly, a simple server running a small number of services does not require a huge rc.conf file. The base install of a bsd system looks fine. But my laptop, which runs services enabling wired and wireless networking, bluetooth, nfs, mobile broadband, gps, powersaving, printing, file sharing, ssh, apache, three different database servers and a host of other things would have a simply enormous rc.conf file. And I need those things - I use my laptop as my development machine when I'm on the road, and I often need to work without internet access. There comes a point where a single file is significantly _less_ readable than a directory of logically named files, and my laptops boot sequence is far beyond that point. Part of that is because many of the init scripts are extremely bloated, but even if they were all lean and mean, it would still be far too much to stuff into a single file without losing overview of what goes where. In fact, from my (admittedly somewhat limited) experience from the BSDs, all moderatly complicated services are started by writing the service startup code into a separate shellscript file, and simply calling that script form rc.conf. All that you've done in that case is make sure that instead of enforcing a consistent file layout, services will use a startup files with different file name conventions and most likely vastly different internals as well.

What Upstart is not

liljencrantz — Sat, 17 Apr 2010 08:27:47 +0000

True, but upstart _should_ be about speed. Not exclusively of course, but seeing how most Linux kernels are running on ARM and other tiny embedded hardware, and even Ubuntu, the main user of upstart, is quickly moving towards netbooks and smartphones, it should be completely obvious that boot speed _is_ one of the critical aspects of initialization that needs to be handled.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

dmag — Sat, 17 Apr 2010 03:58:58 +0000

I just started playing with runit. It is another init replacement that takes the unix philosophy to heart. It solves neat problems like "every service shouldn't have to have the same boilerplate deamonization code".

I'd be interested if anyone has any thoughts on the benefits of upstart vs runit.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

foom — Mon, 12 Apr 2010 15:25:20 +0000

Another problem with set -u is that bash has a very irritating bug: it calls an explicitly defined empty array unset! That makes it very hard to do things like splice arguments into a command line correctly. E.g. something like this...The expansion "${x[@]}" fails with
bash: x[@]: unbound variable
instead of correctly expanding into no arguments.

#!/bin/bash
set -u
extraargs=()
if [$# -ge 1]; then
extraargs += ("--filename" $1)
fi

"command" "--dowhatever" "${x[@]}"

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

nhippi — Mon, 12 Apr 2010 08:45:35 +0000

> I have found upstart violating the KISS principle. SysV init is simple and easy to configure.

It is not. You are just *used* to SysV. Common mistake but big difference. And I'm not saying that upstart is simple. Generally the simplest and fastest startup is a hardcoded script aka what the embedded people tend to do...

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

keybuk — Mon, 12 Apr 2010 06:09:20 +0000

For a long time, I strongly considered having Upstart run all of its shell processes with "set -u" (it already runs them all with "-e"). Unfortunately "-u" has surprising results, even for those who are used to shell programming!

For example, the following is a very common way to test whether a parameter is not in fact set:

if [ -z "$MOUNTPOINT" ]; then
    MOUNTPOINT=/tmp
fi

The use of $MOUNTPOINT in the test is still considered an expansion, thus if you try and run this, you get:

/proc/self/fd/4: 1: MOUNTPOINT: parameter not set

There are obviously ways to write the code so that it works with "-u", e.g.:

MOUNTPOINT=${MOUNTPOINT+/tmp}

but I figured that more people wouldn't know these, and would simply get frustrated at Upstart's apparent inability to run (in their mind) perfectly cromulent shell scripts.

What Upstart is not

sladen — Sun, 11 Apr 2010 12:04:09 +0000

Upstart is not about speed, it is about enabling a distro to boot on modern dynamic hardware setups (eg. the system needs to wait for the correct USB hard disk with the root filesystem to be inserted, rather than just dying).

...and what is "new" is that Upstart is not dependency-based, it is the opposite. You start from nothing, and see where you end up; there is no presumption that you'll end up with a functioning system but Upstart will launch what it can based on the combination of presently available of hardware, functioning network connection(s), and the configured state of system software—all of which may change over time.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

cjwatson — Sat, 10 Apr 2010 08:09:35 +0000

I think the LinuxPlanet article placed a bit too much weight on that. Upstart won't subsume udev as far as I understand it; udev rules and Upstart jobs often work together - it's more that Upstart can respond to udev-generated events to start things that aren't suitable for being started from udev. And while subsuming cron has been a long-term goal since Upstart was first designed, the plan was always to make it work really well as an init daemon before getting into the security issues that arise once Upstart is supervising non-root jobs and taking configuration from non-root users.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

cjwatson — Sat, 10 Apr 2010 07:49:51 +0000

The init(5) manual page might be helpful.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

rilder — Sat, 10 Apr 2010 05:35:30 +0000

There is nothing new(or better) in parallel service startup of upstart. SysV can be(and has been) modified for parallel startup of services.
I have used bootchart before to obtain the booting phase timings.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

rilder — Sat, 10 Apr 2010 05:27:35 +0000

SysV init won't handle it. It should be handled by higher level daemons/applications. Having said that, let me point out that there is/was another project which was intended to do these things -- hal. Now it is being deprecated and being replaced by smaller alternatives which do one thing only like upower/devicekit or replaced already existing ones like udev(Xorg is doing this).

But I also understand the scenario you are mentioning -- more like zeroconf -- in these cases upstart may help/work because it provides an unified framework (so that there is no duplication of effort). But making an init system do all these may prove to be costly later when there are too many things it is made to handle (like cron etc.). Events can already be handled by existing frameworks like udev, so any application can be written to read/handle them.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

rilder — Sat, 10 Apr 2010 05:12:12 +0000

Actually the one I use(Arch Linux) is based on a modified BSD SysV style -- so there is no /etc/rcX.d/(where X is runlevel I assume).. There are just /etc/rc.conf, /etc/rc.sysinit, /etc/inittab(for runlevels) etc.. Functions in these can also be hooked. Keeps configuration much much simpler and clean.
That being said, I should also mention that I used upstart long time back, so much may have changed(though it doesn't look like that from wiki). The wiki mentions asynchronous execution capability of upstart as an unique trait. It isn't. There are others(one I mentioned above) which support it.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

Lukehasnoname — Fri, 09 Apr 2010 22:43:06 +0000

It's been a while since I've set up a BSD system, but Upstart on Ubuntu isn't that complicated in its hierarchy.

Startup scripts that actually do work are still traditionally in /etc/init.d/

The configuration files, telling when services are to start/stop/etc. are in /etc/init/, rather than /etc/rcX.d/.

doesn't seem too much more complicated to me. I believe the benefits of Upstart are great; its use cases at present are well justified. I haven't contemplated whether I agree with it taking over mounting drives or cron, though. Also, it needs to document its uses, setup, and syntax FORMALLY (example wikis are not enough, Scott!).

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

rilder — Fri, 09 Apr 2010 18:52:58 +0000

I would like to point out the complexity in managing configuration in upstart. Having multiple directories and symlinks among them makes updating config a very very clumsy process. On the other hand, approach of BSD style SysV to club together all in one file(rc.conf) keeps it clean and error free.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

drag — Fri, 09 Apr 2010 18:22:44 +0000

Quite frequently I use mobile devices were the network topography changes.

For instance I may boot up into my home environment were I am comfortable running services like Avahi, Samba, SSh, etc etc.

Other times I may boot up into a situation were I have no network, or I boot up networks that are insecure and I want to present a hardened exterior to the network.

And the network situation can change while the machine is active.

Now event-driven init system like upstart can handle these situations in a effective manner; even though it's not fully taken advantage of yet by Linux distros.

How would this be handled by SysV style system?

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

nix — Fri, 09 Apr 2010 17:13:59 +0000

It's enough that *not everybody* uses set -u. Even in critical places like startup scripts and security-upgrade scripts issuing rm -r! :)

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

jspaleta — Fri, 09 Apr 2010 17:06:35 +0000

Has anyone seen an analysis of boot times which goes into detail about how much boot time savings native upstart jobs are actually providing?

If the general purpose distros are all still running upstart in hybrid mode, it makes it difficult for me to get a clear picture of what upstart's boot time savings actually are compared to pure SysV init.

I understand that theoretically the parallel service start up its capable of should reduce boot times. But I haven't seen an analysis which measures that specifically. I'd love a reference if someone has one.

-jef

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

MisterIO — Fri, 09 Apr 2010 16:50:35 +0000

You're right, I didn't notice that. Still, the distinction between started-by-upstart and started-by-hand doesn't make much sense IMO. Just write properly all the scripts, without relying on the assumption that they're started by upstart, in particular when that assumption can put you at such a risk.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

brouhaha — Fri, 09 Apr 2010 16:19:06 +0000

I agree that SysV is lacking and that having a better replacement such as upstart is a good idea. But I don't think that having upstart also subsume the functionality of cron and udev is necessarily a good idea. I'd want to see some pretty good justification that it will result in real benefits due to the integration.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

larryr — Fri, 09 Apr 2010 16:15:27 +0000

It seems to me more and more common (on Ubuntu at least) for these system level tools to rely on higher level applications like GUIs to ensure valid configurations. There may be configuration files are which are human readable, if not human understandable without reverse engineeering, but people who actually edit them by hand do so at their own peril.

I prefer this way, though, to situations with just a DBus API and a GUI script which uses the DBus API, and no documentation at all except maybe something from doxygen. I do not mind people who edit config files being second-class citizens, but I guess at some point things will become essentially like Windows.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

vonbrand — Fri, 09 Apr 2010 15:36:43 +0000

"Make everything as simple as possible, but not simpler."
-- Albert Einstein

The SysV style is certainly simple (and the BSDish idea of one humongous script is even simpler), but has been found wanting. In particular, today's systems have to react to hardware appearing and dissapearing ("probe, configure, and set up at boot" is not enough anymore) and a host of other random events. It does make sense to try to bring them all under one roof.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

mdz@debian.org — Fri, 09 Apr 2010 15:24:46 +0000

It doesn't really matter whether symlinks or files are used; your point is that the start/stop configuration is separate from the script. You're right that upstart doesn't have that separation at present, and both are stored in the job configuration file. This does complicate the creation of tools to manage services.

However, upstart's method is quite sufficient for the scenario which was described above, i.e. switching a job to "manual mode". These files virtually never change in post-release updates, so this is a minor consideration.

In response to your final point, Ubuntu 10.04 LTS still uses a "hybrid" arrangement, just as 9.10 did, though a few more jobs have been migrated to upstart.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

vonbrand — Fri, 09 Apr 2010 15:14:47 +0000

Please look at the bug's discussion. Here the real "wishlist" item is modifying upstart so that it isn't necessary to add a lot of boilerplate code to each script. This boilerplate is currently in charge of cheching sanity, and thus critical (and being repetitive and boring, easy to get wrong).

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

vonbrand — Fri, 09 Apr 2010 15:10:24 +0000

That stops the job at hand, doesn't disable it... It boils down to what is more reasonable to do on next boot with a job that was stopped on shudown:

Each boot should offer the same configuration, set permanently in some way. That a particular job was or not running on shutdown is inconsecuential.
A reboot is simply a temporary service interruption, it makes sense to keep stopped jobs stopped, and so on.

The first above is easier to handle, and has the benefit that a reboot gets you into a known good state (so a reboot is useful to unwedge a machine, also in case it got its jobs messed up in some unknown way).

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

vonbrand — Fri, 09 Apr 2010 15:00:10 +0000

The fix is at hand, the problem is to get all the users to upgrade their wetware...

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

rilder — Fri, 09 Apr 2010 15:00:00 +0000

I have found upstart violating the KISS principle. SysV init is simple and easy to configure. Hooks can be added to it.

"over tasks such as or plugging in external devices like thumb drives (currently handled by udev and hal), or running programs at specific times (currently handled by cron)."

-- this is where the problem lies.. Do not try to do more than one thing and fail at everything(eventually).

Deleting the whole file system

clugstj — Fri, 09 Apr 2010 14:35:03 +0000

This bug deletes your whole file system and he (Scott James Remnant) responds that you shouldn't run the script in question by hand without giving it some obscure command line option, implying that it's not a bug. Then he complains that people get upset at him? I don't understand what kind of a response he thinks he will get when he acts like this.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

kreijack — Fri, 09 Apr 2010 14:25:00 +0000

Of course by hand everything is possible. But what happens if you update the package ? The system asked you to decide which scripts (the old one with your configuration, and the new one with the default configuration) have to maintain..

With the sysV soft-link approach, the package-manager is able to update the script leaving the start/stop configuration untouched. With upstart this is a step backward.

A tool which want to manage the services, has to modify a file... From a packager POV this is a mess.

I don't want to say that upstart is fully wrong, but there are some "design issue" which have to be resolved. To me the point is that upstart is too young to be used in a LTS (long term support) distribution.

Debian (for example) still use the old sysv approach (at the time an hybrid: upstart + sysv links); the same is true for Ubuntu 9.10

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

MattPerry — Fri, 09 Apr 2010 14:24:23 +0000

It's still marked as wishlist. One bug can affect multiple projects so if you look at the top, mountall has a fix applied while upstart still sees it as wishlist.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

mdz@debian.org — Fri, 09 Apr 2010 14:03:29 +0000

I discovered that with sysvinit it not possible to switch a job to "manual mode". Even though a job is stopped via the command

# /etc/init.d/jobname; stop

in the next reboot the it will be restarted.

It is possible to move the /etc/rc?.d/S??jobname file to another location in order to avoid the "automatic" behavior...

Seriously, though...this is easy to do with upstart. The /etc/init/jobname.conf file describes precisely which events will cause the job to be started and stopped. If you don't specify any such events (i.e. comment out the "start on" and "stop on" lines), then it won't be started automatically.

So by commenting out these lines, you get the behavior you want: it won't start at boot time, but "start jobname" and "stop jobname" will still work.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

MisterIO — Fri, 09 Apr 2010 14:03:09 +0000

After this I definitely have doubts about this thing! Not because of the bug itself, but because of the discussion that took place there. The maintainer at a certain point flagged that bug as _wishlist_! I certainly don't want to have anything running on my pc wrote by someone who thinks that kind of bug is of the whishlist class.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

cesarb — Fri, 09 Apr 2010 13:46:06 +0000

> this is a perennial Unix problem

Nobody uses set -u? (set -eu is even better)

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

nix — Fri, 09 Apr 2010 13:37:21 +0000

Yeah, but /etc/init is a name-clash with 1980s-vintage Unix systems, while /etc/init.d would be a name-clash with *actual* running systems *now*. Since 'init' is plainly a good basename for init scripts, the solution chosen looks good to me.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

marcH — Fri, 09 Apr 2010 11:58:23 +0000

A different problem I often face is when I start/kill frequently a service under development in order to test it. At some point upstart will flag it as unstable and refuse to start it. My current workaround (shame) is to reboot the machine.

Maybe upstart should consider that a command run from a tty is always right?

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

marcH — Fri, 09 Apr 2010 11:53:00 +0000

Please let me rephrase:

I am glad they stopped naming the jobs directory "event.d"

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

stumbles — Fri, 09 Apr 2010 11:03:39 +0000

I've not used upstart but if it behaves as you say, I would call that a regression or at least unacceptable.

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

kreijack — Fri, 09 Apr 2010 10:36:53 +0000

I discovered that with upstart it not possible to switch a job to "manual mode". Even though a job is stopped via the command

# stop <jobname>

in the next reboot the it will be restarted.

It is possible to move the init/<jobname>.conf file to another location in order to avoid the "automatic" behavior. But so it is not possible to start it manually via the "start" and "stop" command.

To me it seems a big regression to the old SysV init script. I am alone to think that ?

Moreover the last time that I tried it, I discovered that my ubuntu 10.4 (bat is a beta) when booted in single mode, mysql was running !

Ubuntu's Success Story: the Upstart Startup Manager (LinuxPlanet)

nix — Fri, 09 Apr 2010 09:46:40 +0000

Now all we need is a fix for the flaw that causes hordes of people to charge into bug reports flaming madly about a bug that, uh, had already been fixed. (It's also the 'most serious' bug they'd ever encountered, which tells me they weren't running Solaris in the 90s, when there was at least one security patch release that did an rm -rf / by mistake through a very similar uninitialized-shell-script-variable flaw... this is a perennial Unix problem.)