Reordering the boot for fun and profit [LWN.net]

From:		Petter Reinholdtsen <pere-AT-hungry.com>
To:		debian-devel-announce-AT-lists.debian.org
Subject:		Reordering the boot for fun and profit
Date:		Thu, 17 Jan 2008 17:49:31 +0100
Message-ID:		<20080117164931.GK26460@klodrik.uio.no>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

For a few years now, I have worked on a replacement for the trusty old
way of organising the Debian boot. Did you ever make a package with an
init.d script, and wonder which sequence number to pick for your
script? I am talking about the numbers in the file names in
/etc/rc[S0-6].d/. Or are you one of the lucky ones that could just ask
for the defaults, and ignore the problem? Picking a good sequence
number is very hard some times, for example when you want to run after
program Z started at sequence number 20 and before program X also
started sequence number 20.

This problem lead to the Linux Software Base (LSB) specification for
init.d script dependencies, and the system I have worked on is a
replacement to make it possible for us to let the sequence numbers be
calculated automatically, based on the declared dependencies in each
individual script.

The current way of inserting init.d scripts, by specifying start and
stop sequence numbers have proven error prone. Edge cases get it
wrong, and there has not been an automatic way to detect these
errors. This was one of the main motivations when I started the work
getting all init.d scripts in Debian to declare their
dependencies. With such dependencies, it is possible to check if the
boot ordering is correct, and also to find a way to order scripts that
fulfil their dependencies. And of course, it also make it possible to
dynamically order the scripts based on their dependencies.

This leads me up to the proposed solution that is now ready to be
tested in unstable. It is now possible to switch your installation of
Debian unstable to use dynamically calculated and dependency based
boot sequencing. It will, as long as the declared dependencies in each
script is correct, make sure the boot sequence is correct.

The implementation is in the insserv package. It does not work with
the file-rc boot method, but I hope this will change in the
future. Installing it will not affect your boot, and after it is
installed, the /usr/share/insserv/ check-initd-order script can be
used to check that the current boot sequence is according to the
declared dependencies.

To switch to a dependency based boot sequencing, it has to be
enabled. It will not change the way the system boot, only the way the
symlinks in /etc/rc*.d/ are updated. The ordering is generated
dynamically by reading the LSB header in each script in /etc/init.d/,
defaults in /usr/share/insserv/overrides/ if the script is missing an
LSB header, and you can override the defaults adding a override file
in /etc/insserv/overrides/ if the script headers should be replaced.

To switch, run

  dpkg-reconfigure insserv

and answer yes to try to enable it. Make sure the insserv version is
1.10.0-3 or newer. When trying to enable it, the insserv scripts will
check the content of /etc/init.d/ and make sure it is sane. Dependency
loops, multiple scripts providing the same service and obsolete init.d
scripts will make it refuse to enable insserv. Those are bugs that
need to be fixed before insserv is enabled.

Give it a try, and report missing LSB headers as bugs against the
packages missing them. Around 70% of the Debian packages with init.d
scripts already got them. The rest are missing. I've reported quite a
few bugs for thsoe already, but there are still some left to report.

I try to keep the status of dependency based boot sequencing available
from <URL: http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot >.
Check it out.

To get this feature ready for Lenny, a lot of work is needed: improve
documentation, reporting patches to BTS for the packages missing
those, talking to the maintainers of packages with missing or wrong
LSB headers, fixing bugs in the insserv package and verifying that the
dependencies in scripts are correct. So I urge everyone to help out
making this happen. The IRC channel #pkg-sysvinit on irc.debian.org
is the current coordination point.

Happy hacking,
- --
Petter Reinholdtsen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHj4a/20zMSyow1ykRAgDTAJwKeu3vUTiHYH4R9oi/W24s/Y/1MgCgzxPU
fSs7mLe+eS0RYFAZzGAEpl0=
=pkcW
-----END PGP SIGNATURE-----


-- 
To UNSUBSCRIBE, email to debian-devel-announce-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

(Log in to post comments)

Reordering the boot for fun and profit

Posted Jan 17, 2008 15:04 UTC (Thu) by drag (subscriber, #31333) [Link]

Ya upstart definately has a lot of potential.

For example I probably don't want things like Avahi, Samba, NFS, Exim, or other such network
things running when I am not connected to a network. Also I probably don't want that stuff
running when I am not connected to a safe network. Sometimes some services may hang or foul up
when networks change or network interfaces suddenly wink in and out of existance. On a server
this sort of thing wouldn't happen, but on my laptop this is a daily occurance. 

Right now I just have things chmod -x /etc/init.d/xxx that I don't want running on every
network I visit.  Then I start them up manually if I need them running. I suppose I could just
have a firewall that I bring up and down, but whatever.

Upstart can solve this problem. With the ability to do very smart dependancy tracking it can
determine what configuration my OS should be in based on whatever criteria and adjust the
services and other stuff accordingly. 

I don't think that Upstart is being used for anything like that right now, except maybe
helping people optimize boot times. It's all just emulating the old static init stuff right
now. It's been a while since I looked at it.

Reordering the boot for fun and profit

Posted Jan 18, 2008 5:47 UTC (Fri) by alankila (subscriber, #47141) [Link]

Yep. Rewriting this seems like a lot of work. And based on my experience, Ubuntu Hardy hasn't
yet done all of it. Some parts such as getty definitely has been upstart-ified, though.

But there is a thorny problem in the method of launch everything asynchronously from events, I
think.

As an example, I have an USB-attached disk that I want to mount on every bootup. Currently
something tries to do fsck on everything a bit too early. The USB disk is not ready, yet the
dumb fsck tries to access it, barfs on missing device file, tells me to log in as root to
correct it.

This is a bit ridiculous. It's supposed to be a removable device. Conceiveably I could plug it
in any time. And I do want it mounted automatically, too. And I know exactly where I want it
mounted. And most of the time, the device is connected and power is on.

Clearly, the only way to do mounts like this is to have udev detect a new device, then figure
out that it's a disk, and check fstab and find that user wants it mounted. So *udev* has to
launch fsck and do that mount, if fsck says it's safe to mount. And that sucks because from
asynchronous daemon you have hard time communicating to user that something is wrong with the
connected device, and he needs to take action.

Assume for instance that there is a disk error during boot time. There's no X yet, no notify
area in console, the system is only half ready. What on Earth can you do? Skipping the drive
might be good idea, but what if /usr is on it? It's conceiveably quite easy to handle failures
of non-critical disk mounts from, say, GNOME (just pop up a messagebox in notify area about
error), but at boot time you are SOL.

So what are the solutions? Start X before starting anything else, maybe? Implement a notify
area for Linux console? Accept that stuff mounted during bootup from fstab is fundamentally
different from anything done later on, and thus fragment the user experience? All these
solutions that come to mind seem to suck for one reason or other. Of all of them, I'd prefer
to drop the stupid splash screen and launch X, actually. Imagine that.

Reordering the boot for fun and profit

Posted Jan 18, 2008 7:19 UTC (Fri) by michaeljt (subscriber, #39183) [Link]

What about letting a console application, i.e. the bootsplash, register for notifications if
it wishes to (this should be possible now)?  And if no application is currently registered for
notifications, simply write to a log and take the most conservative decision possible?  One of
the things which I slightly dislike about all this new udev/hal/etc infrastructure is that it
is all so graphically oriented, although there is no intrinsic reason for it.  It would be so
nice for X11 and the command line to be equally privileged citizens (in fact, more often than
not the command line is still the privileged one).

Reordering the boot for fun and profit

Posted Jan 18, 2008 7:58 UTC (Fri) by alankila (subscriber, #47141) [Link]

Possible. You are right in that the splash probably already could deliver notifications. On
the other hand, why not just start X instead?

Reordering the boot for fun and profit

Posted Jan 18, 2008 12:59 UTC (Fri) by nix (subscriber, #2304) [Link]

udev, graphically oriented? WTF? It's purely-textual, doesn't link with X, 
and as far as I know there are no X tools that talk to it. It *is* quite 
hotplug-oriented, but that's sort of its raison d'etre...

It's true that hal/dbus is sort of graphically oriented, in that there is 
a per-X-session message bus; but more than that hal itself is distribution 
oriented, with the maintainers brushing off or ignoring outright segfaults 
if you're not running a binary packaged by a major distro. Given that hal 
runs as root, this is more than slightly disturbing.

Reordering the boot for fun and profit

Posted Jan 19, 2008 6:29 UTC (Sat) by michaeljt (subscriber, #39183) [Link]

Point taken regarding udev.  Regarding the rest though, I would certainly appreciate say my
CDs being automounted when X is not currently running.

Reordering the boot for fun and profit

Posted Jan 24, 2008 3:54 UTC (Thu) by nix (subscriber, #2304) [Link]

I do that with udev :) (ejecting on unmount is probably doable too, but I haven't looked at
it. umounting on eject I haven't looked at, as without revoke() it's never going to work
reliably.)

Reordering the boot for fun and profit

Posted Jan 17, 2008 15:19 UTC (Thu) by sbergman27 (subscriber, #10767) [Link]

"""
Calling them "halt", "reboot", and "singleuser" would be too easy.
"""

Indeed.  Debian could call them "halt", "reboot", and "singleuser".

Ubuntu would call them "Stop", "Restart" and "Just Me".

Fedora would not be able to decide for their next release, and call them "Zero", "Six" and
"One" as placeholders, until the Ubuntu scheme caught on and they would rename to "Zero -
Stop", "Six - Restart", and "One - Just Me"  with a popup advising you of the fact that their
devs do not approve of the Ubuntu naming scheme but that if you, as a user, choose to use it,
that is your business, but implying that they will think less of you for it.

OpenSuse would take a long time to decide, but eventually end up calling them Yast-Stop,
Yast-Reboot, and Yast-Single-User.

Slackware users would continue to enjoy the 0,6,1 naming scheme, and ask defensively, what it
is you don't like about tgz archives.

PCLinuxOS fans would not really know what is being talked about but point you at
distrowatch.com to show that they're the best.

A few Debian straglers would claim that the change will only lead to RPM hell.

Jonathan Schwartz would blog about it, attributing all of the benefits of this wonderful
transition to Java.

+5: Funny (NT)

Posted Jan 17, 2008 18:39 UTC (Thu) by pflugstad (subscriber, #224) [Link]

Since I have to have some text here, else LWN comment editor complains....

Reordering the boot for fun and profit

Posted Jan 17, 2008 18:49 UTC (Thu) by TxtEdMacs (subscriber, #5983) [Link]

This has the structural tone of a classic.  We need a place to store it, but not just a place
to enshrine its wisdom and wit.  It must be open to allow a branch where it can be embellished
and polished to the finest finish possible, albeit, still allowable by the Lords of the Wiki.
While the original should be venerated as the Core.  The Branch should be known by a related
but better naming, I modestly suggest it become the Kernel-Humorous.

Just think to the delicious mayhem when the Gentoo-DOM add their non-taxable, non-profit two
cents.  And think of the better Debian embodied in the Free Solaris skewing free of reason.
Of course, it would not be done until the better Unix build as exhibited by the
Windows-Whatever were allowed to make their mark.  Only then would it be done with their six
point "x".  Ah the horror ...

Reordering the boot for fun and profit

Posted Jan 27, 2008 14:25 UTC (Sun) by djlin (subscriber, #44831) [Link]

As a point of order, Gentoo actually *already* does this. It starts with runlevel "boot" and then moves to runlevel "default." The scripts themselves contain information about dependencies and the init system takes care of everything. It's actually fairly nice.

http://www.gentoo.org/doc/en/handbook/handbook-x86.xml? part=2&chap=4

Of course, I've never actually set up my system to actually start X from my init scripts, so I'm not sure what that would entail...

Reordering the boot for fun and profit

Posted Jan 17, 2008 19:46 UTC (Thu) by Cato (subscriber, #7643) [Link]

I think this is the funniest comment I've ever seen on LWN...

Funny comment

Posted Jan 17, 2008 21:04 UTC (Thu) by pr1268 (subscriber, #24648) [Link]

I, too, chuckled. I'm glad to see that there's still some creative Unix/Linux humor out there. :-)

Reordering the boot for fun and profit

Posted Jan 17, 2008 22:05 UTC (Thu) by ikm (subscriber, #493) [Link]

> Indeed.  Debian could call them "halt", "reboot", and "singleuser"
> Ubuntu would call them "Stop", "Restart" and "Just Me".

Let's rename /etc as well, shall we?

Reordering the boot for fun and profit

Posted Jan 18, 2008 2:03 UTC (Fri) by michaeljt (subscriber, #39183) [Link]

Please do, it is probably older than half the readership.

Reordering the boot for fun and profit

Posted Jan 18, 2008 4:43 UTC (Fri) by smitty_one_each (subscriber, #28989) [Link]

Gentoo would trigger a dip in the world petroleum supply with a flamewar on a dev list,
leaving no survivors with the will to effect any changes, and just keep the old stuff due to
inertia.

Reordering the boot for fun and profit

Posted Jan 17, 2008 15:35 UTC (Thu) by AJWM (subscriber, #15888) [Link]

Well, on _my_ system, runlevels go up to eleven.

More seriously, why does runlevel 4 exist?  Somebody must have used it for something
somewhere, but no distro that I can recall does.  X11 with no networking? (Would that even
work? I guess if X is configured not to use sockets for local access.)

Reordering the boot for fun and profit

Posted Jan 17, 2008 17:33 UTC (Thu) by sholdowa (guest, #34811) [Link]

Level 4 was originally graphical without networking. However, that's pretty useless anyway,
and involved turning the networking from level 3 off, so nobody ever used it.

This is the only thing that I just can't understand why debian has to screw it up, and use
level 2 for everything, graphical or no.

As for the sequencing, it's not really rocket science - it's alphabetical! Originally the
numbering was such that the Kill script + Startup script = 100 ( or 1000 if you used HP-UX ).
That way, the services were stopped in the reverse order of startup, so there's no problem.

eg S80apache and K20apache, S81tomcat and K19tomcat

Reordering the boot for fun and profit

Posted Jan 18, 2008 0:59 UTC (Fri) by midg3t (subscriber, #30998) [Link]

I think the Debian approach is to hide the archaic concept of runlevels and avoid this
0,6,1,2,3,4,5 confusion altogether.

There are facilities for shutting down, rebooting, booting in rescue mode, and [re]starting
the GUI. Joe User doesn't need to be introduced to this wacky concept of runlevels.

(But he does need to know which display manager is the right one.)

Reordering the boot for fun and profit

Posted Jan 18, 2008 13:59 UTC (Fri) by vmole (subscriber, #111) [Link]

This is the only thing that I just can't understand why debian has to screw it up, and use level 2 for everything, graphical or no.

Because when this decided, many years ago, there was no consensus on what belonged in the various run-levels, nor what the default should be. To avoid ongoing flame wars, we decided to just make them all the same, and let the user adjust as desired. This actually works pretty well, since most users don't have any real use for anything between "single-user" and "everything". The growth of wireless and portables has, since then, made such distinctions more interesting, but even then the useful distinction is not "network/no-network" but "which network am I connected to", which runlevels don't directly solve.

Runlevel 4 mystery

Posted Jan 17, 2008 21:25 UTC (Thu) by pr1268 (subscriber, #24648) [Link]

> More seriously, why does runlevel 4 exist? Somebody must have used it for something somewhere, but no distro that I can recall does.

Slackware uses runlevel 4 as the X11/GUI runlevel (equivalent to runlevel 5 in most other distros). Slackware's runlevel 5 is not used by default but is configured identically as runlevel 3.

Disclaimer: I'm a devout Slackware user (for more than three years, now). I actually use runlevel 5 as my default GUI run level by way of starting kdm via rc.local script.

And, no, I don't defensively ask others what they don't like about .tgz archives (even though I still think sbergman27's post is funny). Besides, we're all Linux enthusiasts, no matter which distro we prefer!

Reordering the boot for fun and profit

Posted Jan 17, 2008 17:07 UTC (Thu) by allesfresser (subscriber, #216) [Link]

As I remember it, the theory is that the runlevels progress in capability (sort of). So level 0 was originally supposed to be a "drop to firmware prompt" since most machines didn't have the ability to power themselves off, level 1 was an single-terminal admin-only maintenance mode, level 2 was a basic multiuser level but perhaps without network, level 3 was multiuser with network but no daemons (NFS, etc.)... you get the picture. I guess level 6 got stuck on at a later time, when machines got "auto-restart" capability or something, rather than dropping down to the firmware prompt.

I may be completely wrong. But that's how I remember it being explained to me.

There's a lot of different ways to arrange the runlevels and what services are run when: see the Wikipedia entry for lots of info.

Reordering the boot for fun and profit

Posted Jan 18, 2008 1:28 UTC (Fri) by Ross (subscriber, #4065) [Link]

I like parallel startup.  BUT!  It is nicer only if the scripts are carefully written.

I have a neat situation on my desktop where I can't log into GDM too quickly after turning it
on, or I cause automount to die (because LDAP lookups fail, because some other network service
isn't up).

It's pretty irritating.

Parallel?

Posted Jan 17, 2008 18:30 UTC (Thu) by vonbrand (subscriber, #4458) [Link]

Has been tried, makes almost no difference (they mostly wait for data to arrive from the disk). And the extra complexity is staggering...

This is something that is discussed to death each year or so.

Parallel?

Posted Jan 17, 2008 19:52 UTC (Thu) by Cato (subscriber, #7643) [Link]

This is not true - reports indicate that initng makes booting much faster due to parallel
execution, and Upstart has the same capability (though it must be configured explicitly to
execute scripts in parallel where possible - see
https://answers.launchpad.net/ubuntu/+question/4763 )

Parallel?

Posted Jan 17, 2008 21:27 UTC (Thu) by Duncan (guest, #6647) [Link]

Gentoo's OpenRC (now an independent project) has had most of this for 
years, altho the parallel stuff only really started working with the 
2.0-rcs -- which also sped things up dramatically as the core is C based 
now, not a shell script (tho shell scripts are still used for ease of 
reconfigurability if necessary for individual services).

Named levels?  Years ago, but still based on the traditional init so folks 
used to it can use that as well.  Here's the (edited) basics 
of /etc/inittab, complete with comments as appropriate:

# Default runlevel.
id:2:initdefault:

# System initialization, mount local filesystems, etc.
si::sysinit:/sbin/rc sysinit

# Further system initialization, brings up the boot runlevel.
# (The boot runlevel contains the really critical services
# that run even in single user mode.)
rc::bootwait:/sbin/rc boot

l0:0:wait:/sbin/rc shutdown
l1:S1:wait:/sbin/rc single
l2:2:wait:/sbin/rc nonetwork
l3:3:wait:/sbin/rc default
# init level 3 and 4 point to default as well,
# unless the admin wants to change them.
l6:6:wait:/sbin/rc reboot
# l9 is my own addition, sort of between single user
# and nonetwork, taking advantage of the fact that
# init can actually handle levels 7-9 too, tho they
# aren't there by default.
l9:9:wait:/sbin/rc basic

# This one is interesting.  Gentoo uses "a" of the a/b/c
# init options to setup xDM, only launching it however
# if it's in the appropriate runlevel
x:a:once:/etc/X11/startDM.sh


All the various level invocations do then is run the scripts in the 
appropriately named directories under /etc/runlevels.

Dependency based ordering?  Again, years ago.  Scripts have default 
dependencies, and as noted in /etc/conf.d/rc ...

# It's possible to define extra dependencies for services like so
#RC_CONFIG="/etc/foo"
#RC_NEED="openvpn"
#RC_USE="net.eth0"
#RC_AFTER="clock"
#RC_BEFORE="local"

OpenRC uses those dependencies to dynamically create a dependency graph 
and start them in the appropriate order.  This info is cached, with the 
files hashed and checked to see if they've changed and the dependency 
graph needs recalculated.

Parallel startup?  The option has been there for years but didn't work all 
that well until the 2.0 rcs, as mentioned above.  Tt works great now 
especially on multi-core/CPU machines loading off of kernel based md/RAID, 
since it can parallelize both the CPU time and disk access that way.  (I'm 
running a dual dual-core Opteron 290, 4-disk md/RAID-1/6/0, makes for a 
nice desktop/workstation with 8 gigs RAM!  Now /that's/ the hardware to 
run a Gentoo system!  With a tmpfs based compiling scratch area and using 
ccache, I can recompile all of KDE4-svn in roughly four hours, two if I do 
it daily -- without disturbing my work flow while I'm doing it! =8^)

Duncan