LWN: Comments on "LPC: Booting Linux in five seconds"

LPC: Booting Linux in five seconds

lkundrak — Mon, 02 Feb 2009 07:09:41 +0000

Check out the alpha version of Moblin distribution, it incorporates the features mentioned in this article.

LPC: Booting Linux in five seconds

tholme — Wed, 05 Nov 2008 21:10:20 +0000

He got rid of the initrd and made som changes to the kernel. You can find them in his fastboot.git-tree

LPC: Booting Linux in five seconds

skeldoy — Mon, 20 Oct 2008 11:04:01 +0000

I got a custom OS built on the latest vanilla kernel and busybox to boot in 10 seconds without changing any code - it hardly does anything - I just wondered where you get you 5 extra seconds from. It has to be the kernel - my initrd and x-startup (all parallel except network) takes less than a second, so all the time that is spent is in the kernel.

I just wondered what kind of stuff you did to the kernel to make it boot faster. Is there some smart stuff we can do to tweak it? Are some of the options real time consuming? Can you post a link to the .config?

I can start my 1978 Ford Pinto in 4 seconds

pgan — Wed, 08 Oct 2008 03:31:34 +0000

Well, using analogies is like two eggs sunny-side-up, flying through space.

Log-in screen

pgan — Tue, 07 Oct 2008 22:41:46 +0000

The log-in screen should be shown as early as possible (before even X) and should not block the boot process. Thus the computer would be usable earlier, because the boot process would continue while the user is typing her password. This is especially important for users who are slow to authenticate -- some users take about 4 seconds. With a normal boot, they would wait n seconds to see the log-in screen, spend m extra seconds logging in, then wait d seconds till they can use the desktop. If they can authenticate after x second while the computer is booting, they only wait max(n - x, m) + d seconds to use the computer.

With a long boot, this also allows users to authenticate themselves whenever is convenient for them, rather than waiting to do it at the right time for the machine.

And, knowing which user is logging in will allow further parallelization of loading the user's desktop components and services. That will reduce the time "d" required to start up the desktop.

Cheating: Not just for Microsoft anymore

Janne — Tue, 07 Oct 2008 12:59:23 +0000

"why does she have to wait 45 seconds for the OS to boot?"

She doesn't. I mean, you can apparently boot Linux in 5 seconds ;).

"In short, why is using a computer with a 1.83GHz Core 2 Duo and 4GB of RAM
full of delays, when using an IPTV set top box with typically 100MHz CPU
and 32MB RAM is not?"

Well, that set-top box is designed to do one thing, and it's designed to do
it on one hardware-configuration. So they can optimize it like crazy. Those
do not apple to Linux.

that said, my DVR takes about 5-10 seconds to boot. My DVD-player takes few
seconds as well. My router takes about 10 seconds to become reachable.

"If you allow delayed service startup to not count, the danger is that
you'll discount services that matter to users."

But the thing is that we already have slow as hell boots, even with delayed
services. Like I said, my laptop running OpenSUSE takes ages to boot, and
when I DO get to the desktop, it's STILL not connected to the network! So
the difference between this fast booting and my OpenSUSE is that Arjans
system takes 5 seconds to boot, after which it takes few more seconds to
connect to the network, whereas my laptop takes 45-50 seconds to boot,
after which it takes few more seconds for it to connect to the network.

In other words: it takes just as long for the two systems to connect to the
network. The difference is that the things that happen before that net-
connection take 45-50 seconds on my laptop, whereas on Arjans EEE it only
takes 5 seconds.

And how would you connect to the WiFi befire getting to the desktop? I
mean, you might need to enter passwords and the like? Should the computer
stop booting and prompt you for your WPA-passowrd? No. It should get you to
the desktop and in to an usable state, and prompt you for any needed WiFi-
passwords as needed.

"Indeed, you're already trying to handwave away an important service for
some use cases as "takes too long, and anyway users can wait"; this is
exactly why someone doing the challenge must set a defined state in which
boot is finished. Given the defined state, I can now look at it, compare it
to my use cases, and say "yes, that's good enough", or "no, I need to work
out how to fit WiFi startup into that 5 seconds"."

But Wifi is not part of what we usually think of when we talk about
"booting". And like I said: I see no difference between this 5-seconds boot
when compared to normal booting, if we think of Wifi alone. In either case,
WiFi is disconnected at the end of the boot.

"Plus, my experience of normal users is very different to yours - they
don't have computers waiting and switched on, they don't start the computer
up without a task in mind, they start the computer thinking "Do I have any
e-mail waiting?""

Sure. But let's compare two scenarios:

Normal distro:

User turns the computer on. It boots for about 45 seconds, and the user
logs in. After he gets to the desktop, he needs to wait for few seconds for
the network to become usable. Then he can check his mail

Arjans EEE:

User turns on the computer. It boots in five seconds. User needs to wait
for few more seconds for network to become usable. Then he can check his
email.

How is that normal distro handling networking better than the EEE is? It's
not online either, at the end of the boot.

removing WiFi from the equation we can focus on the subject at hand:
booting. WiFi relaies on other things that are totally outside the scope of
this project.

I can start my 1978 Ford Pinto in 4 seconds

nix — Mon, 06 Oct 2008 20:46:16 +0000

Thank you for completing this discussion by providing the obligatory
Stupid Car Analogy(TM). (It is well known that all computing discussions
must eventually include a stupid and inappropriate analogy to the
automobile.)

I can start my 1978 Ford Pinto in 4 seconds

gumby — Mon, 06 Oct 2008 18:32:24 +0000

Providing its not to cold, I can turn ignition to start, fire up the cylinders put car in gear put foot on the gas, let clutch out in 4 seconds on a good day.

Why are computers so slow at starting, maybe it takes them a while to warm up and get those electrons flowing.

LinuxBIOS / CoreBoot

endecotp — Mon, 06 Oct 2008 10:58:49 +0000

How much more time could we save by booting using CoreBoot (aka LinuxBIOS) instead of the manufacturer's supplied BIOS ROM?

Cheating: Not just for Microsoft anymore

pboddie — Sat, 04 Oct 2008 18:36:10 +0000

Well in the windows camp, for example, you can't even successfully operate the start menu.

Indeed. What's the point of showing the user the desktop if they can't actually use it? That's the question the Windows developers should be asking themselves.

Cheating: Not just for Microsoft anymore

farnz — Sat, 04 Oct 2008 11:11:33 +0000

Then you're missing the entire point of the exercise - why does she have to wait 45 seconds for the OS to boot? Why should she be waiting up to a minute for things to be ready to use? Why can't she turn the computer on and use it immediately?In short, why is using a computer with a 1.83GHz Core 2 Duo and 4GB of RAM full of delays, when using an IPTV set top box with typically 100MHz CPU and 32MB RAM is not? Granted, the tasks are different, but the computer is so much more powerful than the STB that it's a fair question.

The idea Arjan and Auke were working to is to set a hard challenge - a typical Linux distro takes 30 seconds to get to the same point, they chose 5 seconds - and remove all the obstacles. If you allow delayed service startup to not count, the danger is that you'll discount services that matter to users. Indeed, you're already trying to handwave away an important service for some use cases as "takes too long, and anyway users can wait"; this is exactly why someone doing the challenge must set a defined state in which boot is finished. Given the defined state, I can now look at it, compare it to my use cases, and say "yes, that's good enough", or "no, I need to work out how to fit WiFi startup into that 5 seconds".

Again, as I've said several times, you can go from "5 seconds power applied to idle and ready" to "3 seconds power applied to usable, 3 more seconds to idle", and that's a much easier task than going from "60 seconds power applied to idle" to "3 seconds power applied to usable, 120 more seconds to idle.

Plus, my experience of normal users is very different to yours - they don't have computers waiting and switched on, they don't start the computer up without a task in mind, they start the computer thinking "Do I have any e-mail waiting?", "Can I look up a recipe to use lemon sole, potatoes and tomatoes?", "What do I need to say in this letter to my bank manager?", "I need to get that proof printed for marking up" and things of that nature. To them, a computer is just a tool; you wouldn't get out your woodworking kit without something to build in mind, so why start up a computer without a job in mind?

Cheating: Not just for Microsoft anymore

Janne — Fri, 03 Oct 2008 19:37:44 +0000

In your example your wife would have to wait for few second for the network to become functional. Im sorry, but I don't really see that as a big issue. Especially considering the huge benefits users would get in "normal" use-cases.

You are describing a situation where the user is twitching to do something with the computer. Where the user even pre-positions the mouse in order to be able to do stuff as fast as possible. Those cases are very, very rare. Normal users do NOT work like that. Normally people don't start their computers thinking "OMG, I need to get online NOW!". And even if they did, the fact that the system boots in 5 seconds, means that they could get online a lot faster than with "normal" boot-sequence.

That said, I don't really understand the complaing about networking. My OpenSUSE-laptop takes ages to boot. And after it finally get to the desktop, it STILL spends few seconds connecting to the network! It would be A LOT better if it booted in 5 seconds, and then spend few seconds connecting to the network.

It makes sense to exclude networking from this experiment, since connecting to the network is not really part of what we call "booting". It's not up to your computer, it's basically being held back by the network. And hacking the boot-sequence of your computer does not touch your network in any shape or form. Network is completely outside the scope of this test. And just because you do not have working networking for few seconds does not mean that the computer is not usable. And still, the "normal" distros are just as slow as far as networking is concerned, so there are no drawbacks in this experiment, as far as networking is concerned.

So how would your wife benefit if the recipe-machine was using Ubuntu (for example)? Instead of booting in 5 seconds, the machine would boot in 45 seconds. Even if you had fully working networking after that 45 seconds (you wouldn't), it would still take about 30 seconds longer to get online than with this fast booting.

"Secondly, note that Firefox's start up will be slowed down if the computer is also trying to bring up the network"

Starting FIrefox taxes the HD and the CPU. Neither of those are taxed when your NIC is getting an IP-address.

"start printing services"

That could be started when the user actually wants to print. Why should we sit around, twiddling our thumbs, when the system starts printing-subsystem, even though the times we print are few and far between?

Cheating: Not just for Microsoft anymore

farnz — Fri, 03 Oct 2008 19:07:32 +0000

You're missing a lot in your example timeline; let me rework it the way it would actually happen:

My wife is in the kitchen, and finds that she's got ingredients she'd like to use for supper, but she'd like to use them for something different to her normal cooking style.
She goes into the living room, and fires up my laptop.
She wait, knowing that as soon as it's running, she'll want to log in.
System boots. She logs in.
She waits, knowing that as soon as it's logged in, she'll want to run Firefox.
She clicks on the Firefox icon on her desktop.
She waits, knowing that Firefox will come up maximised. Her mouse is hovering over where the bookmark for the recipe site she usually uses (when on my laptop, not hers - she has a much more complex set up on her own machine) will be.
Firefox starts enough that she can click on that bookmark. If the network is not running by this point, things are broken for her, as she can't go to the ingredient search and work out she can cook with the contents of our fridge.
She finds a recipe she'd like to try, and prints it - she likes to make notes on paper of changes she makes to a recipe while cooking, so that she can remember them easily later.

Looking at that timeline, one important thing is clear - she has already decided to go to a particular bookmarked website before she touches the power button. All the time taken between her pushing the power button and the website displaying is wasted by the computer. Thus, the only time the OS has free to waste is the time taken by Firefox to start up after the OS has started.

Secondly, note that Firefox's start up will be slowed down if the computer is also trying to bring up the network, start printing services and generally do other things at the same time. It's incredibly hard to accurately judge how much idle time you have around application start time.

Finally, in my experience, services are not delayed after thinking about use cases. Instead, the delay is based on getting the bare minimum up and running to let me log in, with functionality taking a while to come up after that. Making the challenge "get to idle in 5 seconds" leaves room for someone to come in later, and build on that work to get a "get to usable in 4 seconds, idle in 6", or even a "get to usable in under a second, but idle takes 6 seconds".

Cheating: Not just for Microsoft anymore

Janne — Fri, 03 Oct 2008 17:25:27 +0000

I really don't understand this argument. So the machine has not really booted, since network (for example) is not up yet? Well, why should it be? Because you want to use the network right after you get to the desktop? Well, think about it: You get to the desktop, but the network is not yet running. You then decide to launch Firefox. It takes you maybe 2-4 seconds to click the Firefox-icon, it takes anout 2-4 seconds for Firefox to launch, and then it takes you another 2-4 seconds to type in an URL in Firefox. That means that the system has 6-12 additional seconds to bring the network up before you actually NEED to have the network up&running. Network might not be running right after you get to the desktop, but it does not need to be, since it would take you several seconds to actually DO something with the network in any case! Do you need to have networking up if you are not actually using the network for anything?

People who complain about services that are not running right after boot, fail to understand the fact that it takes several seconds for the user to actually DO anything with the system in any case. And that means that the system has quite a bit of time to bring any missing service up before the user actually needs it. Whenever I log in to a machine, I do not launch apps in a split-second or start frantically working right away. No, it takes me quite a few seconds to get up to speed. And the system can use those seconds to load any needed services that are not yet up.

Of course there are wrong ways to do this, like what MS does with XP, where the system is actually un-usable for longish time after you get to the desktop.

Are the .config available

ummmwhat — Fri, 03 Oct 2008 13:32:24 +0000

Hi, I just want to know if you can post the .config of your presentation's kernel.

Thanks in advance.

Best regards...

message output

reed — Fri, 03 Oct 2008 12:48:12 +0000

I noticed from your video that there is a minimum of message printouts on the console. Output takes a lot of time, especially when there's such a flood of it, and it's easy to forget about.

LPC: Booting Linux in five seconds

pabs — Fri, 03 Oct 2008 11:03:39 +0000

A better way to do that is:

apt-get install debian-goodies ; checkrestart -v

That gives you package names and init scripts to run.

Cheating: Not just for Microsoft anymore

dgm — Fri, 03 Oct 2008 07:50:31 +0000

> only the people who use a feature should pay the price for it in terms of boot time

In fact, this should be applicable to any other system resource: CPU, memory and disk usage.

LPC: Booting Linux in five seconds

jordoex — Fri, 03 Oct 2008 07:44:01 +0000

Looks like Ubuntu has some work to do for Jaunty Jackalope if they want to impress anybody now!

LPC: Booting Linux in five seconds

jwmittag — Thu, 02 Oct 2008 12:13:16 +0000

Some time ago there was this MachBoot Linux Live-CD (read "Mach" as in "Supersonic Speed", not as in "Research Microkernel obsoleted 20 years ago and only kept alive by Apple and Hurd"). Their current record is 5.68s, which is pretty darn fast considering that CD-ROMs are both slow and have absolutely abysmal seek times.

Interesting times …

LPC: Booting Linux in five seconds

renox — Thu, 02 Oct 2008 08:36:12 +0000

Agreed, I understand the desire to do 'booting in 5s' without cheating, but CUPS is probably one exception here where cheating is okay:
1) it's unlikely to take much CPU or IO load (if it does then it's a bug) so starting it in the background won't be felt as a slowdown for the user.

2) users are unlikely to print just after starting the computer so launching the desktop before CUPS is okay.

3) the critical part for the user here is the 'time to print' if having to find a network printer really slow downs printing then it make sense to have CUPS running all the time, not on demand.

LPC: Booting Linux in five seconds

salimma — Tue, 30 Sep 2008 21:27:25 +0000

And given that you don't really hear a chorus of start-up chimes from business travelers' laptops once the electronics gag has been lifted, that's obviously what they've been doing all along: suspending.

Nobody fully turns off their PDAs too, as far as I can tell. Though you hope they do remember to put them to in-flight mode...

LPC: Booting Linux in five seconds

syntropy — Tue, 30 Sep 2008 16:37:02 +0000

I have been booting in a rough eight seconds (GRUB->X) for months without the problematic issues of readahead and deferring startup to the 'background'. My system is loaded and synchronized in 8 seconds flat, without tearing out subsystems and destroying so many internals and hacks on Fedora.

Kyuba.

LPC: Booting Linux in five seconds

keybuk — Sat, 27 Sep 2008 15:27:00 +0000

Although according to your boot chart, you're not starting xinetd either ;)

Starting CUPS on demand isn't quite ideal sadly, if the user is using
network printers, then CUPS needs to have already been running for about
30s before they try and print otherwise the network printer won't be shown.

Sadly we can't "start CUPS 30s before the user needs it" ;)

Though I entirely agree in principle that CUPS is not part of the critical
path of the boot sequence, if the system is idle at some point, it's worth
starting -- even if that means the disk isn't entirely inactive after boot.

The most ideal way to deal with these kinds of services is to start them
after boot, when the system is idle - with low CPU and I/O scheduler
priority so that user activity takes precedence - or activate them on
demand by xinetd or D-Bus when the user actually needs them.

That way, if the user clicks print and cups wasn't running yet, it gets
brought up -- or if the user leaves their PC alone for a bit, we start cups
for the next time they click print.

Of course, even better would be if we could do network printer discovery
via other means like Avahi -- but then Avahi falls into the exact same
"start on idle or first demand, whichever is sooner" category

Cheating: Not just for Microsoft anymore

keybuk — Sat, 27 Sep 2008 15:22:22 +0000

Actually, it isn't true.

If you look at the Ubuntu boot sequence, while gdm isn't the last thing
that we start, it's not far off. Very few things are started after gdm,
and this is mostly because X takes quite a while to start so we have some
"free time" here.

We deliberately don't keep starting services throughout login, because as
you note, I do all my timings until full desktop login which is when the
computer is actually usable - not just to when the gdm screen is up.

When is "save" really "maybe save"?

njs — Fri, 26 Sep 2008 21:43:57 +0000

fsync doesn't mean "hey kernel actually write this to disk", it means "write this to disk *now*" (and in practice, "write everything to disk *now*", because our filesystems are not that great). If you're running it async, then you get exactly the same semantics as a plain write. The kernel already guarantees that stuff will be written to disk within n seconds, with n a tweakable parameter that's important for power saving. I guess I just don't see the advantages of moving that to a million per-app settings, all using an inappropriate interface.

If the goal is to reach a point where we can throw away a lot of data instead of flushing it to disk at shutdown, then this approach is making a classic mistake: it's trying to mark everything that *does* need to go to disk, and hoping that eventually everything will be marked and we'll be able to flip the switch and throw everything else away. The better approach is to mark stuff that isn't important, like some fcntl to request "power-loss semantics" for writes to some file; then you could get some win immediately, and expand it incrementally over time.

I doubt this is easy or important enough to actually get the coordinated effort needed to implement it, but that's how I'd do it...

LPC: Booting Linux in five seconds

bronson — Fri, 26 Sep 2008 17:09:12 +0000

Because -- in theory -- you wouldn't have to restore every application, (don't worry about freezing the memory used by all those useless panel apps). And the apps themselves could be smarter (Firefox wouldn't try to persist its in-memory caches). Crash-only apps could just be terminated and re-launched (IM clients, mail clients, etc), no need to write them all out.

I admit, it's all speculation at this point. :)

When is "save" really "maybe save"?

dmarti — Fri, 26 Sep 2008 15:05:34 +0000

Applications could be smart about this. The answer might be something like: fsync on save if 100 user actions or 10s of CPU time have been spent on the file since the last save. Or fsync on save if the file has gone from a broken state (invalid HTML, spelling errors, audio that clips, program that won't compile) to a fixed state.

(And you could always do the fsync in a separate thread or process, so the app is responsive again.)

They are

dmarti — Fri, 26 Sep 2008 13:42:36 +0000

Actually, this stuff does look like it's on the way to becoming "real." Ted Ts'o and Keith Packard (filesystem and X) were in the crowd, and both sounded interested in accepting this work, or something based on it, into the "real" versions of the software. The actual readahead program is supposed to be on the Moblin site soon -- so Fedora, Ubuntu and the rest can download and integrate it if they want to.

Now what about shutdown?

njs — Fri, 26 Sep 2008 02:24:50 +0000

Uh... so if I hit "save" while using my app, that app should always fsync before returning? That sounds awful. (I guess the alternative is that we just throw away the files that users thought they had saved, but that seems suboptimal.)

IIRC emacs (used to?) do this by default, and until I disabled that it was unusable on a laptop, because hitting C-x C-s blocked everything for a few seconds waiting for the drive to spin up. ELISP SMASH

LPC: Booting Linux in five seconds

njs — Fri, 26 Sep 2008 02:12:14 +0000

I thought restore from hibernate spent most of its time reading the (potentially multi-gigabyte) memory image from disk. Booting doesn't need to load 500 megabytes of firefox into memory, etc.; is there some reason to think that restoring your individual apps would be faster than restoring the hibernate memory image?

LPC: Booting Linux in five seconds

Hawke — Fri, 26 Sep 2008 00:41:15 +0000

So why aren't these optimizations being done "for real", with the intent of being packaged for normal distros? It's great to know that in theory it can be done, but it's of little benefit if no one actually does it...

Again: arjan, ....

hummassa — Thu, 25 Sep 2008 17:40:46 +0000

us want linky linky :-) so us can tune our eee701 to boot in 5 seconds too...
:-)

LPC: Booting Linux in five seconds

kamil — Thu, 25 Sep 2008 16:18:36 +0000

My understanding is that the logic behind turning off your laptop during the take off/landing is not about electronic interference, but more about you being alert to what's going on on board. Also keep in mind that they don't just ask you to turn them off, but also to stow them. They simply don't want anything to obstruct the escape routes in case of an emergency, including any wires around your head.

What I'm trying to say here is that in practice, suspending your laptop during take off/landing as opposed to turning it off is perfectly fine. Your average flight attendant will likely flatly deny that when asked, but those people are not paid to think...

LPC: Booting Linux in five seconds

msmeissn — Thu, 25 Sep 2008 15:39:15 +0000

CUPS does printer autodiscovery (zeroconf style, but on other ports), so
some people want it running to find printers. :/

And it manages local queues occasionaly, which was one reason I heard to
have it run all the time. (Not sure if this applies.)

Ciao, Marcus

Now what about shutdown?

dmarti — Thu, 25 Sep 2008 15:13:44 +0000

If the app really needed that data it would have called fsync a while ago -- so just halt. Applications are going to have to be able to handle coming back after a kernel panic or a kicked-out power cord anyway.

LPC: Booting Linux in five seconds

BenHutchings — Thu, 25 Sep 2008 15:01:44 +0000

I don't know which servers you're looking at, but I generally find them to be relatively slow to boot. Whereas Microsoft has made fast booting (from the BIOS to the Windows loader) a requirement for putting the Windows logo on desktops and laptops.

Other fun features for "crashable" apps

dmarti — Thu, 25 Sep 2008 14:59:51 +0000

Programs that come back in a messed-up state after a crash make baby Jesus cry. The vi clone "elvis" had good recovery very early, and we have enough disk space now to do it even for very large media files.

A user actions log that an app could replay might have other uses, too. Of course there's deep undo and bug reporting, and automatic macro writing by identifying common steps might be useful. There was even a legal case a few years ago where a composer couldn't prove he had created a certain audio file, because he couldn't put the sliders of his GUI audio app on the exact right pixel.

LPC: Booting Linux in five seconds

arjan — Thu, 25 Sep 2008 14:56:09 +0000

It's from the start of the desktop environment (in our case XFCE) until it's done (in the "screen painted, cpu and disk idle" sense).

Yes 2 seconds is very generous and more minimalistic desktops can go much faster than that. We decided to pick XFCE since, well, first of all, we could hack the code as kernel guys, and second, it has a feature set that at least is close to what the "big guys" do (which code we didn't feel we could hack as kernel guys).

Can XFCE go faster than 2 seconds? probably. We met our budget for it after one small patch so we stopped working on it and focused on other parts instead.

This may sound weird but our project really was about booting in 5 seconds, not about booting fast or fastest ;-).... we engineered it budget driven, which really helped us to spend our time effectively, specifically, on those parts of the boot process that were over budget, rather than getting bogged down in one part and spend 3 seeks making it go from 2.0 to 1.9 seconds.

LPC: Booting Linux in five seconds

arjan — Thu, 25 Sep 2008 14:39:02 +0000

With multi-core or hyperthreading it is worth doing some things asynchronous.
But it's a subtle but important difference between doing things in parallel and doing some selected things asynchronous.

The key thing is to do the "critical path" of the boot sequential and as tightly packed as possible, while doing non-critical (to the boot) things asynchronous.