Booting Debian in 14 seconds (Debian-administration) [LWN.net]

Configuring a kernel with built-in drivers

Posted Nov 11, 2008 20:58 UTC (Tue) by rvfh (guest, #31018) [Link] (12 responses)

A few thoughts for laptop/desktop users:
- rather than recompiling a kernel with some modules built-in, I think it would be nice to see if there is not a way to link existing modules into the existing kernel binary. Something like file.ko -> file.o, then 'ld bzImage file.o'... Note that some things are better left as modules.
- it would be nice to have the SATA support built into the kernel anyway, thus not requiring an initrd in most cases, as most computers need it.
- USB gets initialised in its own time, and some initrds wait for it to be ready in case the root fs was on it. Getting rid of the initrd might get rid of this often useless delay

This being said, 2.6.27 solved the standby on my Amilo Xi 2428, and I only use that these days. Takes only a few seconds to be exactly as I'd left it. Suspend-to-disk was already reliable and I use that if I need to switch off the power. This is not the case of all computers though...

Configuring a kernel with built-in drivers

Posted Nov 11, 2008 22:07 UTC (Tue) by dlang (guest, #313) [Link] (4 responses)

why do you say that some things are better left as modules?

Reasons for preferring modules?

Posted Nov 12, 2008 8:27 UTC (Wed) by pjm (guest, #2080) [Link] (3 responses)

I don't speak for the original poster, but a few reasons I've heard are:

At least in the past, some things needed to be as modules to specify parameters. I don't recall the details, e.g. whether the parameters could reasonably be specified at compile time for a given computer. Maybe I/O base address or IRQ for a network card or something? Maybe no longer relevant.
Relatedly, I think one sometimes wants to have two copies of a module, with different parameters in each.
Useful for plug-and-play hardware (or filesystem or network protocol or whatever) that, who knows, the user might just one day use. Or for hardware (etc.) the user owns but rarely uses. Keeping it out of the static kernel slightly reduces memory usage & boot time. It may be only a small difference for a single module, but better nonetheless, and it adds up.
Relatedly, I think some drivers are harmful to have compiled in if that hardware isn't present.
For out-of-tree drivers, or drivers you're hacking on yourself.

Reasons for preferring modules?

Posted Nov 12, 2008 8:54 UTC (Wed) by rvfh (guest, #31018) [Link] (2 responses)

Another point I'd add is when the driver does not play well with suspend/hibernate, and thus needs to be remove before and re-added after.

Some drivers also have little issues which can be solved by removing then re-adding then, but I agree that it's a bug.

Last but not least, removing the module is sometimes the only way to put the corresponding device in low-power mode.

Reasons for preferring modules?

Posted Nov 12, 2008 11:00 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

most of the time there is a way to specify the parameters on the kernel command line (although the documentation on how to do this is very poor)

the ioports is a variation of the issue above (and very unlikly to be needed on modern hardware)

the suspend problem is valid

I didn't think that it was possible to load a module multiple times, are you sure this is possible?

for hardware that you don't have you don't want to compile the drivers in, but that's not what is being talked about here. what's being talked about is making it so that the kernel you boot has drivers compiled in for all the hardware in your machine. other things can be compiled as modules and used by the hotplug system as needed. the thing that we are trying to avoid is the overhead in the coldplug case.

for things that you are developing, or out-of-tree drivers you may not want to or be able to compile them in. but again, that's not the situation that is being addressed here, and compiling in the drivers for the hardware that your system _does_ have doesn't prevent you from compiling other drivers as modules.

all these articles have included having many modules available for USB devices, but it still makes a significant difference in the boot time of the systems.

Reasons for preferring modules?

Posted Nov 12, 2008 11:58 UTC (Wed) by baruchb (subscriber, #6054) [Link]

> most of the time there is a way to specify the parameters on the kernel
> command line (although the documentation on how to do this is very
> poor)

The first few paragraphs of the Documentation/kernel-parameters.txt document look quite clear to me.

Configuring a kernel with built-in drivers

Posted Nov 11, 2008 23:00 UTC (Tue) by endecotp (guest, #36428) [Link] (6 responses)

> rather than recompiling a kernel with some modules built-in, I think
> it would be nice to see if there is not a way to link existing modules
> into the existing kernel binary. Something like file.ko -> file.o, then
> 'ld bzImage file.o'

That sounds like an excellent idea. It would give a lot of flexibility to users of distribution kernels who would otherwise need to compile a kernel from source. Are there any experts reading this who know how practical it is?

Configuring a kernel with built-in drivers

Posted Nov 11, 2008 23:16 UTC (Tue) by dlang (guest, #313) [Link] (4 responses)

there are three advantages to compiling the drivers into the kernel instead of as modules

1. no time wasted probing to see if the module should be loaded (especially for the 90% of the modules that end up not being loaded)

2. things that the kernel can be compiled to do because it knows everything is there (for example async scsi inititalization)

3. optimizations the compiler can make on non-modular code (for example, modules waste 1/2 page or ram per module, all calls to module code are more expensive 'far' calls as the compiler can't know where in memory the module code will be loaded)

pre-linking seperatly compiled modules may save some time for #1, but won't help #2 and #3. and even for #1 you could probably achieve very similar results by just deleting the unneeded modules

Bzzzzt!

Posted Nov 12, 2008 12:28 UTC (Wed) by AnswerGuy (guest, #1256) [Link] (3 responses)

I'm pretty sure that every one of your three points is pretty bogus.

I know for sure #1 is based on an invalid assumption for a large number of cases. Many drivers (particularly for things like USB, fibre channel, firewire, etc.) don't perform any probing during kernel initialization. They probe for device presence when there is an attempt to access the device (or, in some cases, for a hotplug event).

I would suspect that this is true of most of the drivers that one would see loaded via kmod. When user space opens a device (or invokes the entry points for binfmt or network protocols, etc) then kmod dispatches a copy of modprobe to load the module and any probing and device initialization must happen at that time. However, even if that module is statically linked into the kernel the probing and initialization is (usually? always?) deferred until the first access.

No attempt to access means no time wasted on such initialization.

I also strongly suspect that #3 is bogus because it's not at all clear how often any statically linked module's exported functionality could be accessed by any means other than the same calling conventions that have already been established. I think it's going to be a far call dispatched through a table regardless of whether the code is statically or dynamically linked. (It's not like some of the optimizations you might get from different types of linking for user space code). In any event I also rather strongly suspect that the difference between the calling types (specifically for device drivers and other bits of functionality that can be modules) isn't going to make any appreciable difference in overall performance. (Most of the time spent in any sort of I/O is time that the CPU was waiting for DMA from the device(s), for example. Any calling and stack handling overhead should be down well under the 1% (line noise) threshold).

In any event you'd have to show me measurements and a detailed description of your methodology before I'd give these claims any credence.

Bzzzzt!

Posted Nov 12, 2008 14:04 UTC (Wed) by endecotp (guest, #36428) [Link]

> When user space opens a device
> then kmod dispatches a copy of modprobe to load the module and any
> probing and device initialization must happen at that time

I can't think of any hardware that I have that works this way. Everything seems to load the necessary modules during hotpluging or coldpluging. Can you give some examples?

Bzzzzt!

Posted Nov 12, 2008 16:00 UTC (Wed) by willy (subscriber, #9762) [Link] (1 responses)

> I know for sure #1 is based on an invalid assumption for a large number
> of cases. Many drivers (particularly for things like USB, fibre channel,
> firewire, etc.) don't perform any probing during kernel initialization.
> They probe for device presence when there is an attempt to access the
> device (or, in some cases, for a hotplug event).

I'm sorry, you're wrong. Drivers that can optionally be modules mark their entry point with module_init(). If compiled as a module, this code is executed at module load time; if it's built it, it's executed as part of do_initcalls() (see init/main.c in the kernel source tree).

You may be confused by some of the asynchronous discovery code; the scsi core spawns a thread for each host and does the lengthy device discovery in that thread. This cuts down boot time substantially on machines with lots of scsi (including fibrechannel).

Claiming that drivers only check for device presence when there's an attempt to access the device is just wrong, though. Sorry.

devfs is dead

Posted Nov 12, 2008 16:05 UTC (Wed) by rvfh (guest, #31018) [Link]

> Claiming that drivers only check for device presence when there's an attempt to access the device is just wrong, though.

Indeed.

I seem to remember that the devfs system provided module loading when the device was accessed, but that's been dead for a while now. Maybe the poster thought it had been preserved?

Configuring a kernel with built-in drivers

Posted Nov 12, 2008 13:50 UTC (Wed) by njh (subscriber, #4425) [Link]

FreeBSD's /boot/loader program can load modules and link them to the
kernel before booting the kernel - this is often used in the FreeBSD
world to make boot-critical device drivers or filesystems available
to the GENERIC kernel (addressing the same sort of problem for which a
modular Linux installation would typically use an initrd).

Doing something similar in Linux would presumably involve teaching GRUB
about Linux modules. I notice that GRUB actually has a "module" command
for loading modules when it is working with an OS kernel that uses the
Multiboot Specification (such as Hurd) ...

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 11, 2008 21:08 UTC (Tue) by mjthayer (guest, #39183) [Link] (8 responses)

A lot of the improvements done to improve boot time seem to involve customising the boot for the particular machine. I look forward to seeing some focus on improvements that don't sacrifice portability, as in the ability to move a partition over to a different physical machine without reconfiguring it. It would also be nice to see more focus on starting services on demand, possibly with some profiling, so that services a particular user uses a lot are autostarted. And perhaps it would be possible to supply a few different kernels optimised for different use cases (e.g. with SATA compiled in as the last poster suggested), while leaving a fallback kernel available from grub.

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 11, 2008 22:56 UTC (Tue) by endecotp (guest, #36428) [Link] (7 responses)

Some of what I propose in the article involves _automatically_ customising the boot for the particular machine, hopefully giving you the best of both worlds. Specifically, I proposed that a static /dev could be built by first booting conventionally so that /dev is filled by udevd, and then archiving those /dev nodes for subsequent udevd-free boots. Also, I proposed that an all-modular kernel could be booted once, and then lsmod used to determine which modules need to be built in, automatically creating a kernel that does so.

As for improvements that don't sacrifice portability, I agree; for example, it would be interesting to study why udevd seems to take so long at boot. Is it the shear number of modprobe and mknod calls that it makes? Or does modprobe constantly have to re-load configuration files every time it is started?

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 7:13 UTC (Wed) by aleXXX (subscriber, #2742) [Link] (6 responses)

I like the idea of not doing hardware detection every time. In most cases
hardware configuration hasn't changed so there should be some way to
reuse the previous results.
Maybe some boot parameter could be used which would force automatic
detection again ? (so you can select "Boot with hardware detection" from
grub when you changed something in your machine)

Alex

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 8:22 UTC (Wed) by mjthayer (guest, #39183) [Link] (4 responses)

Or a kernel option for only falling back to the initrd if the root filesystem fails to mount (this occurred to me some time after I wrote my first post).

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 9:02 UTC (Wed) by rvfh (guest, #31018) [Link] (3 responses)

I suppose this could easily be made automatic:

if (last boot ok) [Ubuntu have apparently introduced such a flag now]
  try usual boot
  if (fails)
    flag last boot no ok
    reboot
else
  try default boot [with initrd, whatever]
  if (fails)
    Houston, we have a problem.

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 9:22 UTC (Wed) by mjthayer (guest, #39183) [Link] (2 responses)

Not quite as good, since that only works after a failed boot. Automatic fallback to the initrd if the root filesystem couldn't be mounted would save that.

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 10:21 UTC (Wed) by rvfh (guest, #31018) [Link] (1 responses)

You're right, and my stuff won't work because it requires access to the grub dir, which requires a running system...

Maybe we could change the kernel to ignore the initrd if root as specified in the root= argument is accessible? That would require a new kernel command line option though, like initrd=?, and to make sure that what the initrd used to do is also done without it.

How much time does it actually save to skip the initrd BTW? Is it worth it?

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 10:31 UTC (Wed) by mjthayer (guest, #39183) [Link]

If I understood correctly, one of the reasons that the initrd takes time to load because it has to be able to handle lots of different situations regarding the boot device and the root filesystem. So it contains a sort of mini-distribution on a ramdisk, including shell and whatnot. I presume that loading the shell, initrd scripts and things are what takes the time. Skipping the initrd seems to be one of the main tricks used to speed up boot time in all these experiments.

Booting Debian in 14 seconds (Debian-administration)

Posted Nov 12, 2008 9:06 UTC (Wed) by endecotp (guest, #36428) [Link]

Someone suggested taking a checksum of the PCI and other device IDs. That should be quick.

Plus BIOS time

Posted Nov 12, 2008 9:36 UTC (Wed) by pjm (guest, #2080) [Link]

Note that both this and the previous “five second boot” are actually measured from when Linux loads, not from power on. According to the article, that adds about another 10 seconds (or 4 seconds when merely rebooting).

That can be greatly reduced by switching to CoreBoot (né LinuxBIOS). Google search for ‘coreboot eee’ for some info on progress on this for Eee machines, or choose hardware already supported by CoreBoot.