Weekly Edition
Return to the Front page
| |||||||||||||
Embedded Linux: Small Root Filesystems
In the first part of this
series I looked at the TinyLinux project, a set
of patches aimed at helping developers reduce both the size of their kernel
images and the amount of memory they use at runtime. But if you're working
to build a working small system you'll need more than a kernel. You need
something for the kernel to manage. You need user space applications and an
environment in which they can run on top of your tiny kernel. And like
your tiny kernel, you need your environment and applications to be as small
as possible.
The Root FilesystemThe Linux kernel works hand in hand with what is called the root filesystem. This is the filesystem upon which the root directory can be mounted and which contains the files necessary to bring the system to a state where other filesystems can be mounted and user space daemons and applications started. Most desktop and server distributions make use of two kinds of root filesystems: the initial root filesystem and the real root filesystem. The former is used to mount and run the latter. The directory structure for a root filesystem can be extremely minimal, as we'll see in a moment, or it can contain the usual set of directories including /dev, /bin, /etc, and /sbin, among others that you see in any desktop Linux distribution. The kernel boot process concludes with the init code (see init/main.c) whose primary purpose is to create and populate an initial root filesystem with a set of directories and files. It then tries to launch the first user mode process to run an executable file found on this initial filesystem. This first process ("init") is always given process ID 1. There are three ways for the kernel to find the file that will be run by the init process. The first method is to use a file specified at boot time with the init= kernel parameter. If this parameter is not set, the kernel tries a series of locations to find a file named "init". These include /sbin/init, /etc/init, and /bin/init. If all these fail, the kernel tries to run any shell it finds at /bin/sh. If this last fallback is not found, the kernel will print an error saying that no init could be found. Once the init process is started it typically begins to launch other user space programs. On a desktop or server system this is known as the sysvinit process and includes the set of scripts found (typically) under /etc/rc.d. The name sysvinit comes from the mechanism used in System V Unix, which defined the naming scheme used for directories and files. On embedded and small footprint systems the init process may be a set of custom designed scripts or even a single application. Some desktop distributions are also beginning to replace sysvinit with alternatives designed with faster booting in mind.
The early root filesystem: Initial RamdisksThe initial root filesystem is known as the initial ramdisk because the filesystem lives in a disk image created by the kernel in RAM. In a desktop or server system, the initial ramdisk is used to load drivers and initialize an environment so that an external storage system (disk or network attached storage) can be mounted. The switch from the initial root filesystem to the real root filesystem is called a pivot. The pivot causes the real root filesystem to be mounted over the initial root filesystem. When that happens, a new init process from the real root filesystem is launched and takes over the process id of 1. At that point the initial ramdisk is no longer needed and the memory can be freed. In an embedded system the initial ramdisk might be the only filesystem ever mounted, since it contains all the user mode applications required. Alternatively, the initial ramdisk might mount a flash drive or other local storage yet still not pivot. Instead the mounted storage might be used as a source for user space applications. In these cases the initial ramdisk is never cleared because the pivot never happens. In the 2.4 kernel, the initial ramdisk was referred to as the initrd image. It was created as a filesystem inside a file. The file was mounted and a filesystem created inside of it. A directory structure for the initial ramdisk was copied into this filesystem. The file was then unmounted, compressed and provided to the kernel via the initrd= kernel parameter at boot time. The initrd file could only be loaded at boot time from an external source (except for the MIPS kernel, which allowed you to embed the image into the kernel). The original ram disk mechanism for the 2.4 kernel created a synthetic fixed sized block device that needed the filesystem driver used when the initrd was created, such as ext2, in order to work with file data. At the end of the boot process the initrd image had to be unmounted in order to clean up memory usage before switching to a more complete root filesystem. In the 2.6 kernel the process of creating and using the initial ramdisk has been somewhat simplified. First, the files are simply collected together in an compressed CPIO file, now referred to as the initramfs instead of initrd. The initramfs file is always embedded in the kernel (for all hardware platforms) even if you don't create one yourself. If you don't, a default CPIO archive is created automatically by the kernel build process. Second, there is no external filesystem required at boot time for the initramfs. Instead, the initramfs is unpacked in a special ramfs-based filesystem called the rootfs. The ramfs filesystem support is built into the kernel and cannot be disabled, so it's always available. Because it doesn't use backing store, it's a simpler system than the mechanism used in the 2.4 kernel. And when the boot process is done with the initramfs, a more complete root filesystem (such as one found on disk) can be directly mounted over it without worrying about wasting a lot of memory.
Why use the initramfs?It would be ideal if the kernel could boot into a minimal state that knew just enough to bring the system to a useful state for the user or environment it will run in. This minimal state would allow the kernel to be as small as possible with as few options compiled in as possible. This is exactly why you use an initramfs. On any system, and most especially on resource limited systems, you want to keep the kernel itself small and dynamically load only those driver modules that are required to make the system finish booting. Most desktop systems use the initramfs to determine what kind of hard drive or other storage is available with a complete root filesystem. In this case the initramfs contains boot scripts and driver modules relevant to bringing up the system. These files are only kept around temporarily while the real root filesystem is mounted and the real init process is started. Because the variety of desktop hardware is large, the initramfs can end up being large and fairly sophisticated as it tries to guess what kind of hardware is about to be mounted. On small systems the situation can be much different. There may not be any additional storage available to hold another, more complete root filesystem. In that case the initramfs becomes the real root filesystem. Because the initramfs is running out of RAM, it will contain only those files and directories absolutely necessary to run the system. Alternatively, a small system might use a dedicated flash drive with read only access to prevent accidental destruction of the bootable system. In that case the initramfs will contain boot scripts that mount the flash device and perform a little trickery to simulate writeable partitions so the system can operate normally.
Creating an initramfsIt's possible to recreate an initial ramdisk that mirrors your running desktop using the mkinitrd script. The problem with using this script is that you're recreating your desktop environment. That's not likely what you're looking for in your embedded system or even a live CD. So we need to look at creating the initramfs manually. The kernel source includes the text file ramfs-rootfs-initramfs.txt under Documentation/filesystems. In this file, under the section titled "Populating initramfs" are instructions for creating a very minimal initramfs. This includes a minimal set of device files, the /proc, /sys and /mnt directories, an init script and a BusyBox binary. We'll get to BusyBox in a moment. Start by creating a directory called "myinitfs":
mkdir myinitfs
Add some basic directories:
mkdir -p myinitfs/{boot,proc,sys,mnt,sbin,dev,lib,usr/bin}
Not all of these are required but you'll want them around to populate with useful tools in your initramfs anyway. Next, add the required device files. If your kernel and user space processes need to be able to output messages then the minimal root filesystem will need a console device. This is created with the mknod command.
mknod -m 644 myinitfs/dev/console c 5 1
If your system is booting from a CD and the root filesystem is in a compressed filesystem image on the CD then you'll also need a loop device.
mknod -m 644 myinitfs/dev/loop0 b 7 0
Of course, your embedded system doesn't have to output messages to a console and it certainly doesn't have to mount any filesystems, so neither of these are required. But if you're creating a live CD you'll want them. After creating the directory structure and adding these two devices, we copy in a shell script for our init program and a compiled copy of the BusyBox binary. The content of the shell script and the makeup of the BusyBox binary are the keys to getting your small system running.
Starting Small: BusyBoxBusyBox is the workhorse of embedded systems. It is a collection of commonly used Unix utilities rolled together into a single binary. The command line utilities usually have fewer options than their standalone counterparts but tend to be functionality similar. The primary goal of BusyBox is to provide a full featured set of utilities for resource limited systems. BusyBox is a well designed package that is extremely easy to use. A graphical configuration utility similar in style to the curses-based kernel configuration utility allows you to choose the utilities you need. The Unix utilities are referred to as applets and the configuration utility lets you pick which applets to include in the binary. The choice of which applets to include depends entirely on the system you're trying to create. For a live CD that mounts a compressed file system from the CD as the real root filesystem (over the initramfs) you would include utilities like losetup, mount and umount, gzip, and tar, along with the basic ls, ash, grep, mkdir, mknod and so forth. The build process for BusyBox is simple. Unpack the BusyBox archive in the current directory (where myinitfs is located). This creates a BusyBox directory. In that directory, create your configuration:
make menuconfig
You'll be prompted to save the configuration, which you should do. In the configuration you should be certain to specify the directory where the build should be installed. While not absolutely required, it saves a copying step later. In the latest version of BusyBox, 1.2.1, look under the BusyBox Settings->Installation Options menu and set the install directory to "../myinitfs". After configuration, you simply build and install the binary:
make
make install
Getting Bigger: LFSBefore looking at the init script I want to mention that, although BusyBox can provide just about everything you need to get the system booted and even provide a runtime environment on its own, you might need far more user space support. If you're looking to extend your system to a full distribution, be sure to look at the LinuxFromScratch.org web site, known more commonly as LFS. Here you'll find step by step instructions on how to build a complete distribution. The LFS is often used in live CD distributions as the runtime system that is loaded from a compressed filesystem off a CD by a BusyBox-based initramfs. Building a live CD from scratch in this manner is a great way to learn what a Linux distribution is all about, from the kernel on up through KDE and GNOME.
A live CD init scriptAt this point you've created a minimal set of utilities and a directory structure suitable for booting (sans the kernel, of course). But you still need the all important init script that kicks things off for the user space environment. I've worked with this init script for some time, which is based on the init script found in an older version of the LFS live CD. It assumes the use of UnionFS and SquashFS for mounting and using compressed filesystem image files from the CD. In my next article in this series I'll look at how and why you would use compressed filesystems like SquashFS along with UnionFS to boot your system. (Log in to post comments)
Embedded Linux: Small Root Filesystems Posted Nov 22, 2006 11:26 UTC (Wed) by nix (subscriber, #2304) [Link] One point that was perhaps not made strongly enough is that pivoting is an initrd thing *only*. initrd and initramfs are separate things and work quite differently. Notably, initrds are whole filesystem images, so once you're done with them you have to get rid of them (hence the nasty pivot_root(2) kludge, the even nastier echo-stuff-into-a-file-under-/proc kludge, and so on). None of this is necessary with initramfses, although the busybox switch_root tool remains useful (principally because deleting the contents of the initramfs would leave a shell script rather stuck when it came to chrooting into the real root filesystem).
I'd also recommend linking busybox (and anything else on your initramfs) agaisnt uClibc or some other tiny libc, and linking it statically, because when you only have one or two binaries, the savings from avoiding inclusion of unused symbols exceed the space wasted by duplicating those symbols in many binaries.
Using initramfses is particularly important for those of us who boot from filesystems that require userspace support to bring up, such as LVM or md/raid with v1 superblocks. It also means you can fsck the root filesystem *before* mounting it (it's always seemed icky and dangerous to me to fsck any mounted filesystem, and why should / be different?).
Also, the extra flexibility makes it really easy to hack in new features on demand. When I had a double disk failure earlier this year, it was a matter of a few minutes' work to bring up the network block device before mounting /; then I could distribute the RAID array containing the root filesystem partially onto another machine across the network. (Of course this was a real kludge and not desirable for long-term use: among other things, because nbd uses TCP/IP connections, whenever the remote machine rebooted, the root filesystem on the local one vanished too: but it meant I could keep running after a fashion until the disks were replaced.)
Plus, because initramfses are built into the kernel, if you have a modular kernel you don't need to worry about the modules getting out of synch with the kernel you're running. (In my experience this was a persistent headache with initrds.)
Embedded Linux: Small Root Filesystems Posted Nov 26, 2006 0:31 UTC (Sun) by giraffedata (subscriber, #1954) [Link] initrd and initramfs are separate things and work quite differently. Notably, initrds are whole filesystem images, so once you're done with them you have to get rid of them (hence the nasty pivot_root(2) kludge ... I don't get the distinction. First of all, there's some confusion because "filesystem image" means two different things. On the one hand, it's the entity that you create when you mount and destroy when you unmount, and by that definition the rootfs filesystem image (which gets loaded with files from a cpio archive) doesn't seem any less whole to me than the initrd filesystem image. pivot_root() allows you to unmount an initrd filesystem, thus destroying it and freeing its memory as well as removing its names from the namespace. Why don't you need to do that with initramfs? In its other meaning, a filesystem image is a copy of the bits that make up a filesystem, such as what the boot loader uses to convey initrd contents to a 2.4 kernel. Pivoting has nothing to do with getting rid of that kind of filesystem image. And it's equivalent to "getting rid of" the cpio archive used with initramfs.
The new initramfs system Posted Nov 22, 2006 12:35 UTC (Wed) by rvfh (subscriber, #31018) [Link] Just one question: if initramfs is built inside the kernel, what is passed to GRUB (with the initrd option) in my distribution? Or is the inclusion optional?
The new initramfs system Posted Nov 22, 2006 14:05 UTC (Wed) by smitty_one_each (subscriber, #28989) [Link] The kernel path is what's passed to GRUB. Think of the kernel image as an egg with a shrunken initramfs chicken stashed in it by the kbuild system.
Well worth perusing:
The new initramfs system Posted Nov 23, 2006 23:37 UTC (Thu) by Thalience (subscriber, #4217) [Link] It does seem to be the case that inclusion is optional. On my Ubuntu Edgy system, /boot/initrd.img-2.6.17-10-generic is a compressed cpio archive, not an ext2 image or whatever.
What I don't know is why they create a separate initramfs, rather than building it into the kernel image.
Re: why separate initramfs Posted Nov 24, 2006 1:40 UTC (Fri) by ldo (subscriber, #40946) [Link] >What I don't know is why they create a separate initramfs, rather than So that it's easier to update the kernel and the initramfs image separately, of course.
Re: why separate initramfs Posted Nov 25, 2006 17:28 UTC (Sat) by Thalience (subscriber, #4217) [Link] That does make some sense, but I've never seen that capability used.
Re: why separate initramfs Posted Nov 27, 2006 6:17 UTC (Mon) by ldo (subscriber, #40946) [Link] >That does make some sense, but I've never seen that capability used. On the contrary. Look at most binary-based Linux distros (that is, most Linux distros), for example (looking at ones on machines I have immediate access to) SuSE or Red Hat. They tend to come with only a small selection of pre-built kernels to choose from. But rather than build in every driver you might need at boot time into the kernel, you have to make an initrd that includes the necessary kernel modules that will need to be loaded even before the root filesystem can be mounted.
Re: why separate initramfs Posted Dec 5, 2006 21:22 UTC (Tue) by hazelsct (guest, #3659) [Link] It's used all the time in Debian. For example, upgrading udev or other utilities, adding or removing uswsusp support (suspend functionality) or the various splash screen programs, etc. Very useful.
Re: why separate initramfs Posted Nov 26, 2006 0:06 UTC (Sun) by landley (guest, #6789) [Link] If you build an initramfs in and then supply an external one, the externalone gets extracted over the built-in one, with the external one winning in case of any conflicts (replacing those files).
So you can always add an external initramfs with _just_ the changed files.
And don't forget... Posted Nov 23, 2006 0:41 UTC (Thu) by Baylink (subscriber, #755) [Link] Microsoft holds a patent on init(1).
;-)
Embedded Linux: Small Root Filesystems Posted Nov 23, 2006 5:41 UTC (Thu) by zipdisk (guest, #8589) [Link] You missed a very used way to easily construct a small root filesystem,the cousin of busybox: buildroot. Basically it will create a full small root file system "a la carte", based on uclibc, busybox and some other applications.
initrd in 2.6 Posted Nov 23, 2006 8:09 UTC (Thu) by ldo (subscriber, #40946) [Link] The article seems to suggest that initrd is no longer used with 2.6 kernels. In fact initrd is still available with 2.6, and is used instead of initramfs by Red Hat and SuSE, that I know of.
the initrd is actually an initramfs Posted Nov 23, 2006 15:18 UTC (Thu) by scottt (subscriber, #5028) [Link] dmesg | grep initramfschecking if image is initramfs... it is
Re: the initrd is actually an initramfs Posted Nov 24, 2006 1:43 UTC (Fri) by ldo (subscriber, #40946) [Link] You're right! I checked the initrd on my SuSE 10.0 system, and it is in fact a compressed cpio archive! No wonder I had no luck trying to mount it as a loopback image :). Another thought, though: with the old loopback initrd, all the memory used was freed after the pivot_root and the dismount. But with the initramfs mechanism putting things into rootfs, those things stay there, taking up memory, being only hidden when the real root filesystem is mounted on /. Does that make sense?
Re: the initrd is actually an initramfs Posted Nov 26, 2006 0:11 UTC (Sun) by landley (guest, #6789) [Link] The switch_root utility essentially does:
cd newroot
This frees up the space used by the initramfs by deleting all the files
The reason you can't do this in a shell script is after the "rm -rf" step,
Re: the initrd is actually an initramfs Posted Nov 27, 2006 6:24 UTC (Mon) by ldo (subscriber, #40946) [Link] Of course, you wouldn't need a special boot-time utility for this, you could do the cleanup at your leisure afterwards, if only something like this were permissible: mount -t rootfs rootfs /mnt But this appears to be ruled out by the NOUSER flag...
Embedded Linux: Small Root Filesystems Posted Nov 23, 2006 9:17 UTC (Thu) by dagb (subscriber, #30984) [Link] I´ll take my chances plugging the following WIP. It shows how to build a simple root fs with a little aid from gentoo's portage, but should be of use to people running other distros as well.
http://impact.ii.uib.no/~dagb/building_initrds_for_fun_an...
Dag B.
Embedded Linux: Small Root Filesystems Posted Nov 23, 2006 18:42 UTC (Thu) by maks (subscriber, #32426) [Link] klibc is a small libc for initramfs it may already enough for the needed apps :http://www.kernel.org/pub/linux/libs/klibc/ hpa maintains it for furture kernel inclusion: http://kernel.org/git/?p=libs/klibc/klibc.git;a=summary
Embedded Linux: Small Root Filesystems Posted Nov 26, 2006 0:38 UTC (Sun) by giraffedata (subscriber, #1954) [Link] One other interesting point about early root filesystems: In a latter day 2.4 kernel, there is a root filesystem even earlier than the initial ramdisk one (initrd). rootfs is built into the kernel, and always contains a handful of filesystem that you need to get up far enough to load and mount the initrd filesystem (the root directory and a few device special files, as I recall). And it never gets unmounted; it's always down there with the real root filesystem mounted over it. I don't know 2.6 booting, but it sounds from the article like 2.6's rootfs is an extension of this.
unable to open initial console - porting linux 2.6.15.4 on AT91RM9200 board Posted Jul 23, 2007 4:21 UTC (Mon) by absaxena (guest, #46379) [Link] Hi,We have been trying to port linux (2.6.15.4) on to a AT91RM9200 board. The kernel boots properly but is unable to bring on the rootfilesystem. Linux 2.4 has been running on this board quite successfully.
We are wondering if anyone faced a similar problem and could provide some suggestions for troubleshooting. Earlier, we tried some suggestions available on the web; unfortunately none of them worked for us.
Relevant details are attached.
Thanks,
A. Here is the boot trace:
Linux Kernel copied to SDRAM
+++++++++FLASH EXT++++MENU+++++++++++++++
File loaded by RS232
+++++++++FLASH EXT++++MENU+++++++++++++++
B. Here is is the kernel configuration:
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
C. BusyBox 1.5.1 configuration
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
|
|||||||||||||