Embedded Linux: Small Kernels

A few years back, LWN noted the introduction of a new project led by Matt Mackall to help trim the fat from a bulging Linux kernel: the TinyLinux project. In a paper presented the 2004 Ottawa Linux Symposium Mackall explained that much code had been added over the years to the kernel to improve performance for certain classes of hardware but that, over time, this code had become less helpful with newer hardware and in some cases even caused performance degradation.

The solution was to provide a mechanism to remove features from the kernel that were unnecessary for certain classes of hardware, making the kernel smaller and more suited to certain environments. This includes embedded devices like those in the consumer electronics market but also older systems (like 386-based hardware and handhelds) which typically have tighter resource restrictions (less memory, smaller caches, reduced storage, and so forth). To this end, Mackall created the TinyLinux project which provides a set of patches (or one giant patch) aimed at making various features in the kernel optional as a way of reducing the size of the kernel.

A Meaningful First Step

Mackall's project based on his original paper is called TinyLinux, and is also known as the -tiny tree. It consists of a number of small patches allowing users to disable various features that otherwise might not be configurable. This includes items like switching from the SLAB allocator to a more space-efficient version called SLOB, configurable IDE and serial PCI hardware support, optional support for asynchronous I/O, sysfs and vm86, and minimizing VT support. There are also patches for debugging with netconsole, kgdb and kgdb-over-ethernet.

TinyLinux comes as a set of patches. Users are free to pick and choose which patches they want to apply to the kernel source. Alternatively they can use a monolithic patch that applies all of the TinyLinux features to the source. Once the patches are applied, TinyLinux features can be enabled under the "General Setup->Configure standard kernel features" menu that is displayed with "make menuconfig".

The goal of the project has always been to build a modern kernel that will run in as little as 2MB of RAM. That includes console, disk and network support. Guidelines set by Mackall for the project include:

1. Anything that isn't applicable to all systems should be configurable.
2. Patches should be small and independent so integrators can choose the ones of value to them.
3. Attempt to make the patches mergeable with the mainline.

TinyLinux Features

Once the selected patches have been applied the configurable options will be found under the General Setup page in the kernel config menu (re: make menuconfig). At the bottom of this page is an option labeled "Configure standard kernel features (for small systems)". This option, which is the CONFIG_EMBEDDED option in the kernel config file, must be set in order to reach the next level menu where the TinyLinux options live.

There are 80 patches in the 2.6.14 release for TinyLinux which add a much smaller set of configurable kernel options. Some of the more interesting options include the following (listed with the menuconfig label followed by the kernel config file option in parenthesis):

Enabled accounting of kmalloc/kfree allocations (CONFIG_KMALLOC_ACCOUNTING)

This patch adds accounting features for kmalloc/kfree calls. While not meaningful in itself for reducing kernel image sizes or runtime memory allocation, this patch can be useful in helping to track down memory leaks and abusers of dynamically allocated memory. The patch adds a /proc/kmalloc entry that can be read to find kmalloc/kfree usage statistics. See the LWN announcement of this patch from 2005 for more details.

BUG() support (CONFIG_BUG)

This patch isn't in the 2.6.14 patch set. It was originally delivered in early 2005 and has since been rolled into the kernel mainline. The config option removes all the kernel BUG and WARN messages. It is said to trim about 35k off the typical kernel as well as make the system slightly faster.

Enable ELF core dumps (CONFIG_ELF_CORE)

This patch allows removing of the code that handles ELF core dumps. Small systems don't tend to need ELF core dumps because there probably isn't any way for the consumer to view the dump, nor do you usually want the consumer to see it. The config option, if not set, strips a large chunk of lines from the fs/binfmt_elf.c file.

Enable inline measurement (CONFIG_MEASURE_INLINES)

When enabled produces data during a kernel compile that can be saved to a file and processed by the count-inlines script to show the number of code instantiations. This option counts instantiations by marking the inline functions as deprecated. If you set this, be prepared for a very verbose build output.

Number of swap files log2 (0 => 1, 5 => 32) (CONFIG_MAX_SWAPFILES_SHIFT)

This sets the maximum number of swap files that can be configured. The value is log2 so 0 means 1 swap file and the maximum, 5, means 32 swapfiles. The old default is 5, and that's the same setting if this option is not changed.

Use full SLAB allocator (CONFIG_SLAB)

If this is not set, then -tiny replaces the advanced SLAB allocator and it's associated kmalloc support with a simpler system called SLOB. From the original post for SLOB from Matt:

SLOB is a traditional K&R/UNIX allocator with a SLAB emulation layer, similar to the original Linux kmalloc allocator that SLAB replaced. It's significantly smaller code and is more memory efficient. But like all similar allocators, it scales poorly and suffers from fragmentation more than SLAB, so it's only appropriate for small systems.

Use mempool allocator (CONFIG_MEMPOOL)

Mempools were an early part of the 2.5 tree that were introduced as part of the (then) new block I/O layer. The goal was to provide a solution to prevent deadlocks for memory requests that had to succeed but could not sleep. For some small system configurations preallocating pools of memory could be considered both unnecessary and a waste of limited resources. However, the introduction of this option raised some interesting concerns over whether mempools really reduced deadlock to zero to begin with and that removing mempools completely might ensure that deadlocks were guaranteed to occur. In any case, use of this option can help with small memory systems but be aware that even Matt has said that "deadlock odds are significantly higher with some usage scenarios."

Working With TinyLinux

TinyLinux was last updated for the 2.6.14 kernel. To find out if these patches really worked to reduce the image size and let the kernel run in as little as 2MB of memory, I experimented with the -tiny patches with this kernel. First, I compiled the kernel for my Via EPIA-M kernel and a stripped down Busybox initramfs that simply booted into a shell prompt.

I then applied the TinyLinux monolithic patch. The kernel built from this is based on the the configuration options specified on the CE Linux Forum page about using TinyLinux. This page is not quite in sync with the latest TinyLinux so I had to modify their suggestions slightly.

The compiled kernels are compressed to boot on the test board. The compressed files show roughly 410KB are saved in the TinyLinux image:

    mjhammel(tty3)$ l linux-2.6.14*
    -rw-r--r--  1 root root 1550312 Jun 25 23:09 linux-2.6.14-via
    -rw-r--r--  1 root root 1139708 Jun 26 22:08 linux-2.6.14-tinylinux

Memory Usage

To find out if TinyLinux really helped, we can first check to see if the text, data and bss sizes in the images changed significantly. The size-delta script (from the CE Linux Forum) program can read the uncompressed Linux kernel images and compare how much of an impact TinyLinux is having:

    $ size-delta vmlinux.via vmlinux.tinylinux
    vmlinux.via  =>  vmlinux.tinylinux
     text:  2695282  2050286  -644996 -23%
     data:   440124   229107  -211017 -47%
      bss:   178912   129976   -48936 -27%
    total:  3314318  2409369  -904949 -27%

As you can see, the final configuration produces up to 27% reduction in size compared to the original Via configuration.

But the things that the TinyLinux patches really affect can only be seen when you check runtime memory usage. The best way to see how the kernel looks at boot time is to check dmesg for the memory usage line. I booted the Via kernel (sans TinyLinux patches) first and checked it's usage:

% dmesg | grep Memory
   Memory: 4028k/8192k available (2179k kernel code, 3756k reserved, 727k data, 160k init, 0k highmem)

Then I tried the TinyLinux kernel. There is a minor problem with using dmesg here. In the config for this kernel, as suggested by the CE Forum configurations, I disabled the printk()'s using a TinyLinux option, but dmesg needs those printk()'s. Turning printk()'s back on increases the memory usage for the kernel. It's a tradeoff that is required to make it easy to see the changes in memory usage at runtime.

The TinyLinux kernel produced this line at boot time:

% dmesg | grep Memory
   Memory: 4028k/8192k available (1794k kernel code, 3072k reserved, 484k data, 136k init, 0k highmem)

The "reserved" number is the amount of memory the kernel has taken out of circulation before anything starts running - it includes the "kernel code" amount and various other things. Both kernels were booted with mem=8M. The TinyLinux kernel saved about 400k in kernel code and close to 700k in reserved memory.

To see if I could use other options (not listed in the CE Forum suggestions) to get the kernel smaller, I tried the following:

	Disabled these:
	- Enable panic reporting code 
	- Enable various size reductions for networking 
	- Enable ethtool support 
	- Enable device multicast support 
	- Enable inline measurement 
	
	Enabled these:
	- Optimize for size 

The results were even better:

% dmesg | grep Memory
    Memory: 5016k/8192k available (1526k kernel code, 2768k reserved, 464k data, 126k init, 0k highmem)

This produced a savings of nearly 1M. And I haven't even tried to strip the kernel of unnecessary drivers yet.

If you want to get more into it, have a look at /proc/slabinfo. It contains the system slab caches and how much memory is committed to each. This is low-level grungy information, but part of what -tiny does is to try to reduce the size of many of the slabs. The "slab" line in /proc/meminfo gives a total of the memory consumed by slabs. For the last kernel I built, meminfo showed a Slab value of 796kB. On the original kernel this value was 872kB.

The Future of TinyLinux

The latest TinyLinux patch set works with the 2.6.14 kernel. Many features from Linux-tiny have already been integrated into the 2.6 mainline kernel and Mackall is in the process of trying to clean up what's left for final merging.

Mackall stated at a CELF presentation that TinyLinux wasn't helping much anymore. Additionally, he wasn't getting a lot of feedback or contributions to the project, making his efforts to create new TinyLinux releases for new kernel releases all the harder. According to Mackall, the problem might have been his quick, early success with the project:

Mackall is in the process of rolling most of the patches into the mainline.

Beyond TinyLinux

What else can you do to reduce kernel size? The CE Linux Forum Open Test Lab offers resources for working with system size. Some suggestions include the use of SquashFS and CramFS for using extremely compact ramdisk based root filesystems. This is a subject I'll take up in my next article on Embedded Linux.

One area not discussed in the use of smaller network stacks, such as the uIP stack. Such solutions are not for the novice systems integrator, however, and go way beyond simple patching and recompiling of the kernel. So, caveat developer.

In the next installment of this series I'm moving past the kernel and up to the root filesystem. The root filesystem is necessary not only to boot but to get access to the applications you're inevitably going to run on your small system. Keeping the root filesystem small involves a mixture of special build tools and utilities along with clever kernel modules. I'll be looking at BusyBox, compressed filesystems like SquashFS and the special UnionFS filesystem.

---

(Log in to post comments)