Leading items

Novell, buyer's remorse, and the patent threat

One cannot help but wonder if Novell's executives aren't having second thoughts about that company's recent deal with Microsoft. Since the announcement, there has been quite a bit of hostile commentary in the community, and there are signs of increasing levels of unhappiness within the ranks of Novell's free software developers. The increase in Novell's stock price turned out to be short-lived. And Microsoft CEO Steve Ballmer has used this deal as his excuse to embark on a FUD campaign which brings back memories of Darl McBride's heyday.

For an example, consider this widely-distributed bit of fun:

Mr. Ballmer is clearly claiming that Linux infringes upon Microsoft's patents, and that Linux users owe money to Microsoft. Novell is fairly clearly seen as having agreed with and validated that claim - otherwise, what, exactly, is Novell paying for? In an attempt to change that perception, Novell has sent out an open letter to the community, saying:

Microsoft has responded with a letter of its own.

It seems that, perhaps, Novell got a slightly different deal than it was expecting at the outset. Presumably Novell's management is smart enough to understand that, if it throws away its community goodwill and runs into problems with the GPL, Novell's Linux business will have a dark future. Presumably, Novell's managers did not want to see their company be the enabler for a new flood of anti-Linux FUD and attempts to divide the community. Seemingly, however, those managers did not think through the consequences of signing this non-license with Microsoft. Thus the open letter and the IRC meeting about the deal, scheduled for November 27.

Microsoft's claims have been met with a "show us the patents" response in parts of the community. Novell's open letter, which refuses to acknowledge the existence of patent issues, is a very similar sort of response. This approach worked well in the SCO case, for a simple reason: there was no substance to that company's wild claims. It is natural to think that the same sort of challenge will work this time around, but that thinking may be a mistake.

The SCO case was, at least in certain phases, based on copyright. Avoidance of copyright problems is relatively easy for a free software project; all that is required is to not accept code of uncertain origin. Truly original work cannot have copyright issues. Microsoft, however, is talking about patents. Anybody who thinks that Microsoft holds no patents which can be applied to Linux has, perhaps, failed to understand the scope of the software patent problem. There is no clear way for a free software project to avoid software patent issues - at least, in parts of the world where such patents are recognized.

An incredible number of patents have been issued covering trivial techniques. One of your editor's favorites is #6,732,359, the primary claim of which is:

This ground-breaking, innovative work was patented in 2004; presumably, nobody ever thought of such a technique before 1999, when the patent was originally filed.

In the real world, anybody trying to enforce a patent like this would be immediately buried in prior art. But there is little comfort to be found there. Even a relatively large company like Novell can only afford to defend so many patent suits, and there are a lot of patents like this one out there. Even if Microsoft does not currently own any patents which could be applied to Linux, there is no doubt that it could acquire some without great difficulty. Unlike SCO's claims, the patent problem is real, whether Novell publicly acknowledges it or not.

If Microsoft had wanted to mount a patent attack against Linux, it could have easily done so by now. There's plenty of reasons which may explain why this has not happened so far. The fact that software patents are not recognized worldwide could well be part of the equation; that is why continued resistance to their imposition in Europe is so important. An attack against Linux certainly would not help Microsoft's position with antitrust authorities. Chances are that almost any company which is buying Linux support services is also a Microsoft customer, and Microsoft might just be smart enough to want to avoid upsetting its own customers. A legal campaign against Linux might well bring together a fearsome coalition of large companies with an interest in defending Linux and blood in its eyes. There is also the simple fact that Microsoft has not, to date, acted much like a patent troll; it has, instead, spent more time on the defendant's side of the courtroom.

None of this gives any sort of real assurance that Linux is safe from such attacks by Microsoft, certainly. One should never underestimate corporate unpredictability - or stupidity. But it does suggest that the risk of a patent attack has not really changed as a result of Novell's arrangement. That risk existed before, and it still does. And, as Mr. Ballmer pointed out, it's not just Microsoft's patents. When the patent attack comes, it will likely originate from a small litigation company which has no customers to offend and no assets to countersue for. Novell (and its customers) will be no safer than the rest of us when this attack happens.

So one might, indeed wonder what Novell thought it was buying. The answer, perhaps, lies in the fact that the net cash flow is very much in Novell's direction. Hundreds of millions of dollars can be hard to turn down. One can hope that this money ends up benefiting both Novell and the free software community that Novell depends on. At the moment, however, it looks like Novell has put itself into a bit of an uncomfortable position.

Comments (42 posted)

Notes from the leading edge

Much of your editor's work has, for the last couple of years, been done on an x86-64 system running the Fedora Development ("Rawhide") distribution. Running Rawhide - just like running any other development distribution - has certainly provided sufficient experience to keep your editor grumpy for some time. Even so, the current state of post-FC6 rawhide is, perhaps, exceptional.

The gnome-terminal package picked up some interesting behavior where it seemingly grows without bound - a 350MB virtual address space on your editor's system, with 76MB resident. It easily outweighs lean-and-mean applications like emacs and liferea - though it remains outclassed by firefox. The cursor does not respond to focus events; your editor has learned to type at terminals even if they look like they are not listening. Occasionally a terminal will get into a mode where it refuses to respond to input until it receives a mouse click or two. And, occasionally, the whole thing just crashes, taking down every terminal window and every associated ssh session.

Your editor's longstanding appreciation for xterm is on the rise again.

In the hope of picking up a fix in a timely manner, your editor has been tracking the Rawhide repository a bit more closely than usual. Or, at least, attempting to do so. It can be quite discouraging to type "yum update" and have yum simply go off forever. Among other things, one must wait a great long time to distinguish this behavior from yum's normal mode of operation. Other times, it comes back very quickly with a message saying, for all practical purposes, "RPM crashed, you lose, sorry."

That is the sort of message that can chill a system administrator's blood. There's no end of system problems which can be addressed by reinstalling a package, perhaps moving to an older version. That is especially true for systems which are following a leading-edge development repository; one simply expects to install an ill-fated package occasionally. But if the package management system itself fails, this important tool goes away and one's ability to restore a sick system is severely compromised. It's about at this point that one begins to think it would have been good to check the system's backups a little more frequently.

Some digging around turns up the fact that these problems are well known and well documented in the bug-tracking system. Also found was a magic command, previously unknown to your editor, which evidently needs to be part of every system administrator's toolkit (at least, those who work with RPM-based systems):

    rm /var/lib/rpm/__db*

Sure enough, every time your editor's system goes nonlinear (i.e. after every "yum update"), removing those cache files makes the problem go away. It would be awfully nice if RPM could figure out for itself that its cache is corrupt and not depend on people to clean up its messes for it. But that, evidently, is more than we should feel entitled to expect.

Still, one could consider taking this issue - perhaps with a patch - to the RPM maintainer. Except that, for the purposes of most distributions, there still is no RPM maintainer. Your editor asked who maintains RPM? back in August, but no distributor has since announced a plan for moving to the current "upstream" version of RPM or establishing a formal fork. The November 20 Fedora Board meeting talked about an upcoming "RPM announcement," but it remains unannounced as of this writing. Getting a handle on the status of that crucial package would be most beneficial for users of RPM-based distributions, whether or not they do silly things like track development repositories.

Comments (38 posted)

Embedded Linux: Small Root Filesystems

In the first part of this series I looked at the TinyLinux project, a set of patches aimed at helping developers reduce both the size of their kernel images and the amount of memory they use at runtime. But if you're working to build a working small system you'll need more than a kernel. You need something for the kernel to manage. You need user space applications and an environment in which they can run on top of your tiny kernel. And like your tiny kernel, you need your environment and applications to be as small as possible.

The Root Filesystem

The Linux kernel works hand in hand with what is called the root filesystem. This is the filesystem upon which the root directory can be mounted and which contains the files necessary to bring the system to a state where other filesystems can be mounted and user space daemons and applications started. Most desktop and server distributions make use of two kinds of root filesystems: the initial root filesystem and the real root filesystem. The former is used to mount and run the latter.

The directory structure for a root filesystem can be extremely minimal, as we'll see in a moment, or it can contain the usual set of directories including /dev, /bin, /etc, and /sbin, among others that you see in any desktop Linux distribution.

The kernel boot process concludes with the init code (see init/main.c) whose primary purpose is to create and populate an initial root filesystem with a set of directories and files. It then tries to launch the first user mode process to run an executable file found on this initial filesystem. This first process ("init") is always given process ID 1.

There are three ways for the kernel to find the file that will be run by the init process. The first method is to use a file specified at boot time with the init= kernel parameter. If this parameter is not set, the kernel tries a series of locations to find a file named "init". These include /sbin/init, /etc/init, and /bin/init. If all these fail, the kernel tries to run any shell it finds at /bin/sh. If this last fallback is not found, the kernel will print an error saying that no init could be found.

Once the init process is started it typically begins to launch other user space programs. On a desktop or server system this is known as the sysvinit process and includes the set of scripts found (typically) under /etc/rc.d. The name sysvinit comes from the mechanism used in System V Unix, which defined the naming scheme used for directories and files. On embedded and small footprint systems the init process may be a set of custom designed scripts or even a single application. Some desktop distributions are also beginning to replace sysvinit with alternatives designed with faster booting in mind.

The early root filesystem: Initial Ramdisks

The initial root filesystem is known as the initial ramdisk because the filesystem lives in a disk image created by the kernel in RAM. In a desktop or server system, the initial ramdisk is used to load drivers and initialize an environment so that an external storage system (disk or network attached storage) can be mounted. The switch from the initial root filesystem to the real root filesystem is called a pivot. The pivot causes the real root filesystem to be mounted over the initial root filesystem. When that happens, a new init process from the real root filesystem is launched and takes over the process id of 1. At that point the initial ramdisk is no longer needed and the memory can be freed.

In an embedded system the initial ramdisk might be the only filesystem ever mounted, since it contains all the user mode applications required. Alternatively, the initial ramdisk might mount a flash drive or other local storage yet still not pivot. Instead the mounted storage might be used as a source for user space applications. In these cases the initial ramdisk is never cleared because the pivot never happens.

In the 2.4 kernel, the initial ramdisk was referred to as the initrd image. It was created as a filesystem inside a file. The file was mounted and a filesystem created inside of it. A directory structure for the initial ramdisk was copied into this filesystem. The file was then unmounted, compressed and provided to the kernel via the initrd= kernel parameter at boot time.

The initrd file could only be loaded at boot time from an external source (except for the MIPS kernel, which allowed you to embed the image into the kernel). The original ram disk mechanism for the 2.4 kernel created a synthetic fixed sized block device that needed the filesystem driver used when the initrd was created, such as ext2, in order to work with file data. At the end of the boot process the initrd image had to be unmounted in order to clean up memory usage before switching to a more complete root filesystem.

In the 2.6 kernel the process of creating and using the initial ramdisk has been somewhat simplified. First, the files are simply collected together in an compressed CPIO file, now referred to as the initramfs instead of initrd. The initramfs file is always embedded in the kernel (for all hardware platforms) even if you don't create one yourself. If you don't, a default CPIO archive is created automatically by the kernel build process.

Second, there is no external filesystem required at boot time for the initramfs. Instead, the initramfs is unpacked in a special ramfs-based filesystem called the rootfs. The ramfs filesystem support is built into the kernel and cannot be disabled, so it's always available. Because it doesn't use backing store, it's a simpler system than the mechanism used in the 2.4 kernel. And when the boot process is done with the initramfs, a more complete root filesystem (such as one found on disk) can be directly mounted over it without worrying about wasting a lot of memory.

Why use the initramfs?

It would be ideal if the kernel could boot into a minimal state that knew just enough to bring the system to a useful state for the user or environment it will run in. This minimal state would allow the kernel to be as small as possible with as few options compiled in as possible. This is exactly why you use an initramfs.

On any system, and most especially on resource limited systems, you want to keep the kernel itself small and dynamically load only those driver modules that are required to make the system finish booting. Most desktop systems use the initramfs to determine what kind of hard drive or other storage is available with a complete root filesystem. In this case the initramfs contains boot scripts and driver modules relevant to bringing up the system. These files are only kept around temporarily while the real root filesystem is mounted and the real init process is started. Because the variety of desktop hardware is large, the initramfs can end up being large and fairly sophisticated as it tries to guess what kind of hardware is about to be mounted.

On small systems the situation can be much different. There may not be any additional storage available to hold another, more complete root filesystem. In that case the initramfs becomes the real root filesystem. Because the initramfs is running out of RAM, it will contain only those files and directories absolutely necessary to run the system.

Alternatively, a small system might use a dedicated flash drive with read only access to prevent accidental destruction of the bootable system. In that case the initramfs will contain boot scripts that mount the flash device and perform a little trickery to simulate writeable partitions so the system can operate normally.

Creating an initramfs

It's possible to recreate an initial ramdisk that mirrors your running desktop using the mkinitrd script. The problem with using this script is that you're recreating your desktop environment. That's not likely what you're looking for in your embedded system or even a live CD. So we need to look at creating the initramfs manually.

The kernel source includes the text file ramfs-rootfs-initramfs.txt under Documentation/filesystems. In this file, under the section titled "Populating initramfs" are instructions for creating a very minimal initramfs. This includes a minimal set of device files, the /proc, /sys and /mnt directories, an init script and a BusyBox binary. We'll get to BusyBox in a moment.

Start by creating a directory called "myinitfs":

    mkdir myinitfs

Add some basic directories:

    mkdir -p myinitfs/{boot,proc,sys,mnt,sbin,dev,lib,usr/bin}

Not all of these are required but you'll want them around to populate with useful tools in your initramfs anyway. Next, add the required device files. If your kernel and user space processes need to be able to output messages then the minimal root filesystem will need a console device. This is created with the mknod command.

    mknod -m 644 myinitfs/dev/console c 5 1

If your system is booting from a CD and the root filesystem is in a compressed filesystem image on the CD then you'll also need a loop device.

    mknod -m 644 myinitfs/dev/loop0 b 7 0

Of course, your embedded system doesn't have to output messages to a console and it certainly doesn't have to mount any filesystems, so neither of these are required. But if you're creating a live CD you'll want them.

After creating the directory structure and adding these two devices, we copy in a shell script for our init program and a compiled copy of the BusyBox binary. The content of the shell script and the makeup of the BusyBox binary are the keys to getting your small system running.

Starting Small: BusyBox

BusyBox is the workhorse of embedded systems. It is a collection of commonly used Unix utilities rolled together into a single binary. The command line utilities usually have fewer options than their standalone counterparts but tend to be functionality similar. The primary goal of BusyBox is to provide a full featured set of utilities for resource limited systems.

BusyBox is a well designed package that is extremely easy to use. A graphical configuration utility similar in style to the curses-based kernel configuration utility allows you to choose the utilities you need. The Unix utilities are referred to as applets and the configuration utility lets you pick which applets to include in the binary. The choice of which applets to include depends entirely on the system you're trying to create. For a live CD that mounts a compressed file system from the CD as the real root filesystem (over the initramfs) you would include utilities like losetup, mount and umount, gzip, and tar, along with the basic ls, ash, grep, mkdir, mknod and so forth.

The build process for BusyBox is simple. Unpack the BusyBox archive in the current directory (where myinitfs is located). This creates a BusyBox directory. In that directory, create your configuration:

    make menuconfig

You'll be prompted to save the configuration, which you should do. In the configuration you should be certain to specify the directory where the build should be installed. While not absolutely required, it saves a copying step later. In the latest version of BusyBox, 1.2.1, look under the BusyBox Settings->Installation Options menu and set the install directory to "../myinitfs".

After configuration, you simply build and install the binary:

    make
    make install

Getting Bigger: LFS

Before looking at the init script I want to mention that, although BusyBox can provide just about everything you need to get the system booted and even provide a runtime environment on its own, you might need far more user space support. If you're looking to extend your system to a full distribution, be sure to look at the LinuxFromScratch.org web site, known more commonly as LFS. Here you'll find step by step instructions on how to build a complete distribution.

The LFS is often used in live CD distributions as the runtime system that is loaded from a compressed filesystem off a CD by a BusyBox-based initramfs. Building a live CD from scratch in this manner is a great way to learn what a Linux distribution is all about, from the kernel on up through KDE and GNOME.

A live CD init script

At this point you've created a minimal set of utilities and a directory structure suitable for booting (sans the kernel, of course). But you still need the all important init script that kicks things off for the user space environment.

I've worked with this init script for some time, which is based on the init script found in an older version of the LFS live CD. It assumes the use of UnionFS and SquashFS for mounting and using compressed filesystem image files from the CD.

In my next article in this series I'll look at how and why you would use compressed filesystems like SquashFS along with UnionFS to boot your system.

Comments (21 posted)