Ksplice provides updates without reboots

While Linux systems generally have a good reputation for uptime, there are sometimes unavoidable reasons that a reboot is required. Typically, that is because of a kernel update, especially one that fixes a security hole. Companies that have long-running processes, or those who require uninterrupted availability, are not particularly fond of this requirement. A new company, Ksplice, Inc. has come up with a way to avoid these reboots by hot-patching a running kernel.

The technique used by the company, unsurprisingly called Ksplice, is free software, which we looked at last November on the Kernel page. (An earlier look, from April 2008, may also be instructive). The basic idea is that by doing a comparison of the original and patched kernels, one can build a kernel module that will patch the new code into the running kernel.

For simple code changes, the process is fairly straightforward. Each kernel is built with a special set of flags to simplify determining which functions have changed as a result of the patch. Those changes are packaged up into the module, and then applied when the module is loaded. Then there is the small matter of ensuring that the kernel is not currently executing any of the functions to be replaced. In order to do that, the kernel is halted while each thread is examined, if none are running the affected code—or have a return address into the code on their stack—the patch is made and the kernel can go on its way. Otherwise, Ksplice delays for a short time and tries again, eventually giving up if it cannot satisfy that condition.

There are several kinds of changes that are much more difficult to handle, particularly data structure changes. For those, someone needs to analyze the changes and write code to handle munging the data structures appropriately. Ksplice has an infrastructure that allows this data structure manipulation to be done while the kernel is halted, but the code itself is, or can be, non-trivial. To a great extent, it is the knowledge of how to do this with Ksplice that the company is offering as a service.

As a test of the technology, the Ksplice developers looked at all of the security problems listed for the kernel over a three-year period (May 2005 to May 2008). Of the 64 Common Vulnerabilities and Exposures (CVE) entries for the kernel that had an impact worse than denial of service, Ksplice was able to patch 56 without any additional code being written. The other eight could be handled with a small amount of code—an average of 17 lines per patch.

As a further demonstration of the Ksplice technique, Ksplice, Inc. is currently offering a free-beer service for Ubuntu 9.04 (Jaunty Jackalope) users. Ksplice Uptrack will allow those users to update their kernels without rebooting them. The Ksplice folks will be tracking the Ubuntu kernel git tree, turning those changes (security and bug fixes) into modules that can be retrieved and applied with the Uptrack client. As described in the FAQ, Uptrack will support the latest release of Ubuntu: 9.04 for now, switching over to 9.10 (Karmic Koala) when that is released.

As noted, Ksplice itself is free software, available under the GPLv2, and the Uptrack client is as well. That leads to a service-oriented free software business model for Ksplice, Inc. While their exact plans are not yet clear, providing similar updates for enterprise kernels (RHEL and SUSE), but charging for those, would seem an obvious next step. Other areas for expansion include other operating systems as well as user-space applications. In an interview, Waseem Daher, co-founder and COO of Ksplice, described the company's goal:

That's a rather ambitious vision, but one that seems in keeping with where things are headed. No matter how fast booting gets, it is still a major annoyance, for servers or desktops. Even restarting applications, particularly things like database servers or desktop environments, leads to lost time and productivity. Whether Ksplice, Inc. can expand their offerings to reach that goal is an open question.

One of the problems that Ksplice will face is competition. In the Linux world, that could come from distributors deciding to start making Ksplice modules themselves, and either charging their customers for them, or adding that capability to their subscription-based support offerings. In the proprietary, closed source world, Ksplice will have to work with the vendors of operating systems and applications so that it can access the source code. Those vendors are most certainly going to want a piece of the pie for that access.

There may also be technical hurdles. One botched kernel update, which led to introducing a serious flaw—security or otherwise—could ruin the company's reputation. That, in turn, might make it much harder to convince new customers. Hot-patching is a subtle, difficult problem to solve completely.

On the other hand, Ksplice has an excellent pedigree; started by four MIT students based on co-founder Jeff Arnold's master's thesis. Ksplice also won MIT's $100,000 entrepreneurship competition—against some stiff competition, one would guess. Arnold's reasons for looking at the problem will resonate with system administrators everywhere: he delayed patching an MIT server to avoid downtime on a busy system, so an attacker took advantage of that window.

It will be interesting to watch both Ksplice and the general idea of hot-patching over the coming years. When Ksplice was first introduced, a Microsoft patent on the technique was noted on linux-kernel, along with protestations of rather old (PDP-11) prior art. How that plays into the fortunes of Ksplice and others who come along will be interesting—potentially sickening—as well.