Canonical's kernel manager, Pete Graner, spoke at UbuCon—held
just prior to SCALE 8x—on the "Ubuntu Kernel Development Process".
In the talk, he looked at how Ubuntu decides what goes into the kernel and
how that kernel gets built and tested. It provided an interesting look
inside the process that results in a new kernel getting released for each
new Ubuntu version, which comes along every six months.
Graner manages a "pretty big" group at Canonical, of 25 people
split into two sub-groups, one focused on the kernel itself and the other on
drivers. For each release, the kernel team chooses a "kernel release lead"
(KRL) who is responsible for ensuring that the kernel is ready for the
release and its users. The KRL
rotates among team members with Andy Whitcroft handling Lucid Lynx (10.04)
and Leann Ogasawara slated as KRL for the following ("M" or 10.10) release.
The six-month development cycle is "very challenging", Graner
said. The team needs to be very careful about which drivers—in-tree,
out-of-tree, and staging—are enabled. The team regularly takes some
drivers from the staging tree, and fixes them up a bit, before enabling
them in the Ubuntu tree so that users "get better hardware
Once the kernel for a release has been frozen, a new branch is created for
the next release. For example, the Lucid kernel will be frozen in a few
weeks, at which point a branch will be made for the 10.10 release. That
branch will get the latest "bleeding edge" kernel from Linus Torvalds's tree
(presumably 2.6.34-rc1), and the team will start putting the additional
patches onto that branch.
The patches that are rolled into the tree include things from linux-next
(e.g. suspend/resume fixes), any patches that Debian has added to its
kernel, then the Ubuntu-specific patchset. Any of those that have been
merged into the mainline can be dropped from the list, but it is a
"very time-consuming effort" to go through the git tree to
figure all of that out. With each new tag from Torvalds's tree, they do a
git rebase on their tree—as it is not a shared development
tree—"see what conflicts, and deal with those".
The focus and direction for the Ubuntu kernel, like all Ubuntu
features, comes out of the Ubuntu Developer Summit (UDS), which is held
shortly after each release to set goals and make plans for the following
release. Before UDS, the kernel team selects some broad topics and creates
blueprints on the wiki to describe those topics. In the past, they have
focused on things like suspend/resume, WiFi networking, and audio; "a
big one going forward is power management", he said.
The specifications for these features are "broad-brush
high-level" descriptions (e.g. John has a laptop and wants to get 10
hours of battery life). The descriptions are fleshed out into various use
cases, which results in a plan of action. All of the discussion,
decisions, plans, and so on are captured on the UDS wiki
One of the longer kernel sessions at UDS looks at each kernel configuration
option (i.e. the kernel .config file) to determine which should be
enabled. New options are looked at closely to decide whether that feature
is needed, but the existing choices are scrutinized as well.
In addition, Graner said that the team looks at the patches and drivers
that were added to the last kernel to see which of those should be
continued in the next release. He pointed to Aufs as a problematic feature
because it always breaks with each new kernel release and can take up to
three weeks to get it working. They have talked about dropping it, because
Torvalds won't merge it into the mainline, but the live CDs need it.
The kernel team has to balance the Ubuntu community needs as well as
Canonical's business needs, for things like Ubuntu One for example, and
come up with a set of kernel features that will satisfy both. The
discussions about what will get in at UDS can get intense at times, Graner said, "Lucid was
pretty tame, but Karmic was kind of heated".
Lucid will ship with
the 2.6.32 kernel which makes sense for a long-term support (LTS) release.
2.6.32 will be supported as a stable tree release for the next several
years and will be shipped with the next RHEL and SLES. That means it will
get better testing coverage which will lead to a "very stable kernel
Each stable tree update will be pulled into the Ubuntu kernel tree, but LTS
updates to the kernel will only be pushed out quarterly unless there is a
"high" or "medium" security fix. For new kernel feature development, new
mainline kernel releases and release
candidates are pulled in by the team as well. Graner gave two examples of new
development that is going on in the Ubuntu kernel trees: adding devicetree
support for the ARM architecture, which will reduce the complexity of
supporting multiple ARM kernels, and the AppArmor security module that is
being targeted for the 2.6.34 kernel.
Once the kernel version has been frozen for a release, the management
of that kernel is much more strictly controlled. The only patches that get
applied are those that have a bug associated with them. Stable kernel
patches are "cherry-picked" based on user or security problems. There is a
full-time kernel bug triager that tries to determine if a bug reporter
has included enough information to have any hope of finding the
problem—otherwise it gets dropped. One way to ensure a bug gets
fixed, though, is to "show the
upstream patch that fixes the problem"; if that happens, it will get
pulled into the kernel, Graner said.
There are general freezes for each alpha, beta, and the final release, but
the kernel must already be in the archive by the time of those freezes. Each
time the kernel itself freezes, it "takes almost a full week to build
all of the architectures" that are supported by Ubuntu. There are
more architectures supported by Ubuntu than any other distribution
"that I am aware of", he said. Each build is done in a
virtualized environment with a specific toolchain that can be recreated
whenever an update needs to be built. All of that means the kernel
needs to freeze well in advance of the general release freeze, typically
about a month before.
Once the kernel is ready, it is tested in Montreal in a lab with 500 or
600 machines. The QA team runs the kernels against all that hardware,
which is also a time-consuming process. Previously, the kernels would be
tossed over the wall for users to test, but "now Canonical is trying
to do better" by dedicating more resources to testing and QA.
Managing kernel releases for a distribution is big task, and the details of
that process are not generally very well-known. Graner's talk helped to
change that, which should allow others to become more involved in the
process. Understanding how it all works will help those outside of the
team do a better job of working with the team, which should result in
better kernels for Ubuntu users.
to post comments)