Debian adds continuous integration

By Nathan Willis
July 16, 2014

In June, Debian formally unveiled the Debian Continuous Integration (debci) project, which is an effort to integrate dependency-based automated testing into the rather sizable Debian package archive. So far, only a fraction of Debian's packages are making use of the service, but many more could be added with a little effort—and the benefits for Debian are potentially large.

For years, Debian has enjoyed the reputation of having one of the largest collections of available packages among Linux distributions; according to various project statistics, there are more than 20,000 source packages in the unstable distribution, which result in more than 40,000 binary packages and an untold number of inter-package dependencies. For a project without a corporate parent and relying on volunteer package maintainers, that adds up to quite a quality-assurance challenge. Some of the larger packages (such as GNOME and KDE) can pull in well over 1,000 dependencies, which makes manual testing at the very least impractical.

Consequently, as debci developer Antonio Terceiro explained in a blog post, there has long been an interest in deploying an automated testing system. A key piece of the puzzle was Ian Jackson's autopkgtest, which can execute tests on installed binary packages within a testbed system. In 2011, the autopkgtest framework was accepted as Debian Enhancement Proposal 8 (DEP-8).

DEP-8 was accompanied by a specification for defining tests within a Debian source package. The autopkgtest tool can run tests against the source package itself, but it can also execute tests designed for the corresponding binary package. The debci effort is geared toward finding regressions (particularly those that might result from changes in a dependency) rather than build errors, though, so it focuses on the binary tests.

Although not all Debian packages include their own tests, quite a few already do. At the end of 2013, Terceiro started experimenting with a tool that would run autopkgtest against the entire archive. That codebase is what eventually became debci.

Currently, an instance of debci is running on ci.debian.net, and it executes tests against roughly 350 packages. The server updates its Apt cache four times per day, looking for updated packages that have autopkgtest tests defined. If a package or any of its dependencies have changed since the last test was run against it, it is scheduled for a new run. In addition, tests are run once every month even if there are no changes.

Several test-execution backends are supported in autopkgtest, including LXC containers, QEMU/KVM virtual machines, and "ssh testbeds" (which assume nothing other than a working ssh connection). Each option provides a different way to isolate the test environment from the host system. The ci.debian.net server makes use of the schroot backend for most packages. Schroot creates a clean chroot environment and, according to the documentation, is the fastest option that provides filesystem isolation, although it has some limitations (such as tests not being able to start system services or alter the network configuration).

For a given package, debci sets up the schroot environment, updates the target package and all of its dependencies, then extracts and executes the target package's test suite using autopkgtest and logs the results. Autopkgtest supports quite a bit of variety in its test output (including no output, simple TEST: FAIL messages, and more detailed reports). The ci.debian.net server publishes new test results once every hour. In addition to global success/failure statistics for the entire archive, each individual package's results are published as an Atom feed, and there is a combined whole-archive feed.

The individual test results are also accessible through the site's web interface. For example, here is the apt package's test-result history. The debci log for each entry indicates what triggered the test (i.e., a change in the package itself or a change in a dependency), the current dependency list, the test result, and the test execution time. The autopkgtest logs and raw test output are also available for each run. In the test run linked to above, the test session seems to have failed due to a configuration problem in the test, rather than a bug in Apt; debci will also mark some problems as tmpfail situations, such as this evolution-data-server test for which the dependencies could not be satisfied.

Terceiro noted in his blog post that the number of packages including test suites has grown from around 200 to its current level of about 350 since the server was first brought online in January. In addition—and perhaps more importantly—the percentage of packages passing all of their own tests has grown considerably over that same time frame, from less than half to over 75%.

Of course, 350 packages is a drop in the bucket compared to 40,000, but one must start somewhere. The fact that the number of package maintainers who are participating is on the rise, coupled with the fact that more tests are passing, suggests that debci is proving to be a useful tool for the Debian project. Terceiro also noted in his blog post that there are now multiple contributors to the project, including current autopkgtest maintainer Martin Pitt and two Google Summer of Code (GSoC) students. It may still take quite a while for debci to achieve majority coverage (much less full coverage) of Debian's vast package archive, but with easy-to-see results, its popularity might also catch on in hurry.