Continuous-integration testing for Intel graphics

Two separate talks, at two different venues, give us a look into the kinds of testing that the Intel graphics team is doing. Daniel Vetter had a short presentation as part of the Testing and Fuzzing microconference at the Linux Plumbers Conference (LPC). His colleague, Martin Peres, gave a somewhat longer talk, complete with demos, at the X.Org Developers Conference (XDC). The picture they paint is a pleasing one: there is lots of testing going on there. But there are problems as well; that amount of testing runs afoul of bugs elsewhere in the kernel, which makes the job harder.

Developing for upstream requires good testing, Peres said. If the development team is not doing that, features that land in the upstream kernel will be broken, which is not desirable. Using continuous-integration (CI) along with pre-merge testing allows the person making a change to make sure they did not break anything else in the process of landing their feature. That scales better as the number of developers grows and it allows developers to concentrate on feature development, rather than bug fixing when someone else finds the problem. It also promotes a better understanding of the code base; developers learn more "by breaking stuff", which lets them see the connections and dependencies between different parts of the code.

CI also helps keep the integration tree working at all times. That means developers can rebase frequently so that their patches do not slowly fall behind as the feature is developed. They don't have to fight with breakage in the integration tree because it is tested and working. The rebases are generally small, since they are frequent, which allows catching and fixing integration and other problems early on.

There are a number of objectives of the CI testing that Intel does. It is meant to provide an accurate view of the current state of the hardware and software. In order to do that, there are some requirements on the test results. They must include all of the information needed to diagnose and reproduce any problems found including the software and hardware configurations and log files. They also need to run quickly so that developers can get fast feedback on a proposed change. To that end, Intel has two levels of testing, one that gives results in 30 minutes or less and another that is more complete but takes up to half a day to complete.

The results need to be visible. Publishing the test results on a web site would work, he said, but it is better to make the results hard to miss. If the patch being tested is something that was posted to a mailing list, the results should be posted as a reply. "Put the data where the developer is." In addition, false positive test failures make developers less apt to believe in the test results, so they must be avoided. Any noise in the results needs to be aggressively suppressed.

Vetter went into some detail of the actual tests that are being run. The fast test suite uses IGT, which consists of mostly Intel-specific tests, on actual hardware. There are 250 test cases that run on 30 machines. That test takes about ten minutes to run, but may not complete that quickly depending on the load on the testing systems.

The next step is the full test suite, which takes six hours of machine time. It tests against multiple trees, including the mainline, linux-next, and various direct rendering manager (DRM) subsystem trees (drm-tip, fixes, next-fixes). Those tests run with lockdep and the kernel address sanitizer (KASAN) enabled. There is a much bigger list of a few thousand test cases that gets run as well, he said. The tests, results, and more are all available from the Intel GFX CI web page.

Pre-merge testing is one of the more interesting and important parts of the test system. It picks up patches from the mailing lists using a modified version of Patchwork and runs the test suites on kernels with those patches. If the fast tests pass, that kicks off the full test; meanwhile the patches can start to be reviewed. Metrics that measure things like test system usage, test latency compared to the 30-minute and six-hour objectives, bugs found, and so on are also measured and reported. The test matrix (example) was "much smaller and full of red" a year ago, Vetter said. "CI is definitely awesome".

Peres filled in some more of the details. He said there were 40 different systems (up from Vetter's 30) and 19 different graphics platforms. Those range from Gen 3 from 2004 to the announced, but not yet available, Gemini Lake. There are a number of "sharded" machines, which are six or eight duplicate systems that can be used to parallelize the testing. The number of tests run has increased from around 22,000 per day in August 2016 to 400,000 per day in August 2017.

There are some things that have been learned along the way, Vetter said. Noise, in the form of false positives, will kill any test system. "Speed matters"; if it takes three days to get initial results, people will ignore them. In addition, Vetter said that pre-merge testing is the only time to catch regressions. Once a feature has been merged, it is impossible to get it reverted because people get attached to their work, but they don't always fix any regressions in a timely manner.

Linux-next is difficult to test with because it requires lots of fixup patches and reverts to get to something that will function. Part of the Intel testing does suspend-resume cycles on the machines which finds a lot of regressions in other subsystems, Vetter said. Greg Kroah-Hartman suggested posting those regressions as a reply to the linux-next announcement, but Vetter said there would be way too much noise with that approach.

Beyond that, trees like linux-next make bisecting problems way too hard. It takes "a lot of hand holding" to make bisect work so that these regressions can be found. It takes more work to ask the subsystem maintainer to get them fixed, so the Intel team ends up reverting things or changing the configuration to avoid those problems. They do try to report them to the maintainers, but the root of the problem is that some subsystem maintainers put unready code into linux-next in the hopes that others will test it for them; that makes it less than entirely useful as an integration tree.

The problems are generally outside of the graphics subsystem and are often exposed by the thousands of suspend-resume cycles that are part of the Intel graphics testing. The 0-Day test service does not do suspend and resume testing, though Kroah-Hartman suggested that it be added if there is an automatic way to test it. Regressions are hard to get fixed even in graphics, Vetter said, and reverts are difficult to sell. That's why pre-merge testing to find problems before the code gets merged is so important.

Peres also had a list of lessons learned, some of which, unsurprisingly, overlapped Vetter's. For one thing, kernel CI testing is unlike CI testing for any other project because it requires booting the machine, sometimes with broken kernels. In the year since the project started, they have realized that anything that is not being continuously tested is likely broken. Once again, this is something that is part and parcel of kernel testing because there are so many different configuration options, many of which cannot even be tested without the right hardware.

New tools were needed as Bugzilla is not a good tool to track test failures. Peres has been working on a CI bug log tool to fill that gap. He hopes to release the code for it by the end of the year once the database format has stabilized. It is also important that the developers own the CI system and that the CI team works for them. It should not be a separate team that reports to a different manager outside of the development team. As the developers start to see the value of the CI system, they will suggest improvements to the system and the tests that will help make the testing better.

Other graphics teams that have an interest in replicating the Intel work should have an easier time of it because much of the groundwork has been laid, Peres said. There is still a need for infrastructure and hardware, of course, along with software for building kernels, deploying them, scheduling test jobs, and the like. Several components are particularly important, including ways to power cycle systems and to resume them from suspend—an external watchdog will also help by restarting systems that are no longer responsive. There is a need for qualified humans both to create bugs and to respond quickly to those bugs and fix them.

There are a number of challenges that are specific to CI testing for the kernel, Peres said. The first is that various kernels will not boot or function properly; it could be the network drivers, filesystem corruption, or something else that may be difficult to automatically diagnose. Getting tracing and log information out of the system, especially during suspend and resume failures or if there is a random crash while running the tests, can be difficult. Using pstore for EFI systems and serial consoles for the others will provide a way to get information out of a broken system. Note that memory corruption can lead to all sort of nastiness, including trashing the partitions of the disk, so an automated way to re-deploy the system will be needed.

Slides [PDF] and a YouTube video of Peres's presentation are available for interested readers.

[I would like to thank the Linux Foundation and the X.Org Foundation for travel assistance to Los Angeles for LPC and Mountain View for XDC.]