Tools for porting drivers

Out-of-tree drivers are a maintenance headache, since customers may want to use them in newer kernels. But even those drivers that get merged into the mainline may need to be backported at times. Coccinelle developer Julia Lawall introduced the audience at Open Source Summit Europe to some new tools that can help make both forward-porting and backporting drivers easier.

She opened her talk by noting that she was presenting step one in her plans, she hoped to be able to report on step two next year some time. The problem she is trying to address is that the Linux kernel keeps moving on. A vendor might create a driver for the 4.4 kernel but, over the next six months, the kernel will have moved ahead by another two versions. There are lots of changes with each new kernel, including API changes that require driver changes to keep up.

That means that vendors need to continually do maintenance on their drivers unless they get them upstream, where they will get forward-ported by the community. But the reverse problem is there as well: once a device becomes popular, customers may start asking for it to run with older kernels too. That means backporting.

There is an obvious methodology to the porting process, Lawall said. Compile the driver with the target kernel and see what breaks. The compiler will likely complain about various things that point to where things are broken. Since the driver is unchanged, any breakage that is reported are from places where the kernel has changed; GCC will point toward the parts of the driver that need to be fixed.

In order to figure out what needs to be done, one can look at other commits in the code history that update the code for those kinds of changes. If the driver is for a webcam, for example, it is probably not the only such driver that has needed to be updated for these changes. To fix the target driver, one can look at other webcam drivers to see what they did, then apply those changes to the target.

The assumption here is that if the driver compiles with the kernel, then it works. That is not always the case, Lawall said, but getting it to compile is progress toward getting it to work.

She showed an example (slides [PDF]) of the lms501kf03 TFT LCD panel driver. It was introduced in Linux 3.9 and she showed how this process works by trying to port it to the 4.6 kernel. First she compiled it with 4.6, which led to a number of GCC complaints. She noted that only two of the errors were significant; those related to two unknown fields that were being initialized in the driver (suspend and resume in struct spi_driver).

She looked for other commits that had removed those two fields and found a commit in the as3935 driver that did so. Looking at the patch, she noted that those functions had moved into a new spi_driver field called pm (which is in the driver sub-structure that is part of spi_driver). She further noted that the functions assigned to suspend and resume had changed their signature; they now took different parameters. She went through the process of seeing how the as3935 driver had changed to accommodate the underlying kernel changes.

Those changes that were made to the as3935 driver can be considered to see if they can be made to the lms501kf03 driver. If they can, then it is fairly straightforward to remove those fields, change the functions, and populate the structure properly for the 4.6 kernel. That is the methodology that she is trying to support with tools.

There are a number of challenges to overcome to get there. The error messages from GCC are redundant and inconsistent. Problems earlier in the code may lead to strange syntax errors reported that muddy the waters, for example. The significant errors can end up requiring multiple changes in the code (as shown by the example). In addition, it may require finding multiple different example commits to fix the problems found. And, in finding those commits, it is important to choose the right ones; using git log with the -g or -s options will bring up a large number of commits that are not relevant. There are lots of structures in the kernel with suspend and resume fields, but the ones of interest for the example are only those in spi_driver structures.

So how can these examples be found? Her proposed solution has two pieces (as yet): a tool to reduce the noise from GCC and a tool to find example commits of interest. The first is called gcc-reduce; it takes the output of GCC, removes the redundant error messages and collects complementary information for those errors from the source code (e.g. the type of a structure that is missing a field). It would be easier to do this work for LLVM, she said, but kernel development is focused on GCC, so that's where the effort has gone.

The second piece is called Prequel and is used to find the commits that provide example solutions to porting issues. In the future, she would like to add a way to auto-generate patches based on the examples found. Prequel, which is related to Coccinelle, uses a notation that will be familiar to Linux developers and others who have used Coccinelle. It effectively incorporates diff-like notation into Coccinelle rules, so a query for patches that remove the suspend field from an spi_driver structure would be as follows:

    @@
    identifier i;
    expression e;
    @@

    struct spi_driver i = {
    - .suspend
	     = e,
    };

Unlike Coccinelle, though, Prequel is simply doing pattern matching, not transformation. The query can be fed to Prequel along with a range of kernel commits and it will produce a list of all of the commits that match the pattern. It will list the commit IDs along with a percentage that indicates how much of the commit was matched by the rule. It can produce that as a text file, but there is also integration with Emacs Org mode and with Vim.

There are some 70,000 kernel commits each year, so it is not practical to extract and match on each one for each query. In order to have reasonable performance, indexes have been created of "words" on and near changed lines.

In general, learning the notation is not needed for driver porting as there are query templates that suffice for the typical cases. There are generally just a small set of porting issues that need to be addressed. Those include undefined variables, functions, fields, or types; the wrong number of arguments to a function or macro; and incompatible types in assignment, initialization, or as function arguments.

In order to evaluate the tools, drivers were ported both forward and back to and from the 4.6 kernel. The forward ports were from 2013 (13 drivers) and from 2015 (ten drivers), while ten backports from 4.6 to the original 2013 version were done. In all of those, 107 issues were reported by gcc-reduce and the Prequel templates were used to find the right changes for 80 of them. For 86% of those 80 issues, the first commit returned by Prequel contained the information of interest.

There are some limitations to the approach. It assumes that each issue is addressed in a single commit, which was not the case for five issues that were found in the experiment. For example, a field may get renamed temporarily (e.g. x to x_new); Prequel finds the first change, but not when x_new gets renamed back to x. Iteration is a possible solution to that problem. In some cases, gcc-reduce can misidentify the root cause of a problem, which happened for six of the issues in the evaluation.

Backporting is typically harder than forward porting, she said. Some of the commits in the history may actually be going from good code to good code (e.g. a cleanup of some kind) or just be fixing a bug. Those can confuse the queries, so there may need to be some more work done there.

The overall project web page has information on the tools and experiments. There are plans to improve the commit-ranking algorithm and to better infer the changes from the commits that are identified. In the Q&A, one audience member wondered if looking at a patch series, rather than an individual patch, might help identify the changes needed. Lawall agreed that it probably would, but that information is not carried in the Git commit stream; it could potentially be synthesized using the author and date information.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for supporting my travel to Prague for OSS Europe.]