Kernel development

Kernel release status

The current 2.6 development kernel is 2.6.29-rc2, released by Linus just before heading out the door to travel to linux.conf.au. "But despite being small, that's probably more important and noticeable to most people: the first slew of regression fixes. We had non-working 3D acceleration on many machines (no compiz! What shall we do without those wobbly windows!) that should be fixed, and a ton of other irritating issues like that." See the short-form changelog for details, or the full changelog for lots of details.

As of this writing, no changeset have been merged into the mainline repository since 2.6.29-rc2. Linus may be a bit busy with his new barbershop career (he agreed to shave Bdale Garbee's beard in response to a challenge at the linux.conf.au "penguin dinner" charity auction) to merge patches for a little while yet.

The current stable 2.6 kernel is 2.6.28.1, released with a long list of fixes on January 18.

For 2.6.27 users, 2.6.27.12 was also released on the 18th. Previously, 2.6.27.11 was released on January 14. Greg Kroah-Hartman says that he plans to maintain 2.6.27 as a relatively long-term release, and Adrian Bunk plans to pick it up thereafter, so fixes should be available for this kernel for quite some time.

Comments (none posted)

Quotes of the week

-- Andrew Morton, worried about adaptive mutexes.

-- Linus Torvalds

-- Andrew Morton on merging SLQB

Comments (2 posted)

LCA: The security panel

The linux.conf.au security miniconf hosted a number of talks on specific security technologies, many of which have been covered here in the past. The final event of the day, though, was a panel discussion covering a wide variety of security issues. Panellists Casey Schaufler (who also doubled as moderator), Russell Coker, James Morris, Z. Cliffe Schreuders, and Kentaro Takeda discussed module stacking, increasing the use of security technology, authoritative hooks, and more.

Module stacking was the first topic of interest. "Stacking" refers to the practice of loading more than one security module, allowing each of them to participate in security decisions. The technique has its appeal; it would allow more tightly-focused modules to be written and mixed together in interesting ways. But stacking of security modules is not currently supported in the Linux kernel - a situation which does not appear to be set to change anytime soon.

Casey, who had raised the issue, answered his own question by saying that he would like to see module stacking supported; it would add to the flexibility of the system. His preferred solution would involve the creation of a special security module which would arbitrate between all the others, deciding which modules get to make decisions in each specific situation. As far as your editor knows, this stacker module does not actually exist yet.

Russell's response was simpler: he would like to see a reasonable number of users actually running with a single security module first. Once that problem has been solved, one can move on to more complicated tasks.

James raised the issue of the "composability problem": the combination of security technologies in ways not anticipated by their designers can lead to unpredictable results. People working on security technologies hate unpredictable results. The SELinux developers tried to deal with this problem by turning SELinux into the one true security module (your editor's term, not James's), so that any security situation could be addressed within a single module. This aspect of SELinux is not really being used, though.

Cliffe's response was that stacking should be allowed, if only to discourage developers from adding parallel sets of hooks to support their own security technologies. James responded, though, that many security-related modules (integrity management, say, or malware scanning) really should have their own API. Kentaro noted that the TOMOYO Linux developers really want their work to be able to coexist with other modules, so he would like to see stacking supported as well.

From there, your editor asked the panelists to follow up on Russell's point: what is it going to take to get people to actually use the security technologies that we have now? A security module does little good if frustrated system administrators simply turn it off as soon as it gets in their way. Casey responded that it was unfortunate that the first security module made available for widespread use (SELinux) was such a complex one. A lot of people really don't need all of the capability which is provided by SELinux. A set of smaller, more understandable security modules would have gained acceptance much more easily. SELinux is far too monolithic; there is no easy way into it.

Russell, instead, suggested that we should look at the history of security. Once it was accepted that all important processes would run as root. Over time, it has been made clear that this is a bad idea, and various system daemons have been moved to other user IDs. Ill-advised practices like running IRC clients as root have been banned. It has taken a long education process to get to this point; this process will have to continue for technologies like SELinux. James agreed that time was required, and noted that, over time, use of SELinux is increasing. Some simple things, like getting administrators to shift SELinux to permissive mode when they run into problems rather than turning it off altogether, can also help in this regard.

In the longer term, though, there is still a need for higher-level tools. The current SELinux policy interface is really the assembly language of (SELinux-based) security; most users should not have to deal with the system at that level. Cliffe agreed, saying that blaming users for turning off security is the wrong thing to do. It is the fault of the security developers, who have not made their tools sufficiently easy to use. Security must be built using higher-level abstractions which users can understand; the technology he is working on (FBAC-LSM), is designed with this goal in mind. Kentaro added that most users don't want to have to think about security; it needs to be implemented so that they don't have to.

From there the panellists went into a rather cloudy discussion of cloud computing. James, after asking what that term really meant, noted that there are useful things to be done in this area, and that Linux offers a number of useful technologies, such as namespaces, which can help. There is, though, a lot of work to be done. Cliffe added that the infrastructure is there for people who want to work on secure cloud computing, but that module stacking would make it easier. Kentaro stated that cloud computing is, in fact, one of the core targets for his work; there is a lot of space for Linux here. We do, however, need to be sure to avoid creating single points of failure, which can bring the whole thing down. Casey's take on this topic was that cloud computing is likely to bring cryptography back to the forefront of security research; when all of your data is on other people's servers, that data needs to be well protected.

Russell took a different tack, noting that the security of a number of current cloud offerings is substandard. They often provide distributions which no longer receive security support, and they provide lots of unpatched software. They are insecure by default and "ripe for harvesting," but it is not easy, in such environments, for even a relatively high-clue user to figure out how to secure things. The real problem, he says, is that there is no business model for better security, so "cloud" providers are not investing in that area.

A member of the audience asked about authoritative hooks. These hooks were a contentious issue early in the development of the Linux security module architecture. LSM is current designed to allow restrictive hooks only: a security module can only make policy tighter than basic Linux discretionary access control would allow. The thinking is that, with restrictive hooks only, a buggy security module cannot make things worse than they were before. Authoritative hooks would, instead, let a security module empower a process to do things which would not otherwise be allowed.

[PULL QUOTE: This policy has not slowed down proprietary security modules, and, at this point, a model allowing authoritative hooks would be better. Making that change would be "a really big deal," though. END QUOTE] Casey reiterated the history behind the current "no authoritative hooks" policy, adding that the kernel developers also feared that authoritative hooks would make the LSM API more suitable for abuse by binary-only modules. Indeed, he says that was the primary reason for disallowing those hooks. But this policy has not slowed down proprietary security modules, and, at this point, a model allowing authoritative hooks would be better. Making that change would be "a really big deal," though. Russell agreed that the "irrational fear" of authoritative hooks remains widespread, but the reassurance provided by their absence may be worth it in the end. Both Cliffe and Kentaro thought that interesting things could be done with authoritative hooks, and that it would be a good time to review just how Linux security modules work.

There was a brief discussion on the feeling that the LSM API is too heavily oriented toward the needs of SELinux. James agreed that it was "SELinux-shaped," but noted that this was a natural result of the fact that SELinux has been the only user of the API for most of its history. Casey noted that things have recently been changed to support the needs of his SMACK module. There have also been some new hooks added to support pathname-based modules like TOMOYO Linux and AppArmor.

Going back to another point raised by Russell, a member of the audience asked what distributions should do once they go past their end of life. Should a system with an unsupported kernel refuse to boot, or, at least, refuse to bring up network interfaces? Russell came back with the obvious response: how would one then update such a system? Casey pointed out that there are an awful lot of routers out there running old, unsupported software. The Internet, he says, is made of expired systems. Russell suggested that ISPs should, perhaps, enforce the use of supported software, and that, maybe, governments could compel such behavior. Cliffe noted that all of this really poses another usability problem; what we really need to do is to make it easy to run a current system. Quite a bit of progress has already been made in this direction.

The final topic had to do with "security mythology," things that "everybody knows" improve security but which really don't. Forced password rotation was one such idea. Casey said that, for some 20 years, everybody "knew" that security meant strong cryptography. There's no real way to address such things except as a people problem. Russell added that there's often no way to know what the consequences of security rules are. James said that there is a real need for technical people to push back against silly security rules. He likened the problem to the early adoption of Linux, where people with clue simply deployed it, then asked for forgiveness later. Cliffe's point of view is that users do not really know when they are being asked to make security decisions, so they don't really know when their actions may be putting their security in peril. And Kentaro agreed, noting that we need to find ways to provide more information to users about what their security technology is really for.

Thereafter the panel broke up, and the PGP key signing party (done, no doubt, in a highly secure manner) began.

Comments (6 posted)

New NFS to bring parallel storage to the masses

Sometime around the end of January or early February, the Internet Engineering Task Force will give its final blessing to the latest version of the venerable Network File System (NFS), version 4.1. While the authors of the standard have stressed that this is a minor revision of NFS, it does have at least one seemingly radical new option, called Parallel NFS (pNFS).

The "parallel" tag of pNFS means NFS clients can access large pools of storage directly, rather than go through the storage server. Unbeknown to the clients, what they store is striped across multiple disks, so when that data is needed it can be called back in parallel, cutting retrieval time even more. If you run a cluster computer system, you may immediately recognize the appeal of this approach.

"We're starting the process of feeding all these patches up to the Linux NFS maintainers," said Brent Welch, the director of software architecture for Panasas who is also one of that storage company's contributors of the pNFS code. He noted that the work for the prototyping and implementing pNFS in Linux, as part of NFS, has been going on for about two years. Ongoing work has included updating both the NFS client and NFS server software.

The code will be proposed for the Linux kernel in two sets, according to Welch. The first set will have the basic procedures for setting up and tearing down pNFS sessions, using Remote Procedure Call (RPC) operations for exchanging IDs and initiating and ending sessions. The development teams are gunning to have this basic outline of pNFS included in the 2.6.30 version of the kernel. The second set, ready for the 2.6.31 version of the kernel, will be a larger patch, including the I/O commands for accessing and changing file layouts as well as reading and writing data. Given that it will take a few more months after the 2.6.31 Kernel for it to be picked up by the major distributions, pNFS probably won't start to be deployed by even the most ambitious IT shops at least until the early part of 2010.

We all know NFS. It allows client machines to mount Unix drives that reside across the network as if they were local disks. Many Network Attached Storage (NAS)-based storage arrays use NFS. With NAS, a lot of hard drives all lie behind a single IP address, the drives are all managed by the NAS box. NAS allows organizations to pool storage, so storage administrators can more fluidly (and hence efficiently) allocate that storage across all users.

In a 2004 problem statement, two of the developers responsible for getting pNFS in motion, Panasas chief technology officer Garth Gibson and Network Appliance (NetApp) engineer Peter Corbett, explained the limitations of this approach, especially in high performance computing environments:

In a nutshell, the potential roadblock with NAS, or any type of NFS-based network storage, is the NAS head, or server, they explained. If too many of your clients hit the NAS server at the same time, then the I/O slows for everyone. You could go back to direct access, but you lose the efficiencies of pooled storage. For cluster computer systems, in which dozens of nodes can be working on the same data set, such partitioned storage just isn't feasible. Nor are multiple storage servers: An NFS-based system can not support multiple servers writing to the same file system.

Gibson and Corbett were early champions of developing pNFS, along with Los Alamos National Laboratory's Gary Grider. Additional work was carried out by engineers at EMC, Panasas, NetApp and other companies. The University of Michigan's Center for Information Technology Integration (CITI), along with members of the IBM Almaden Research Center are developing a pNFS implementation for Linux, both for clients and storage servers.

pNFS will allow clients to connect directly to the storage devices they need, rather than go through a storage gateway of some sort. The folks behind pNFS like to say that their approach separates the control traffic from the data traffic. When a client requests a particular file or block of storage, it sends a request to a server called the Metadata Server (MDS), which returns a map of where all the data resides within the storage network. The client can then access that data directly, according to permissions set by the file system. Once that storage is altered, the client notifies the MDS of the changes, which updates the file layout.

Since pNFS allows clients to talk directly to the storage devices, as well as permitting client data to be striped across multiple storage devices, the client can enjoy a higher I/O rate than would be had simply by going through a single NAS head—or by communicating with a single storage server. In 2007, three developers from the IBM Almaden Research Center, Dean Hildebrand, Marc Eshel and Roger Haskin, demonstrated [PDF] at the Supercomputing 2007 conference (SC07) how three clients could saturate a 10 gigabit link by drawing data from 336 Linux-based storage devices. Such throughput "would be hard to achieve using standard NFS in terms of accessing a single file," Hildebrand said. "We wanted to show that pNFS could scale to the network hardware available."

pNFS is largely made up of three sets of protocols. One protocol is for the mapping, or layout, of resources, which resides on the client. It interprets and utilizes the data map returned from the metadata server. The second is the transport protocol, which also resides on the client. It coordinates data transfer between the clients and storage devices. The transport protocol handles the actual I/O with the storage devices. A control protocol will synchronize the metadata server with the storage devices. This last protocol is the only one not specified by NFS—It will be left to storage the vendors, though much of the work that this protocol will do can be codified in NFS commands.

pNFS can work with three types of storage—file-based storage, object-based storage and block-based storage. The NFSv4.1 protocol itself contains the file-based storage protocol. Additional RFCs are being developed for object and block protocols. File-based storage is what most system administrators think of as storage; it is the standard approach of nesting files within a hierarchical set of directories. Block-based storage is used in Storage Area Networks (SANs), in which the applications access disk space directly, by sending the Small Computer System Interface (SCSI) commands over Fibre Channel, or, increasingly of late, TCP/IP via the Internet SCSI (iSCSI) protocol. Object-based storage is somewhat of a newer beast, a parallel approach that involves embedding the data itself with self-describing metadata.

A word on semantics: Keep in mind that just as NFS is not a file system itself, neither is pNFS. NFS provides the protocols to work with remote files as if they were local. Likewise, pNFS offer the ability to work with files managed by a parallel file system as if they were on a local drive, handling such tasks as setting permissions and ensuring data integrity. Fortunately, a number of parallel file systems have been spawned over the past few years that should work easily with pNFS. On the open source front, there is the the parallel Virtual File System (pVFS). Perhaps the most widely-used open-source parallel file system now in use is Lustre, now overseen by Sun Microsystems. On the commercial front, Panasas' PanFS file system has been successfully deployed in high performance computer clusters, as has IBM's General Parallel File System (GPFS). All of these approaches use a similar idea—let the clients talk to the storage server's devices directly, while having some form of metadata server keep track of the storage layout. But most other options rely on using a single vendor's gear.

"The main advantage [to using pNFS] is expected to be on the client side", noted CITI programmer J. Bruce Fields, who does the NFS 4.1 testing on Linux servers. With most parallel file systems you have to do some kernel reconfigurations on the clients so that they can work with the file systems. With the prototype Linux client, you can run a standard mount command and get the files you need. "The client will automatically negotiate pNFS and find the data servers. By the time we're done that should work on any out-of-the-box Linux client from the distribution of your choice", he says.

The advantage that pNFS will bring is familiarity, and that it will come already built in as part of NFS. Since NFS is a standard component in almost all Linux kernel builds, that will greatly reduce the amount of work administrators need to do to set up a parallel file system for Linux servers. Most administrators are more familiar with the general operating procedures of NFS, much more so than dealing directly with, say, Lustre, which requires numerous kernel patches and a different mindset when it comes to understanding commands.

pNFS should help storage vendors as well, as they will not have to port client software to numerous Linux distributions. Welch, for instance, noted that Panasas has to maintain code for dozens of different Linux distributions. Instead, they can rely on NFS and focus on storage devices. Already, Panasas, NetApp, EMC, IBM and have all promised [PDF] to support pNFS in at least some of their storage products, according to a collective talk some of the developers gave last month at the SC08 conference. Sun Microsystems also plans to support pNFS in Solaris.

And while much of the early focus of pNFS has been for large scale cluster operations, one day it may be feasible that even workstations and desktops will use pNFS in some form. LANL's Gary Grider pointed out that, "at some point, having several teraflops may even be possible in your office, in which case you may need something more than just NFS for data access for such a powerful personal system. pNFS may end up being handy in this environment as well."

Indeed. Once upon a time we were limited to working on files on our own machines, FTP'ing in anything that was located elsewhere. But NFS allowed us to mount drives across the network with a relatively simple command. Now, pNFS may take simplify things a step further, by allowing to us to pull in and write large files or myriad files with a speed that we can now only dream about. At least that is the promise of pNFS.

Comments (11 posted)

Semantic patching with Coccinelle

We've all been there: You're tracking down some evil bug, and you have the sudden chilling realization that you're going to have to refactor an enormous chunk of code to fix it. You break out in a cold sweat as you run a quick grep over the source base: hundreds of lines of code to change! And the change is too complex to do with a script because it depends on the calling context, or requires adding a new variable to every caller.

This happened to me last month when I was adding support for 64-bit file systems to e2fsprogs. I thought I was nearly finished when I discovered I needed to write (yet another) new interface and convert (yet another) several hundred lines of code to it. The changes were complex enough that I couldn't use a script, and simple enough that I wanted to claw my eyes out with the soul-killing boredom of doing it by hand. That's when the maintainer, Theodore Ts'o, suggested I look at Coccinelle (a.k.a., spatch).

Coccinelle

Coccinelle is a tool to automatically analyze and rewrite C code. Coccinelle (pronounced cock'-see-nel) means "ladybug" in French, a name chosen because ladybugs eat other bugs. Coccinelle is not just another scripting language; it is aware of the structure of the C language and can make much more complex changes than are possible with pure string processing. For example, Coccinelle can make a particular change only in functions which are assigned to a function pointer in a particular type of array — say, the create member of struct inode_operations.

The input to the tool is the file(s) to be changed and a "semantic patch," written in SmPL (Semantic Patch Language). SmPL looks a like a unified diff (a patch) with some C-like declarations mixed in. Here's an example:

    @@
    expression E;
    identifier fld;
    @@

    - !E && !E->fld
    + !E || !E->fld

This semantic patch fixes the bug in which the pointer is tested for NULL — and then dereferenced if the pointer is NULL. An example of a bug this semantic patch found in the Linux kernel (and automatically generated the fix for):

    --- a/drivers/pci/hotplug/cpqphp_ctrl.c
    +++ b/drivers/pci/hotplug/cpqphp_ctrl.c
    @@ -1139,7 +1139,7 @@ static u8 set_controller_speed(struct controller
    *ctrl, u8 adapter_speed, u8 hp_
            for(slot = ctrl->slot; slot; slot = slot->next) {
                    if (slot->device == (hp_slot + ctrl->slot_device_offset))
                            continue;
    -               if (!slot->hotplug_slot && !slot->hotplug_slot->info)
    +               if (!slot->hotplug_slot || !slot->hotplug_slot->info)
                            continue;
                    if (slot->hotplug_slot->info->adapter_status == 0)
                            continue;

(More on the semantic patch format later.)

Coccinelle is designed, written, and maintained by Julia Lawall at the Department of Computer Science at University of Copenhagen, Gilles Muller and Yoann Padioleau at the Ecole des Mines de Nantes, and René Rydhof Hansen at the Department of Computer Science of Aalborg University. Coccinelle is licensed under the GPL, however, it is written in OCaml, so the potential developer base is somewhat limited.

The original goal of Coccinelle was to automate as much as possible the task of keeping device drivers up to date with the latest kernel interfaces. But the end result can do far more than that, including finding and fixing bugs and coding style irregularities. Over 180 patches created using Coccinelle have been accepted into the Linux kernel to date.

Coccinelle quickstart

Like many languages, SmPL is best learned through example. We'll run through one simple example here just to get started. After that, the Coccinelle web page has some documentation and a plethora of examples.

First, download Coccinelle and install it. I used the source version rather than any of the precompiled options. The Coccinelle binary is called spatch.

As our example, say we have program with a lot of calls to alloca() that we would like to replace with malloc(). alloca() allocates space on the stack and can be more efficient and convenient than malloc(), but it is also compiler-dependent, non-standard, easy to use incorrectly, and has undefined behavior on failure. (Replacing alloca() with malloc() isn't enough, we also have to check the return value — but that will come later.)

Here is the C file we are working on:

    #include <alloca.h>

    int
    main(int argc, char *argv[])
    {
            unsigned int bytes = 1024 * 1024;
            char *buf;

            /* allocate memory */
            buf = alloca(bytes);

            return 0;
    }

We could make the replacement using a scripting language like sed:

$ sed -i 's/alloca/malloc/g' test.c

But this will replace the string "alloca" anywhere it appears. The resulting diff:

    --- test.c
    +++ /tmp/test.c
    @@ -1,4 +1,4 @@
    -#include <alloca.h>
    +#include <malloc.h>
 
     int
     main(int argc, char *argv[])
    @@ -6,8 +6,8 @@
             unsigned int bytes = 1024 * 1024;
             char *buf;
 
    -        /* allocate memory */
    -        buf = alloca(bytes);
    +        /* mallocte memory */
    +        buf = malloc(bytes);
 
             return 0;
     }

We can tweak our script to handle 90% of the cases:

    $ sed -i 's/alloca(/malloc(/g' test.c

But this script doesn't handle the case where a second function name has the first as a suffix, it depends on a particular coding style in which no white space comes between the function name and the open parenthesis, etc., etc. By now our simple sed script is a hundred-character monster. It can be done, but it's a pain.

In Coccinelle, we'd use the following semantic patch:

    @@ expression E; @@

    -alloca(E)
    +malloc(E)

Put the C file in test.c and the above semantic patch in test.cocci and run it like so:

    $ spatch -sp_file test.cocci test.c

It should produce the following diff:

    --- test.c
    +++ /tmp/cocci-output-17416-b5450d-test.c
    @@ -7,7 +7,7 @@ main(int argc, char *argv[])
             char *buf;
 
             /* allocate memory */
    -        buf = alloca(bytes);
    +        buf = malloc(bytes);
 
             return 0;
     }

Let's look at the semantic patch line by line.

    @@ expression E; @@

This declares the "metavariable" E as a variable that can match any expression — e.g., 1 + 2, sizeof(x), strlen(name) + sizeof(x) * 72. When spatch processes the input, it sets the value of E to the argument to alloca(). The "@@ @@" syntax is chosen to resemble the line in a unified diff describing the lines to be patched. I don't find the resemblance particularly helpful, but the intention is well-taken.

    -alloca(E)

This line says to remove any call to the function alloca(), and to save its argument in the metavariable E for later use.

    +malloc(E)

And this line says to replace the call to alloca() with a call to malloc() and use the value of metavariable E as its argument.

Now, we also want to check the return value of malloc() and return an error if it failed. We can do that too:

    @@
    expression E;
    identifier ptr;
    @@

    -ptr = alloca(E);
    +ptr = malloc(E);
    +if (ptr == NULL)
    +        return 1;

The resulting diff:

    --- test.c
    +++ /tmp/cocci-output-17494-22a573-test.c
    @@ -7,7 +7,8 @@ main(int argc, char *argv[])
             char *buf;
 
             /* allocate memory */
    -        buf = alloca(bytes);
    +        buf = malloc(bytes);
    +        if (buf == NULL)
    +                return 1;
 
             return 0;
 }

Semantic patches can be far more complex. One of my favorite examples is the move of reference counting of the Scsi_Host structure out of drivers. Changing this required adding an argument to the function signature and removing a declaration and several other lines from each SCSI driver's proc_info function. The semantic patch, explained in detail in their OLS 2007 slides [PPT] [ODP], does all of this automatically. I recommend reading and re-reading this example until it sinks in.

Experience

My first experience with Coccinelle was mixed. In theory, Coccinelle does exactly what I want — automate complex changes to code — but in practice the implementation is beta quality. I successfully used Coccinelle to make hundreds of lines of changes with less than a hundred lines of semantic patches, but only after working directly with the developers to get bug fixes and help figuring out SmPL features. Coccinelle is one of those schizophrenic projects situated on the boundary between academic research and practical software development.

One of the first hurdles I had to overcome was teaching Coccinelle about the macros in my code. Coccinelle has to do all its own parsing and pre-processing — you can't just run the input C code through cpp because then you'd have to map the post-processor output back to the original code. Macros will sometimes confuse it enough that it gives up parsing a function until it reaches the next safe grammatical starting point (e.g., the next function) — which may mean that it doesn't process most of the file. To get around this, you can create a list of macros and feed them to spatch with the -macro_file option. (Yes, that's one dash — one of my pet peeves about Coccinelle is the non-standard command-line option style.) For example, here are a few lines from the macro file I used for e2fsprogs:

    #define EXT2FS_ATTR(a)
    #define _INLINE_ inline
    #define ATTR(a)

You can build the list of macros by hand, but spatch has a feature that helps find them automatically. The -parse_c option makes spatch list the top ten parsing errors, which will include the macro name. For example, some of the output from running spatch -parse_c on e2fsprogs:

    EXT2FS_ATTR: present in 85 parsing errors
    example:

          static int check_and_change_inodes(ext2_ino_t dir,
                                      int entry EXT2FS_ATTR((unused)),
                                      struct ext2_dir_entry *dirent, int
                                      offset,
                                      int  blocksize EXT2FS_ATTR((unused)),

Coccinelle has improved significantly in the past few weeks. The 0.1.2 release had a number of bugs that made spatch unusable for me. The next release, 0.1.3, fixed those bugs and with it I was able to make practical, real-world patches. The 0.1.4 release will be out shortly. The developers wrote and released more documentation, including a description of all the command-line options [PDF] and a grammar for SmPL. Many more example spatch scripts are available now. The best reference for learning Coccinelle continues to be the slides from their 2007 OLS tutorial and the associated paper [PDF]. White space handling is improving; originally Coccinelle didn't care much about white space and frequently mangled transformations involving it, which is a problem if you want to take the hand out of hand-editing. One of my semantic patches left a dangling semi-colon in the middle; the developers sent me a patch to fix it within a few days.

One thing I am absolutely certain of: learning Coccinelle and writing semantic patches was way more fun than making the changes by hand or using regular expressions. I also had much greater confidence that my changes were correct; it is remarkably pleasant to make several hundred lines of changes and have the result compile cleanly and pass the regression tests the first time.

Related work

If you really want to, you can do everything Coccinelle can do by writing your own scripts — after all, code is code. But you have to deal with all the little corner cases — e.g., to C, white space is all the same, generally speaking, but regular expressions care intensely about the difference between a space, a newline, and a tab. Use the right tool for the job — if you're just replacing a variable name and your first script works, great. If you're changing a calling convention or moving the allocation and freeing of an object to another context, give a tool like Coccinelle a try.

In terms of power and flexibility, Coccinelle is similar to the Stanford compiler checker [PDF] (commercialized by Coverity). While the compiler checker is far more mature and has better flow analysis and parsing, Coccinelle can generate code to fix the bugs it finds. Most importantly, Coccinelle is open source, so developers can find and fix bugs themselves.

Some IDEs include tools to automatically refactor code, which is one aspect of what Coccinelle does. I have never personally used one of these IDE refactoring tools and can't compare it with Coccinelle, but my friends who have report that their stability leaves something to be desired. Xrefactory is a refactoring tool available on *NIX platforms which is fully integrated with Emacs and XEmacs. It is not open source and requires the purchase of a license, although one version is available for use free of charge.

Conclusion

Coccinelle is an open source tool that can analyze and transform C code according to specified rules, or semantic patches. Semantic patches are much more powerful than patches or regular expressions. The tool is beta quality right now but usable for practical tasks and the developers are very responsive. It's worth learning for any developer making a non-trivial interface change.

Comments (64 posted)