LWN.net Weekly Edition for June 12, 2008

Google announces Gadgets for Linux

Google recently announced the release of their Gadgets for the Linux desktop, and, unlike some of their other desktop offerings, they released it under a free software license. While it is not earth-shattering technology, Gadgets does provide some interesting features and amusing diversions. It also generates some hope that Google is getting better at understanding what free software users are looking for, so perhaps things like the Google Desktop for Linux will be better integrated and more useful in the future.

Gadgets are a cross-platform way to create simple applications that can run on web pages and desktops. The gadget API provides a means to retrieve content from other sites and display it along with a user interface. Many kinds of applications can be created, from clocks and calendars to RSS-feedreaders and "picture of the day" viewers.

There are numerous gadgets available, a semi-random collection on a KDE desktop can be seen at left. Google has created a handful of gadgets, but the vast majority are available from others in various categories including News, Sports, Finance, Fun and Games, Technology, and Communication. The gadget browser shown below, at right, allows easy access to an amazing number of choices, many of which are variations on a theme.

To get started with gadgets, it is first necessary to build the tool. Google does not yet provide .rpm or .deb files for various distributions. The "how to build" page was useful, but there was some difficulty in trying to translate the dependencies into Fedora 9 package names. A page in a language I don't know needed no translation, however. Linux commands, it seems, are multi-lingual.

Building from the Apache-licensed source tarball was straightforward after that. Gadgets for Linux comes in both GTK+ and Qt flavors which allows for integration with the two dominant Linux desktop environments. The screenshots accompanying this article are from the Qt version, but a bit of a look at the GTK+ version seemed roughly the same—though the Qt version lacks the sidebar dock.

This is a beta release, perhaps more of a beta than many Google releases, so there are still a fair number of glitches. Perhaps 20% of the gadgets tried had one problem or another, with some seeming not to function at all. Having no experience with gadgets on other platforms, it was not clear whether these were caused by bugs in the gadgets themselves or the desktop gadget program.

The main benefit of the gadget API seems to be the cross-platform capabilities. Gadgets can run—largely unchanged—on Linux, Mac OS X, or Windows, but can also run in browsers on web pages at social networking sites or on other pages. If the API can deliver that wide of a range of platform choices, it could open up a much wider audience for folks that want to develop their gadgets on Linux.

Still missing is one of the tools recommended for developing gadgets, Gadget Designer, which is only available for Windows. The documentation for creating a gadget make it look like a tedious exercise in XML manipulation and Javascript programming, but there may be tools available or in development to make some of that easier.

Overall, gadgets look like an interesting project. There is really nothing new about the kinds of applications that can be built using the API, but there are few choices to build those kinds of programs in a truly cross-platform way. Google's choice to support Linux—and support it well—accompanied by the code under a free software license is, perhaps, the best news of all.

Comments (7 posted)

An interview with Jim Ready

Jim Ready has a long history in the embedded systems market. Most recently, he became the founder of MontaVista, now one of the most successful embedded Linux companies. A recent LWN article took issue with some of Jim's comments; it only seemed fair to give him the opportunity to present his side of the story. Thus, this interview. We asked several questions about MontaVista and its approach to Linux marketing, and Jim took quite a bit of time to answer them in detail. So, without further ado...

You have been working in the embedded Linux market for some years. How has that market changed over that time? What do you think are the prospects for embedded Linux now?

Where do you think MontaVista's sweet spot is in that market?

Embedded systems vendors have, as a group, been criticized for their lack of participation in the free software development process. Are you happy with MontaVista's level of contribution? What, in your mind, are some of the highlights of MontaVista's community participation?

Your recent article in Military Embedded Systems was seized upon by a proprietary embedded vendor as proof that Linux is too expensive and difficult for embedded applications. Assuming you disagree with his conclusions, where do you think his reasoning went wrong?

Your article suggests that an embedded systems manufacturer using Linux would start by assembling the kernel and development toolchain by hand. Why do you think they would do that? Even in the absence of vendors like MontaVista, there are numerous options which do not require assembling systems at such a low level; why would a vendor not use one of them?

There were some interesting numbers in that article. Where did the 5000 messages/day for kernel.org come from - which lists?

You say: "a recent security patch that took all of 13 lines of code to implement against an embedded Linux system would have taken more than 800k lines of source patches to implement if the previous trail of patches had been ignored." How was that number arrived at? Which security patch were you describing? How could it possibly require 800,000 lines of patches "to implement" this security fix?

At least some of MontaVista's marketing would appear to focus on making Linux look scary. Are you not concerned that this approach might have the effect of making Linux in general look less attractive and, thus, playing into the hands of proprietary systems vendors?

Is there anything else you would like to pass on to LWN's readers?

We would like to thank Jim for taking the time to answer our questions.

Comments (2 posted)

Implications of pure and constant functions

Introduction

Attributes and why you should use them

Free Software development is often a fun task for developers, and it is its low barrier to entry (on average) that makes it possible to have so much available software for so many different tasks. This low barrier to entry, though, is also probably the cause of the widely varying quality of the code of these projects.

Most of the time, the quality issues one can find are not related to developers' lack of skill, but rather to lack of knowledge of how the tools work, in particular, the compiler. For non-interpreted languages, the compiler is probably the most complex tool developers have to deal with. Because a lot of Free Software is written in C, GCC is often the compiler of choice.

Modern compilers are also supposed to do a great job at optimizing the code by taking code, often written with maintainability and readability in mind, and translating it into assembler code with a focus on performance. Code analysis for optimization (which is also used for warnings about the code) has the task of taking a semantic look at the code, rather than syntactic, and identifying various fragments of algorithms that can be replaced with faster code (or with code that uses a smaller memory footprint, if the user desires to do so).

This task is a pretty complex one and relies on the compiler knowing about the function called by the code. For instance, the compiler might know when to replace a call to a (local, static) function with its body (inlining) by looking at its size, the number of times it is called, and its content (loops, other calls, variables it uses). This is because the compiler can give a semantic value to the code for a function, and can thus assess the costs and benefits of a particular transformation at the time of its use.

I specified above that the compiler knows when to inline a function by looking at its content. Almost all optimizations related to function calls work this way: the compiler, knowing the body of a function, can decide when it's the case to replace a call with its body; when it is possible to completely avoid calling the function at all; and when it is possible to call it just once and thereby avoid multiple calls. This means, though, that these optimization can be applied only to functions that are defined in the same unit wherein they are used. These functions are usually limited to static functions (functions that are not defined as static can often be overridden both at link time and runtime, so the compiler cannot safely assume that what it finds in the unit is what the code will be calling).

As this is far from optimal, modern compilers like GCC provide a way for the developer to provide information about the semantics of a function, through the use of attributes attached to declarations of functions and other symbols. These attributes provide information to the compiler on what the function does, even though its body is not available. Consequently, the compiler can optimize at least some of its calls.

This article will focus on two particular attributes that GCC makes available to C developers: pure and const, which can declare a function as either pure or constant. The next section will provide a definition of these two kinds of functions, and after that I'll get into an analysis of some common optimizations that can be performed on the calls of these functions.

As with all the other function attributes supported by GCC and ICC, the pure and const attributes should be attached to the declarative prototype of the function, so that the compiler know about them when it finds a call to the function even without its definition. For static functions, the attribute can be attached to the definition by putting it between the return type and the name of the function:

    int extern_pure_function([...])
      __attribute__((pure));
    int extern_const_function([...])
      __attribute__((const));

    int __attribute__((pure)) static_pure_function([...]) {
      [...]
    }

    int __attribute__((const)) static_const_function([...]) {
      [...]
    }
    

Pure and Constant Functions

For what concerns the scope of this article, functions can be divided into three categories, from the smallest to the biggest: constant functions, pure functions and the remaining functions can be called normal functions.

As you can guess, constant functions are also pure functions, but pure functions cannot be not all pure functions are constant functions. In many ways, constant functions are a special case of pure functions. It is, therefore, best to first define pure functions and how they differ from all the rest of the functions.

A pure function is a function with basically no side effect. This means that pure functions return a value that is calculated based on given parameters and global memory, but cannot affect the value of any other global variable. Pure functions cannot reasonably lack a return type (i.e. have a void return type).

GCC documentation provides strlen() as an example of a pure function. Indeed, this function takes a pointer as a parameter, and accesses it to find its length. This function reads global memory (the memory pointed to by parameters is not considered a parameter), but does not change it, and the value returned derives from the global memory accessed.

A counter-example of a non-pure function is the strcpy() function. This function takes two pointers as parameters. It accesses the latter to read the source string, and the former to write to the destination string. As I said, the memory areas pointed to by the parameters are not parameters on their own, but are considered global memory and, in that function, global memory is not only accessed for reading, but also for writing. The return value derives directly from the parameters (it is the same as the first parameter), but global memory is affected by the side effect of strcpy(), making it not pure.

Because the global memory state remains untouched, two calls to the same pure function with the same parameters will have to return the same value. As we'll see, it is a very important assumption that the compiler is allowed to make.

A special case of pure functions is constant functions. A pure function that does not access global memory, but only its parameters, is called a constant function. This is because the function, being unrelated to the state of global memory, will always return the same value when given the same parameters. The return value is thus derived directly and exclusively from the values of the parameters given.

The way a constant function "consumes" pointers is very different from the way other functions do: it can handle them as both parameter and return value only if they are never dereferenced, for accessing the memory they are referencing would be a global memory access, which breaks the requirements of constant functions.

Of course these requirements have to apply not only to the operations in the given function, but also recursively to all the functions it calls. One function can at best be of the same kind of the least restrictive kind of function it calls. So when it calls a normal function it can't be but a normal function itself, if it only calls pure functions it can be either pure or normal, but not constant, and if it only calls constant functions it can be constant.

As with inlining, the compiler will be able to decide if a function is pure or constant, in case no attribute is attached to it, only if the function is static (with the exception of special cases for freestanding code and other advanced options). When a function is not static, even if it's local, the compiler will assume that the function can be overridden at link- or run-time so it will not make any assumption based on the body for the definition it may find.

Optimizing Function Calls

Why should developers bother with marking functions pure or constant, though? As I said, these two attributes help the compiler to know some semantic meaning of a function call, so that it can apply higher optimization than to normal functions.

There are two main optimizations that can be applied to these kinds of functions: CSE (Common Sub-expression Elimination) and DCE (Dead Code Elimination). We'll soon see in detail, with the help of the compiler itself, what these two consist of. Their names, however, are already rather explicit: CSE is used to avoid duplicating the same code inside a function, usually factoring out the code before branching or storing the results of common operations in temporary variables (registers or stack), while DCE will remove code that would never be executed or that would be executed but never used.

These are both optimization that can be implemented in the source code, to an extent, reducing the usefulness of declaring functions pure or constant. On the other hand, as I'll demonstrate, doing so often reduces the readability of the code by obscuring the actual algorithm in favor of making it faster. This does not apply to all cases though, sometimes, doing the optimization "manually", directly in the source code, makes it more readable, and makes the code resemble the output of the compiler more.

About Assemblers and Examples

When talking about optimization, it's quite difficult to visualize the task of the compiler, and the way the code morphs from what you read in the C source code into what the CPU is really going to execute. For this reason, the best way to write about them is to use examples, showing what the compilers generates starting from the source code.

Given the way in which GCC works, this is actually quite easy. You just need to enable optimization and append the -S switch to the gcc command line. This switch stops the compiler after the transformation of C source code into assembly, before the result is passed to the assembler program to produce the object file.

Although I suspect a good fraction of the people reading this article would be comfortable reading IA-32 or x86-64 assembly code, I decided to use the Blackfin ^[1] assembly language, which should be readable for people who have never studied a particular assembly language.

The Blackfin assembler is more symbolic than IA-32: instead of having operations named movl and addq, the operations are identified by their algebraic operators (=, +), while the registers are merely called R1, R2 and so on.

Calling conventions are also quite easy to understand: for all the cases we'll look through in the article (at most four parameters, integers or pointers), the parameters are passed through the registers, starting in order from R0. The return value of the function call is also stored in the R0 register.

To clarify the examples which will appear later on, let's see how the following C source code is translated by GCC into Blackfin code:

    int somefunction(int a, int b, int c);
    void somestringfunction(char *pA, char *pB);

    int globalvar;

    void test() {
      somestringfunction("foo", "bar");
      globalvar = somefunction(11, 22, 33);
    }
      

becomes:

	    .section	.rodata
	    .align 4
    L$LC$0:
	    .string	"foo"
	    .align 4
    L$LC$1:
	    .string	"bar"
    .text;
	    .align 4
    .global _test;
    .type _test, STT_FUNC;
    _test:
	    LINK 12;
	    R0.H = L$LC$0; 
	    R0.L = L$LC$0;
	    R1.H = L$LC$1;
	    R1.L = L$LC$1;
	    call _somestringfunction; 
	    R0 = 11 (X); 
	    R1 = 22 (X);
	    R2 = 33 (X);
	    call _somefunction;
	    P2.H = _globalvar; 
	    P2.L = _globalvar;
	    [P2] = R0; 
	    UNLINK;
	    rts; 
	    .size	_test, .-_test
      

As the Blackfin does not have 32 bit immediate load, you have to load high and low addresses separately (in whichever order); the assembler will take care of properly loading the high 16 bits of the label to the upper part of the register, and the low 16 bits to the lower part.

Once the parameters are loaded, the function is called almost identically to any other call operation on other architectures; note the prefixed underscore on symbols' names.

Integers, both constant or parameters and variables, are also loaded for calls in the registers. Blackfin doesn't have 32 bit immediate loading, but if the constant to load fits into 16 bits, it can be loaded through sign extension by appending the (X) suffix.

When accessing a global memory location, the P2 pointer is set to the address of the memory location...

... and then dereferenced to assign that memory area. Being a RISC architecture, Blackfin does not have direct memory operations.

The return value for a function is loaded into the R0 register, and can be accessed from there.

The rts command is the return from subroutine, and usually indicates the end of the function, but like the return statement in C, it might appear in any place of the routine.

In the following examples, the preambles with declarations and data will be omitted whenever these are not useful to the discussion.

Concerning optimization levels, the code will almost always be compiled with at least the first optimization level enabled (-O1). This both because it makes the code cleaner to read (using register-register copy for parameters passing, instead of saving to the stack and then restoring from that) and because we need optimization enabled to see how they are applied.

Also, most of the times I'll refer to the fastest alternative. Most of what I say, though, applies also to the smaller alternative when using the -Os optimization level. In any case, the compiler always weighs the cost-to-benefit ratio between the optimized and the unoptimized version, or between different optimized versions. If you want to know the exact route the compiler takes for your code, you can always use the -S switch to find out.

DCE and Unused Variables

One area where DCE is useful is to avoid operations that result in unused data. It's not that uncommon that a variable is defined by an operation, complex or not, and is then never used by the code, either because it is intended for future expansion or because it's a remnant of older code that has been removed or replaced. While the best thing would be to get rid of the definition entirely, users expect the compiler to produce a good result with sloppy code too, and that operation should not be emitted.

The DCE pass can remove all the code that has no side effect, when its result is not used. This includes all mathematical operations and functions known to be pure or constant (as neither are allowed to change the global state of the variables). If a function call is not known to be at least pure, it may change the global state, and its call will not be eliminated, as shown in the following code:

    int someimpurefunction(int a, int b);
    int somepurefunction(int a, int b)
      __attribute__((pure));

    int testfunction(int a, int b, int c) {
      int res1 = someimpurefunction(c, b);
      int res2 = somepurefunction(b, c);
      int res3 = a + b - c;

      return a;
    }
      

Which, once compiled with -O1, ^[2] produces the following Blackfin assembler:

    _testfunction:
	    [--sp] = ( r7:7 );

	    LINK 12;
	    R7 = R0;
	    R0 = R2;
	    call _someimpurefunction;
	    R0 = R7;
	    UNLINK;
	    ( r7:7 ) = [sp++];

	    rts;
      

As you can see, the call to the pure function has been eliminated (the res2 variable was not being used), together with the algebraic operation but, the impure function, albeit having its return value discarded, is still called. This is due to the fact that the compiler emits the call, not knowing whether the latter function has side effects on the global memory state or not.

This is equivalent to the following code (which produces the same assembler code):

    int someimpurefunction(int a, int b);

    int testfunction(int a, int b, int c) {
      someimpurefunction(c, b);

      return a;
    }
      

The Dead Code Elimination optimization can be very helpful to reduce the overhead caused by code written to conform to C89 standard, where you couldn't mix variables (and constant) declarations with executable code.

In those sources, you had to declare variables at the top of the function, and then start to check for prerequisites. If you wanted to make it explicit that some variable had to keep its value, by making it constant, you would often have to fill them before the prerequisites could be checked.

Without discussing legacy code, it is also useful when writing debug code, so that it doesn't look out of place from the use of lots of #ifdef directives. Take for instance the following code:

    #ifdef NDEBUG
    # define assert_se(x) (x)
    #else
    void assert_se(int boolean);
    #endif

    char *getsomestring(int i) __attribute__((pure));
    int dosomethinginternal(void *ctx, int code, int val);

    int dosomething(void *ctx, int code, int val) {
      char *string = getsomestring(code);

      // returning string might be a sub-string of "something"
      // like "some" or "so"
      assert_se(strncmp(string, "something", strlen(string)) == 0);

      return dosomethinginternal(ctx, code, val);
    }
      

The assert_se macro has different behavior from the standard assert, as it has side effects, which basically means that the code passed to the assertion is called even though the compiler is told to disable debugging. This is a somewhat common trick, although its effects on readability are debatable.

With getsomestring() pure, when compiling without debugging, the DCE will remove the calls to all three functions: getsomestring(), strncmp() and strlen() (the latter two are usually declared as pure by both the C library and by GCC's built-in replacements). This because none of these functions have a side effect, resulting in a very short function:

    _dosomething:
	    LINK 0;
	    UNLINK;
	    jump.l _dosomethinginternal;
      

If our getsomestring() function weren't pure, even though its return value is not going to be used, the compiler would have to emit the call, resulting in rather more complex (albeit still simple, compared with most real-world functions) assembler code:

    _dosomething:
	    [--sp] = ( r7:5 );

	    LINK 12;
	    R7 = R0;
	    R0 = R1;
	    R6 = R1;
	    R5 = R2;
	    call _getsomestring;
	    UNLINK;
	    R0 = R7;
	    R1 = R6;
	    R2 = R5;
	    ( r7:5 ) = [sp++];

	    jump.l _dosomethinginternal;
      

Common Sub-expression Elimination

The Common Sub-expression Elimination optimization is one of the most important optimizations performed by the compiler, because it's the one that, for instance, replaces multiple indexed accesses to an array so that the actual memory address is calculated just once.

What this optimization does is to find common operations executed on the same operands (even when they are not known at compile-time), decide which ones are more expensive than saving the result in a temporary (register or stack), and then swapping the code around to take the cheapest course.

While its uses are quite varied, one of the easiest ways to see the work of the CSE is to look at the code generated when using the ternary if operator. Let's take the following code:

    int someimpurefunction(int a);
    int somepurefunction(int a)
      __attribute__((pure));

    int testfunction(int a, int b, int c, int d) {

      int res1 = someimpurefunction(a) ? someimpurefunction(a) : b;
      int res2 = somepurefunction(a) ? somepurefunction(a) : c;
      int res3 = a+b ? a+b : d;

      return res1+res2+res3;
    }
      

The compiler will optimize the code as:

    _testfunction:
	    [--sp] = ( r7:4 );

	    LINK 12;
	    R7 = R0;
	    R5 = R1;
	    R4 = R2;
	    call _someimpurefunction;
	    cc =R0==0;
	    if !cc jump L$L$2;
	    R6 = R5;
	    jump.s L$L$4;
    L$L$2:
	    R0 = R7;
	    call _someimpurefunction;
	    R6 = R0;
    L$L$4:
	    R0 = R7;
	    call _somepurefunction;
	    R1 = R0;
	    cc =R0==0;
	    if cc R1 =R4; /* movsicc-1b */
	    R0 = R5 + R7;
	    cc =R0==0;
	    R2 = [FP+36];
	    if cc R0 =R2; /* movsicc-1b */
	    R1 = R1 + R6;
	    R0 = R1 + R0;
	    UNLINK;
	    ( r7:4 ) = [sp++];

	    rts;
      

As you can see, the pure function is called just once, because the two references inside the ternary operator are equivalent, while the other one is called twice. This is because there was no change to global memory known to the compiler between the two calls of the pure function (the function itself couldn't change it – note that the compiler will never take multi-threading into account, even when asking for it explicitly through the -pthread flag), while the non-pure function is allowed to change global memory or use I/O operations.

The equivalent code in C would be something along the following lines (it differs a bit because the compiler will use different registers):

    int someimpurefunction(int a);
    int somepurefunction(int a)
      __attribute__((pure));

    int testfunction(int a, int b, int c, int d) {

      int res1 = someimpurefunction(a) ? someimpurefunction(a) : b;
      const int tmp1 = somepurefunction(a);
      int res2 = tmp1 ? tmp1 : c;
      const int tmp2 = a+b;
      int res3 = tmp2 ? tmp2 : d;

      return res1+res2+res3;
    }
      

The Common Sub-expression Elimination optimization is very useful when writing long and complex mathematical operations. The compiler can find common calculations even though they don't look common to the naked eye, and act on those.

Although sometimes you can get away with using multiple constants or variables to carry out temporary operations so that they can be re-used in the following calculations, leaving the formulae entirely explicit is usually more readable, as long as the formulae are not intended to change.

Like with other algorithms, there are some advantages to reducing the source code used to calculate the same thing; for instance you can easily make a change directly to the definition of a constant and get the change propagated to all the uses of that constant. On the other hand, this can be quite a problem if the meaning of two calculations is very different (and thus can vary in different ways with the evolution of the code), and just happen to be calculated in the same way at a given time.

Another rather useful place where the compiler can further optimize code with CSE, where it wouldn't be so nice or simple to do manually in the source code, is where you deal with static functions that are inlined by the compiler.

Let's examine the following code for instance:

    extern int a;
    extern int b;

    static inline int somefunc1(int p) {
      return (p * 16) + (3 << a);
    }

    static inline int somefunc2(int p) {
      return (p * 16) + (4 << b);
    }

    extern int res1;
    extern int res2;
    extern int res3;
    extern int res4;

    void testfunc(int p1, int p2)
    {
      res1 = somefunc1(p1);
      res2 = somefunc2(p1);
      res3 = somefunc1(p2);
      res4 = somefunc2(p2);
    }
      

In this code, you can find four basic expressions: (p1 * 16), (p2 * 16), (3 << a) and (4 << b). Each of these four expressions is used twice in the somefunc() function. Thanks to the CSE, though, the code will calculate each of them once, even though they cross the function boundary, producing the following code:

    _testfunc:
	    [--sp] = ( r7:7 );

	    LINK 0;

	    R0 <<= 4;
	    R1 <<= 4;

	    P2.H = _a;
	    P2.L = _a;
	    R2 = [P2];
	    R7 = 3 (X);
	    R7 <<= R2;

	    P2.H = _b;
	    P2.L = _b;
	    R2 = [P2];
	    R3 = 4 (X);
	    R3 <<= R2;

	    R2 = R0 + R7;
	    P2.H = _res1;
	    P2.L = _res1;
	    [P2] = R2;
	    P2.H = _res2;
	    P2.L = _res2;
	    R0 = R0 + R3;
	    [P2] = R0;

	    R7 = R1 + R7;
	    P2.H = _res3;
	    P2.L = _res3;
	    [P2] = R7;
	    R1 = R1 + R3;
	    P2.H = _res4;
	    P2.L = _res4;
	    [P2] = R1;
	    UNLINK;
	    ( r7:7 ) = [sp++];

	    rts;
      

As you can easily see (the assembly was modified a bit to improve its readability, the compiler re-ordered loads of registers to avoid pipeline stalls, making it harder to see the point), the four expressions are calculated first, and stored respectively in the registers R0, R1, R7 and R3.

These kinds of sub-expressions are usually harder to see in the code and also harder to implement. Sometimes they get factored out on their own parameter, but that can be more expensive during execution, depending on the calling conventions of the architecture.

Cheats

As I wrote above, there are some requirements that apply to functions that are declared pure and constant, related to not changing or accessing global memory; not executing I/O operations; and, of course, not calling further impure functions. The reason for this is that the compiler will accept what the user declares the function to be, whatever its body is (as it's usually unknown by the compiler at the call stage).

Sometimes, though, it's possible to fool the compiler so that it treats impure functions as pure or even constant functions. Although this is a risky endeavor, as it might truly cause bad code generation by the compiler, it can sometimes be used to force optimization for particular functions.

An example of this can be a lookup function that scans through a global table to return a value. While it is accessing global memory, you might want the compiler to promote it to a constant function, rather than simply to a pure one.

Let's take for instance the following code:

    const struct {
      const char *str;
      int val;
    } strings[] = {
      { "foo", 31 },
      { "bar", 34 },
      { "baz", -24 }
    };

    const char *lookup(int val) {
      int i;
      for(i = 0; i < sizeof(strings)/sizeof(*strings); i++)
	if ( strings[i].val == val )
	  return strings[i].str;

      return NULL;
    }

    void testfunction(int val, const char **str, unsigned long *len) {
      if ( lookup(val) ) {
	*str = lookup(val);
	*len = strlen(lookup(val));
      }
    }

      

If the lookup() function is only considered a pure function, as it is, adhering to the rules we talked about at the start of the article, it will be called three times in testfunction(), like this:

    _testfunction:
	    [--sp] = ( r7:7, p5:4 );

	    LINK 12;
	    R7 = R0;
	    P5 = R1;
	    P4 = R2;
	    call _lookup;
	    cc =R0==0;
	    if cc jump L$L$17;
	    R0 = R7;
	    call _lookup;
	    [P5] = R0;
	    R0 = R7;
	    call _lookup;
	    call _strlen;
	    [P4] = R0;
    L$L$17:
	    UNLINK;
	    ( r7:7, p5:4 ) = [sp++];

	    rts;
      

Instead, we can trick the compiler by declaring the lookup() function as constant (the data it is reading is constant, after all, so at a given parameter it will always return the same result). If we do that, the three calls will have to return the same value, and the compiler will be able to optimize them as a single call:

    _testfunction:
	    [--sp] = ( p5:4 );

	    LINK 12;
	    P5 = R1;
	    P4 = R2;
	    call _lookup;
	    cc =R0==0;
	    if cc jump L$L$17;
	    [P5] = R0;
	    call _strlen;
	    [P4] = R0;
    L$L$17:
	    UNLINK;
	    ( p5:4 ) = [sp++];

	    rts;
      

In addition to lookup functions on constant tables, this trick is useful with functions which read data from files or other volatile data, and cache it in a memory variable.

Take for instance the following function that reads an environment variable:

    char *get_testval() {
      static char *cachedval = NULL;

      if ( cachedval == NULL ) {
	cachedval = getenv("TESTVAL");
	if ( cachedval == NULL )
	  cachedval = "";
	else
	  cachedval = strdup(cachedval);
      }

      return cachedval;
    }
      

This is not truly a constant function, as its return value depends on the environment. Even so, assuming that the environment of the process is left untouched, its return value will never change between calls. Even though it will affect the global state of the program (as the cachedval static variable will be filled in the first time the function is called), it can be assumed to always return the same value.

Tricking the compiler into thinking that a function is constant even though it has to load data through I/O operations, as I said, is risky, as the compiler will think there is no I/O operation going on; on the other hand, this trick might make a difference sometimes, as it allows the expression of functions in more semantic ways, leaving it up to the compiler to optimize the code with temporaries, where needed.

One example can be the following code:

    char *get_testval() {
      static char *cachedval = NULL;

      if ( cachedval == NULL ) {
	cachedval = getenv("TESTVAL");
	if ( cachedval == NULL )
	  cachedval = "";
	else
	  cachedval = strdup(cachedval);
      }

      return cachedval;
    }

    extern int a;
    extern int b;
    extern int c;
    extern int d;

    static int testfunc1() {
      if ( strcmp(get_testval(), "FOO") == 0 )
	return a;
      else
	return b;
    }

    static int testfunc2() {
      if ( strcmp(get_testval(), "BAR") == 0 )
	return c;
      else
	return d;
    }

    int testfunction() {
      return testfunc1() + testfunc2();
    }
      

Note: To make sure that the compiler won't reduce the three function calls to their return values right away, the static sub-functions return values taken from global variables; the meanings of those variables are not important.

Considering the above source code, if get_testval() is impure, as the compiler will automatically find it to be, it will be compiled into:

    _testfunction:
	    [--sp] = ( r7:7 );

	    LINK 12;
	    call _get_testval;
	    R1.H = L$LC$2;
	    R1.L = L$LC$2;
	    call _strcmp;
	    cc =R0==0;
	    if !cc jump L$L$11 (bp);
	    P2.H = _a;
	    P2.L = _a;
	    R7 = [P2];
    L$L$13:
	    call _get_testval;
	    R1.H = L$LC$3;
	    R1.L = L$LC$3;
	    call _strcmp;
	    cc =R0==0;
	    if !cc jump L$L$14 (bp);
	    P2.H = _c;
	    P2.L = _c;
	    R0 = [P2];
	    UNLINK;
	    R0 = R0 + R7;
	    ( r7:7 ) = [sp++];

	    rts;
    L$L$11:
	    P2.H = _b;
	    P2.L = _b;
	    R7 = [P2];
	    jump.s L$L$13;
    L$L$14:
	    P2.H = _d;
	    P2.L = _d;
	    R0 = [P2];
	    UNLINK;
	    R0 = R0 + R7;
	    ( r7:7 ) = [sp++];

	    rts;
      

As you can see, the get_testval() is called twice, even though its result will be identical. If we declare it constant, instead, the code of our test function will be the following:

    _testfunction:
	    [--sp] = ( r7:6 );

	    LINK 12;
	    call _get_testval;
	    R1.H = L$LC$2;
	    R1.L = L$LC$2;
	    R7 = R0;
	    call _strcmp;
	    cc =R0==0;
	    if !cc jump L$L$11 (bp);
	    P2.H = _a;
	    P2.L = _a;
	    R6 = [P2];
    L$L$13:
	    R1.H = L$LC$3;
	    R0 = R7;
	    R1.L = L$LC$3;
	    call _strcmp;
	    cc =R0==0;
	    if !cc jump L$L$14 (bp);
	    P2.H = _c;
	    P2.L = _c;
	    R0 = [P2];
	    UNLINK;
	    R0 = R0 + R6;
	    ( r7:6 ) = [sp++];

	    rts;
    L$L$11:
	    P2.H = _b;
	    P2.L = _b;
	    R6 = [P2];
	    jump.s L$L$13;
    L$L$14:
	    P2.H = _d;
	    P2.L = _d;
	    R0 = [P2];
	    UNLINK;
	    R0 = R0 + R6;
	    ( r7:6 ) = [sp++];

	    rts;
      

The CSE pass combines the two calls to get_testval with one. Again, this is one of the optimizations that are harder to achieve by manually changing the source code since the compiler can have a larger view of the use of its value. A common way to handle this is by using global variables, but that might require one more load from the memory, while CSE can take care of keeping the values in registers or on the stack.

Conclusions

After what you have read about pure and constant functions, you might have some concerns about the average use of them. Indeed, in a lot of cases, these two attributes allow the compiler to do something you can easily achieve by writing better code.

There are two objectives you have to keep in mind that are related to the use of these (and other) attributes. The first is code readability because sometimes the manually optimized functions are harder to read than what the compiler can produce. The second is allowing the compiler to optimize legacy or external code.

While you might not be too concerned with letting legacy code or code written by someone else get away with slower execution, a pragmatic view of the current Free Software world should take into consideration the fact that there are probably thousands lines of code of legacy code around. Some of that code, written with pre-C99 declarations, might be even using libraries that are being developed with their older interface, which could be improved by providing some extra semantic information to the compiler through use of attributes.

Also, it's unfortunately true that extensive use of these attributes might be seen by neophytes as an easy solution to let sloppy code run at a decent speed. On the other hand, the same attributes could be used to identify such sloppy code through analysis of the source code.

Although GCC does not issue warnings for all of these cases, it already warns for some of them, like unused variables, or statements without effect (both triggered by the DCE). In the future more warnings might be reported if pure and constant functions get misused.

In general, like with many other GCC function attributes, their use is tightly related to how programmers perceive their task. Most pragmatic programmers would probably like these tools, while purists will probably dislike the way these attributes help sloppy code to run almost as fast as properly written code.

My hopes are that in the future better tools will make good use of these and other attributes on different levels than compilers, like static and dynamic analyzers.

---

^[1] The Blackfin architecture is a RISC architecture developed by Analog Devices, supported by both GCC and Binutils (and Linux, but I'm not interested in that here).

^[2] I have chosen -O1 rather than -O2 because in the latter case the compiler performs extra optimization passes that I do not wish to discuss within the scope of this article.

Comments (45 posted)

Page editor: Jake Edge

Security

SCADA system vulnerabilities

Core Security released a security advisory on 11 June that details a fairly pedestrian stack-based buffer overflow vulnerability. This is similar to hundreds or thousands of this kind of flaw reported over the years except for one thing: it was found in large industrial control systems for things like power and water utility companies. That there is a vulnerability is not surprising—there are certainly many more—but it does give one pause about the dangers of connecting these systems to the internet.

The bug was found in a Supervisory Control and Data Acquisition—better known as SCADA—system and could be exploited to execute arbitrary code. Given that SCADA systems run much of the world's infrastructure, an exploit of a vulnerable system could have severe repercussions. The customers of Citect, the company that makes the affected systems, include "organizations in the aerospace, food, manufacturing, oil and gas, and public utilities industries."

Makers of SCADA systems nearly uniformly tell their customers to keep those systems isolated from the internet. But as Core observes: "the reality is that many organizations do have their process control networks accessible from wireless and wired corporate data networks that are in turn exposed to public networks such as the Internet." So, the potential for a random internet bad guy to take control of these systems does exist.

None of that should be particularly surprising when you stop to think about it, but it is worrying. Many SCADA systems—along with various other control systems—were designed and developed long before the internet started reaching homes and offices everywhere. They were designed for "friendly" environments, with little or no change for the hostile environment that characterizes today's internet. Also, as we have seen, security rarely gets the attention it deserves until some kind of ugly incident occurs.

Even for systems that were designed recently, there are undoubtedly vulnerabilities, so it is a bit hard to believe that they might be internet-connected. According to the advisory, though, SCADA makers do not necessarily require that the systems be physically isolated from the network, instead customers can "utilize technologies including firewalls to keep them protected from improper external communications."

Firewalls—along with other security techniques—do provide a measure of protection, but with the stakes so high, it would seem that more caution is required. It is probably convenient for SCADA users to be able to connect to other machines on the LAN, as well as to the internet, but with that convenience comes quite a risk. Even systems that are just locally connected could fall prey to a disgruntled employee exploiting a vulnerability to gain access to systems they normally wouldn't have.

One can envision all manner of havoc that could be wreaked by a malicious person (or government) who can take over the systems that control nuclear power plants, enormous gas pipelines, or some chunk of the power grid. Unfortunately, it will probably take an incident like that to force these industries into paying as much attention to their computer security as they do to their physical security.

Comments (5 posted)

New vulnerabilities

kernel: arbitrary code execution

Package(s):	kernel	CVE #(s):	CVE-2008-1673
Created:	June 9, 2008	Updated:	August 20, 2008
Description:	From the Debian advisory: Wei Wang from McAfee reported a potential heap overflow in the ASN.1 decode code that is used by the SNMP NAT and CIFS subsystem. Exploitation of this issue may lead to arbitrary code execution. This issue is not believed to be exploitable with the pre-built kernel images provided by Debian, but it might be an issue for custom images built from the Debian-provided source package.
Alerts:	Mandriva MDVSA-2008:174 2008-08-19 SuSE SUSE-SA:2008:038 2008-07-29 Ubuntu USN-625-1 2008-07-15 SuSE SUSE-SA:2008:035 2008-07-21 Fedora FEDORA-2008-5454 2008-06-20 Mandriva MDVSA-2008:113 2008-06-13 Fedora FEDORA-2008-5308 2008-06-12 rPath rPSA-2008-0189-1 2008-06-11 Debian DSA-1592-2 2008-06-09 Debian DSA-1592-1 2008-06-09

Comments (none posted)

kernel: arbitrary code execution

Package(s):	kernel	CVE #(s):	CVE-2008-2358
Created:	June 9, 2008	Updated:	August 13, 2008
Description:	From the Debian advisory: Brandon Edwards of McAfee Avert labs discovered an issue in the DCCP subsystem. Due to missing feature length checks it is possible to cause an overflow they may result in remote arbitrary code execution.
Alerts:	Mandriva MDVSA-2008:167 2008-08-12 Ubuntu USN-625-1 2008-07-15 Fedora FEDORA-2008-5893 2008-07-02 CentOS CESA-2008:0519 2008-06-26 Red Hat RHSA-2008:0519-01 2008-06-25 SuSE SUSE-SA:2008:030 2008-06-20 Mandriva MDVSA-2008:112 2007-06-12 Debian DSA-1592-2 2008-06-09 Debian DSA-1592-1 2008-06-09

Comments (none posted)

net-snmp: buffer overflow

Package(s):	net-snmp	CVE #(s):	CVE-2008-2292
Created:	June 11, 2008	Updated:	August 6, 2008
Description:	From the CVE entry: Buffer overflow in the __snprint_value function in snmp_get in Net-SNMP 5.1.4, 5.2.4, and 5.4.1, as used in SNMP.xs for Perl, allows remote attackers to cause a denial of service (crash) and possibly execute arbitrary code via a large OCTETSTRING in an attribute value pair (AVP).
Alerts:	SuSE SUSE-SA:2008:039 2008-08-01 Gentoo 200808-02 2008-08-06 Slackware SSA:2008-210-07 2008-07-29 Mandriva MDVSA-2008:118 2007-06-19 Fedora FEDORA-2008-5224 2008-06-11 Fedora FEDORA-2008-5218 2008-06-11

Comments (none posted)

openoffice.org: integer overflow

Package(s):	openoffice.org	CVE #(s):	CVE-2008-2152
Created:	June 11, 2008	Updated:	July 16, 2008
Description:	OpenOffice.org has reported an integer overflow vulnerability in rtl_allocateMemory().
Alerts:	Mandriva MDVSA-2008:138-1 2008-07-11 Gentoo 200807-05 2008-07-09 Mandriva MDVSA-2008:137 2008-07-08 CentOS CESA-2008:0537 2008-06-27 CentOS CESA-2008:0538 2008-06-14 Red Hat RHSA-2008:0538-01 2008-06-12 Red Hat RHSA-2008:0537-01 2008-06-12 Fedora FEDORA-2008-5247 2008-06-11 Fedora FEDORA-2008-5239 2008-06-11 Fedora FEDORA-2008-5143 2008-06-11

Comments (none posted)

snort: detection rules bypass

Package(s):	snort	CVE #(s):	CVE-2008-1804
Created:	June 6, 2008	Updated:	June 11, 2008
Description:	From the CVE entry: preprocessors/spp_frag3.c in Sourcefire Snort before 2.8.1 does not properly identify packet fragments that have dissimilar TTL values, which allows remote attackers to bypass detection rules by using a different TTL for each fragment.
Alerts:	Fedora FEDORA-2008-5045 2008-06-06 Fedora FEDORA-2008-5001 2008-06-06 Fedora FEDORA-2008-4986 2008-06-06

Comments (none posted)

tomcat: insufficient input sanitizing

Package(s):	tomcat5.5	CVE #(s):	CVE-2008-1947
Created:	June 10, 2008	Updated:	August 29, 2008
Description:	From the Debian advisory: It was discovered that the Host Manager web application performed insufficient input sanitizing, which could lead to cross-site scripting.
Alerts:	SuSE SUSE-SR:2008:014 2008-07-04 Debian DSA-1593-1 2008-06-09 Red Hat RHSA-2008:0648-01 2008-08-27 CentOS CESA-2008:0648 2008-08-28

Comments (none posted)

ucd-snmp: possible spoof

Package(s):	ucd-snmp	CVE #(s):	CVE-2008-0960
Created:	June 10, 2008	Updated:	August 6, 2008
Description:	From the Red Hat advisory: A flaw was found in the way ucd-snmp checked an SNMPv3 packet's Keyed-Hash Message Authentication Code (HMAC). An attacker could use this flaw to spoof an authenticated SNMPv3 packet.
Alerts:	Gentoo 200808-02 2008-08-06 SuSE SUSE-SA:2008:039 2008-08-01 Slackware SSA:2008-210-07 2008-07-29 Mandriva MDVSA-2008:118 2007-06-19 Fedora FEDORA-2008-5224 2008-06-11 Fedora FEDORA-2008-5218 2008-06-11 Fedora FEDORA-2008-5215 2008-06-11 CentOS CESA-2008:0528:01 2008-06-11 CentOS CESA-2008:0529 2008-06-10 Red Hat RHSA-2008:0529-01 2008-06-10 Red Hat RHSA-2008:0528-01 2008-06-10

Comments (none posted)

Page editor: Jake Edge

Kernel development

Release status

Kernel release status

The current 2.6 development kernel is 2.6.26-rc5, released on June 4. As is usual at this point in the release cycle, it is mostly bug fixes and the like. There are a fair number of changes in the core kernel code, mostly for scheduler issues, including some reverts for some performance regressions. "Another week, another batch of mostly pretty small fixes. Hopefully the regression list is shrinking, and we've fixed at least a couple of the oopses on Arjan's list." See the long-format changelog for all the details. A 2.6.26-rc6 release is probably coming soon.

The current -mm tree is 2.6.26-rc5-mm2 which is a bug fix for 2.6.26-rc5-mm1, also released this week. The main additions are the unprivileged mounts tree and a "large number of deep changes to memory management."

The current stable 2.6 kernel is 2.6.25.6, released on June 9. It has a whole pile of bugfixes, with none that are specifically called out as security related. "It contains a number of assorted bugfixes all over the tree. Users are encouraged to update." See the LWN announcement for some discussion about potential security issues with this release. Also, note that 2.6.25.5 was released on June 7 with "one security bug fix. If you are using CIFS or SNMP NAT you could be vulnerable and are encouraged to upgrade."

For older kernels: 2.4.36.6 was released on June 6. "It only fixes a vulnerability in the netfilter ip_nat_snmp_basic module (CVE-2008-1673). If you don't use it, you don't need to upgrade."

Comments (2 posted)

Kernel development news

A new kernel tree: linux-staging

There's a new kernel tree in town. The linux-staging tree was announced by Greg Kroah-Hartman on 10 June. It is meant to hold drivers and other kernel patches that are working their way toward the mainline, but still have a ways to go. The intention is to collect them all together in one tree to make access and testing easier for interested developers.

According to Kroah-Hartman, linux-staging (or -staging as it will undoubtedly be known) "is an outgrowth of the Linux Driver Project, and the fact that there have been some complaints that there is no place for individual drivers to sit while they get cleaned up and into the proper shape for merging." By collecting the patches in one place, it will increase their visibility in the kernel community, potentially attracting more developers to assist in fixing, reviewing, and testing them.

The intent is for -staging to house self-contained patches—Kroah-Hartman mentions drivers and filesystems—that should not affect anyone who is not using them. Because of that, he is hoping that -staging can get included in the linux-next tree. As he says to Stephen Rothwell, maintainer of -next, in the announcement:

The -next tree is meant for things that are headed for inclusion in the "N+1" kernel (where 2.6.N is the release under development), so including code not meant for that release is bending the rules a bit. As of this writing, Rothwell has not responded to the request to include -staging, but it would clearly benefit those patches to have a wider audience—with only a small impact on -next. There is no set timeline for patches to move from -staging into mainline, Kroah-Hartman says:

The -staging tree is seen as a great place for Kernel Janitors and others who are interested in learning about kernel development to get their start. The announcement notes: "The code in this tree is in desperate need of cleanups and fixes that can be trivially found using 'sparse' and 'scripts/checkpatch.pl'." In the process of cleaning up the code, folks can learn how to create patches and how to get them accepted into a tree. From there, the hope is that more difficult tasks will be undertaken—with -staging or other kernel code—leading to a new crop of kernel hackers.

The current status of -staging shows 17 patches, most of which are drivers from the Linux Driver Project. Kroah-Hartman is actively encouraging more code to be submitted for -staging, as long as it meets some criteria for the tree. The tree is not meant to be a dumping ground for drivers that are being "thrown over the wall" in hopes that someone else will deal with them. It is also not meant for code that is being actively worked on by a group of developers in another tree somewhere—the reiser4 filesystem is mentioned as an example—it is for code that would otherwise languish.

The reaction on linux-kernel has so far been favorable, with questions being asked about what kinds of patches are appropriate for the tree, in particular new architectures. The -staging tree fills a niche that has not yet been covered by other trees. It also serves multiple purposes, from giving new developers a starting point to providing additional reviewing and testing opportunities for new drivers and other code. With luck, that will hasten the arrival of new features—along with new developers.

Comments (2 posted)

A summary of 2.6.26 API changes

The 2.6.26 development cycle has stabilized to the point that it's possible to look at the internal API changes which have resulted. They include:

• At long last, support for the KGDB interactive debugger has been added to the x86 architecture. There is a DocBook document in the Documentation directory which provides an overview on how to use this new facility. Some useful features (e.g. KGDB over Ethernet) are not yet supported, but this is a good start.

• Page attribute table (PAT) support is also (again, at long last) available for the x86 architecture. PATs allow for fine-grained control of memory caching behavior with more flexibility than the older MTRR feature. See Documentation/x86/pat.txt for more information.

• ioremap() on the x86 architecture will now always return an uncached mapping. Previously, it had taken a more relaxed approach, leaving the caching as the BIOS had set it up. The practical result was to almost always create uncached mappings, but with occasional exceptions. Drivers which depend on a cached mapping will now break; they will need to use ioremap_cache() instead. See this article for more information on this change and caching in general.

• The generic semaphores patch has been merged. The semaphore code also has new down_killable() and down_timeout() functions.

• The final users of struct class_device have been converted to use struct device instead. The class_device structure, along with its associated infrastructure, has been removed.

• The nopage() virtual memory area operation has been removed; all in-tree code is now using fault() instead.

• The object debugging infrastructure has been merged.

• Two new functions (inode_getsecid() and ipc_getsecid()), added to support security modules and the audit code, provide general access to security IDs associated with inodes and IPC objects. A number of superblock-related LSM callbacks now take a struct path pointer instead of struct nameidata. There is also a new set of hooks providing generic audit support in the security module framework.

• The now-unused ieee80211 software MAC layer has been removed; all of the drivers which needed it have been converted to mac80211. Also removed are the sk98lin network driver (in favor of skge) and bcm43xx (replaced by b43 and b43legacy).

• The ata_port_operations structure used by libata drivers now supports a simple sort of operation inheritance, making it easier to write drivers which are "almost like" existing code, but with small differences.

• A new function (ns_to_ktime()) converts a time value in nanoseconds to ktime_t.

• Greg Kroah-Hartman is no longer the PCI subsystem maintainer, having passed that responsibility on to Jesse Barnes.

• The seq_file code now accepts a return value of SEQ_SKIP from the show() callback; that value causes any accumulated output from that call to be discarded.

• The Video4Linux2 API now defines a set of controls for camera devices; they allow user space to work with parameters like exposure type, tilt and pan, focus, and more.

• On the x86 architecture, there is a new configuration parameter which allows gcc to make its own decisions about the inlining of functions, even when functions are declared inline. In some cases, this option can reduce the size of the kernel's text segment by over 2%.

• The legacy IDE layer has gone through a lot of internal changes which will break any remaining out-of-tree IDE drivers.

• A condition which triggers a warning from WARN_ON will now also taint the kernel.

• The get_info() interface for /proc files has been removed. There is also a new function for creating /proc files:

      struct proc_dir_entry *proc_create_data(const char *name, mode_t mode,
  					    struct proc_dir_entry *parent,
  					    const struct file_operations *proc_fops,
  					    void *data);
  

  This version adds the data pointer, ensuring that it will be set in the resulting proc_dir_entry structure before user space can try to access it.

• The klist type now has the usual-form macros for declaration and initialization: DEFINE_KLIST() and KLIST_INIT(). Two new functions (klist_add_after() and klist_add_before()) can be used to add entries to a klist in a specific position.

• kmap_atomic_to_page() is no longer exported to modules.

• There are some new generic functions for performing 64-bit integer division in the kernel:

      u64 div_u64(u64 dividend, u32 divisor);
      u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder);
      s64 div_s64(s64 dividend, s32 divisor)
      s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder);
  

  Unlike do_div(), these functions are explicit about whether signed or unsigned math is being done. The x86-specific div_long_long_rem() has been removed in favor of these new functions.

• There is a new string function:

       bool sysfs_streq(const char *s1, const char *s2);
  

  It compares the two strings while ignoring an optional trailing newline.

• The prototype for i2c probe() methods has changed:

       int (*probe)(struct i2c_client *client, 
                    const struct i2c_device_id *id);
  

  The new id argument supports i2c device name aliasing.

One change which did not happen in the end was the change to 4K kernel stacks by default on the x86 architecture. This is still a desired long-term goal, but it is hard to say when the developers might have enough confidence to make this change.

Comments (4 posted)

Andrew Morton on kernel development

Andrew Morton is well-known in the kernel community for doing a wide variety of different tasks: maintaining the -mm tree for patches that may be on their way to the mainline, reviewing lots of patches, giving presentations about working with the community, and, in general, handling lots of important and visible kernel development chores. Things are changing in the way he does things, though, so we asked him a few questions by email. He responded at length about the -mm tree and how that is changing with the advent of linux-next, kernel quality, and what folks can do to help make the kernel better.

Years ago, there was a great deal of worry about the possibility of burning out Linus. Life seems to have gotten easier for him since then; now instead, I've heard concerns about burning out Andrew. It seems that you do a lot; how do you keep the pace and how long can we expect you to stay at it?

What can we do to help?

Is it your opinion that the quality of the kernel is in decline? Most developers seem to be pretty sanguine about the overall quality problem. Assuming there's a difference of opinion here, where do you think it comes from? How can we resolve it?

There have been a number of kernel security problems disclosed recently. Is any particular effort being put into the prevention and repair of security holes? What do you think we should be doing in this area?

I have sensed that there is a bit of confusion about the difference between -mm and linux-next. How would you describe the purpose of these two trees? Which one should interested people be testing?

Do you have any specific goals for the development of the kernel over the next year or so? What would they be?

In the early-May discussions, Linus said a couple of times that he does not think code review helps much. Do you agree with that point of view?

How would you describe the real role of code review in the kernel development process?

There clearly must be quite a bit of communication between you and Linus, but much of it, it seems, is out of the public view. Could you describe how the two of you work together? How are decisions (such as when to release) made?

Is there anything else you would like to say to LWN's readers?

We would like to thank Andrew for taking time to answer our questions.

Comments (39 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.26-rc5. (June 5, 2008)

• Chris Wright: Linux 2.6.25.5. (June 7, 2008)

• Andrew Morton: 2.6.26-rc5-mm1. (June 9, 2008)

• Andrew Morton: 2.6.26-rc5-mm2. (June 10, 2008)

• Steven Rostedt: 2.6.25.4-rt5. (June 9, 2008)

• Steven Rostedt: 2.6.24.7-rt13. (June 9, 2008)

• yoann padioleau: Full git history of Linux. (June 9, 2008)

• Steven Rostedt: 2.6.25.4-rt6. (June 9, 2008)

• Willy Tarreau: Linux 2.4.36.6. (June 9, 2008)

• Greg KH: linux-staging tree created. (June 10, 2008)

Core kernel code

• Ben Hutchings: cputopology: Always define CPU topology information [5th try]. (June 9, 2008)

• Oleg Nesterov: schedule: fix TASK_WAKEKILL vs SIGKILL race. (June 9, 2008)

• Dmitry Baryshkov: Clocklib: generic clocks management framework. (June 9, 2008)

• Peter Zijlstra: hotplug fixes. (June 10, 2008)

Development tools

• eranian@googlemail.com: perfmon2 minimal: introduction. (June 10, 2008)

Device drivers

• David Woodhouse: firmware: moving drivers to request_firmware(). (June 9, 2008)

• steiner@sgi.com: GRU Driver. (June 10, 2008)

Documentation

• Michael Kerrisk: man-pages-2.80 and man-pages-fr-2.80.0 are released. (June 9, 2008)

Filesystems and block I/O

• Alan D. Brunelle: Added in user-injected messages into blk traces. (June 9, 2008)

• Jan Kara: New split of quota headers. (June 9, 2008)

• Martin K. Petersen: Block/SCSI Data Integrity Support. (June 9, 2008)

Memory management

• npiggin@suse.de: speculative page references, lockless pagecache, lockless gup. (June 9, 2008)

• Nick Piggin: mm: vmap rewrite. (June 9, 2008)

• kosaki.motohiro@jp.fujitsu.com: page reclaim throttle v7. (June 9, 2008)

• Rik van Riel: VM pageout scalability improvements (V10). (June 9, 2008)

Architecture-specific

• Yinghai Lu: x86: make generic arch support NUMAQ v3. (June 9, 2008)

• Glauber Costa: Improve x86 smpboot integration. (June 9, 2008)

• Haavard Skinnemoen: Make at91_nand usable on AVR32. (June 9, 2008)

Virtualization and containers

• Keika Kobayashi: per-task-delay-accounting: add memory reclaim delay. (June 9, 2008)

• KOSAKI Motohiro: introduce task cgroup (#task restrictioon for prevent fork bomb by cgroup). (June 9, 2008)

Page editor: Jake Edge

Distributions

News and Editorials

openSUSE merges forums ahead of 11.0 release

The openSUSE project announced this week it has merged its three largest English-language community support forums under one big green umbrella and relaunched it as the openSUSE Forums. According to data supplied by openSUSE, the combined number of suseforums.net, suselinuxsupport.de, and openSUSE Novell support forum members was in the tens of thousands &mdash a number expected to rise with the upcoming release of openSUSE 11.0.

Even though the new forums are already up and running smoothly, the team has no intention of resting on its laurels. They're already working on implementing similar changes with forums in other languages and better integration with the rest of the site.

Project Manager Rupert Horstkötter says there are also plans for a "user-rating for the whole openSUSE community, integrated with forums.opensuse.org, and all other openSUSE services. Besides all of that, we hope to be able to attract more independent forum communities for the official openSUSE forums."

Keith Kastorff, the site admin for suseforums.net says the idea began to take shape during an openSUSE project meeting back in 2007. "A big topic was the need for an 'official' openSUSE forum, and the duplication of effort, expertise, and resources we had in play," he recalls. "I volunteered to reach out to some of the independent SUSE focused forums to see if I could generate any interest in a merge." Then he contacted people involved with Novell and suselinuxsupport.de and "things moved forward from there."

Kastorff says getting the project underway was slow going at first and admits that some members were wary of Novell's involvement. "The open source community is sometimes skeptical of commercial players, but we found nothing but tremendous support from Novell," he says.

It's not surprising there were a number of technical hurdles to overcome in bringing the three forums together. One of the main issues included an inability to merge the member databases and it was eventually decided to simply archive them within a section of the new forum. "Like any project, we had to make compromises to achieve the end goal," says Kastorff. "We knew going in we had different cultures in play, and there were times the dialogs between the various merging staffs got intense, but the team's strong commitment to bettering the openSUSE community kept us focused on the prize."

Indeed, it was a team effort. More than 30 people worked behind the scenes to import the help sections of the separate forums and archive over 400,000 posts prior to launching forums.opensusue.org. In order for the project to work, the various groups &mdash each with their own goals and ideas &mdash needed to work together and trust in the end goal.

Horstkötter says it was "a lot of work to combine different cultures into one big forum for the openSUSE community, but it was a great time. I feel like I met some new friends during the project."

"We had three teams &mdash one from Novell, two from different grassroots projects that had sprung up to serve the community and had developed their own style and ways of working together," recalls openSUSE Product Manager Michael Löffler "To merge the three, the staff for each forum had to be comfortable putting all their eggs in one basket (Novell hosting the forums) and agreeing on a common set of rules, moderation guidelines, etc. It took some time and effort to work everything out, but I think that the three teams are working quite well together now."

Just as important as teams working together is the impact that merged forums will have on the openSUSE community overall. "Having a unified forum means that all interested users can converse and support one another in one location &mdash so you don't have the duplication of effort." says Löffler. "I'm really glad [they] launched in time for 11.0 &mdash I expect that a lot of new users are going to be interested in openSUSE with this release, and I am very happy we have the forums to help support them."

Comments (none posted)

New Releases

64 Studio 2.1 'A Minha Menina' released!

The latest version of 64 Studio, 2.1 'A Minha Menina', has been released. "Version 2.1 is the first update to the second stable release of 64 Studio. It is named after a song by Jorge Ben, recorded by Os Mutantes and covered by The Bees." 64 Studio is a remix of Debian 4.0 'Etch', focused on digital content creation, including audio, video, graphics and publishing tools.

Full Story (comments: none)

Debian Installer lenny beta 2 released

The second beta of the Debian lenny installer is available for testing. Click below for a look at the improvements and known issues in this release.

Full Story (comments: none)

Mandriva Flash 2008 Spring released

Mandriva has announced the release of Mandriva Flash 2008 Spring, the new release of its bootable distribution on a USB key. "Mandriva Flash 2008 Spring is based on the new release of Mandriva Linux. It doubles the capacity of the key from 4GB for the previous version to 8GB, and comes in an attractive white casing. Flash 2008 Spring's new installation feature lets you install Mandriva Linux 2008 Spring permanently onto any system with just a few clicks." It's available now from the Mandriva Store.

Comments (none posted)

openSUSE Build Service 1.0 RC 1 released

The openSUSE Project has released the first release candidate of the openSUSE Build Service 1.0. "With the release candidate, all the features are now in place to support external collaboration with the community to build openSUSE in the open. Developers can now submit contributions to openSUSE directly at build.opensuse.org/."

Full Story (comments: none)

Distribution News

Fedora

Fedora Board Recap 2008-JUN-03

Here's a look at the June 3, 2008 meeting of the Fedora board. Topics include Codeina, Fedora 9 Postmortem, and Fedora Organizational Structure.

Full Story (comments: none)

Gentoo Linux

Nominations open for the Gentoo Council 2008/2009

Nominations for the Gentoo Council 2008/2009 are open now and will be open for the next two weeks. Only Gentoo developers may be nominated and only Gentoo developers may vote.

Full Story (comments: none)

Mandriva Linux

Celebrating 10 years of Mandriva

Mandriva just celebrated its Tenth Anniversary, both as a company and as a distribution. "The Mandriva community celebrated in style over the last weekend in May, with a party in the Eiffel Tower in Paris attended by many staff, former staff, community members and partners. There was also an - indoor - picnic, and the now-traditional Dance Dance Revolution party."

Comments (1 posted)

Other distributions

FreeBSD supported branches update

FreeBSD has announced an end-of-life for FreeBSD 5.5, FreeBSD 6.1, and FreeBSD 6.2. "Users of these releases are advised to upgrade promptly to FreeBSD 6.3 or FreeBSD 7.0, either by downloading an updated source tree and building updates manually, or (for i386 and amd64 systems) using the FreeBSD Update utility as described in the FreeBSD 6.3 and FreeBSD 7.0 release announcements."

Comments (none posted)

Distribution Newsletters

Ubuntu Weekly Newsletter #94

The Ubuntu Weekly Newsletter for June 7, 2008 covers Ubuntu Global Bug Jam, New Members, Newly Approved LoCos, Canonical Showcases Ubuntu Netbook Remix at Computex, Kubuntu Specs in Full, Ubuntu at OSCON, Ubuntu Server receives positive reviews, Mobile devices driving Ubuntu-Shuttleworth, Ubuntu UK podcast #7, Acer bets big on Linux, and much more.

Full Story (comments: none)

Fedora Weekly News Issue 130

The Fedora Weekly News for June 8, 2008 looks at an interview with Jim Whitehurst, Ubuntu 8.04 vs Fedora 9, LinuxTag Reports, Fedora Open Day 2008, and much more.

Full Story (comments: none)

DistroWatch Weekly, Issue 256

The DistroWatch Weekly for June 9, 2008 is out. "Ever since the launch of ASUS Linux Eee PC late last year, the ultra-portable computer market has turned into a major battleground of operating systems. Who will win? Microsoft with its thick wallet and pressure tactics or Linux with its low cost and open development model? Last week's Computex in Taipei revealed surprising differences between the ways hardware manufacturers embrace this exciting market. In the news section, Debian announces upcoming freeze of "Lenny", Mandriva celebrates its 10-year birthday, Canonical releases Ubuntu Netbook Remix, and FreeBSD updates the End-of-Life dates for its current and past releases. Also in this week's issue, a good collection of search resources for CentOS and RHEL users, and a list of valuable third-party repositories for openSUSE 11.0. Finally, with the annual package database update on DistroWatch, do let us know which new packages you want us to include in the tracking process."

Comments (none posted)

Distribution meetings

Recordings of Linuxtag '08

Recordings from talks that took place at the openSUSE day at Linuxtag are available online. Almost all are in German.

Full Story (comments: none)

Page editor: Rebecca Sobol

Development

Detect and record video movement with Motion

Motion is a video application that monitors a video4linux device such as a USB camera and records movement within the image:

An installation of Motion was performed on a machine with a 3Ghz Athlon 64 processor running Ubuntu 7.04 (Feisty Fawn). The most recent version of Motion (v 3.2.10.1) was downloaded, the file was uncompressed and untared. The normal configure, make and make install steps were performed. If one wishes to record mpeg movies, the libavcodec and libavformat libraries must be installed prior to running configure.

The make install step needed a bit of manual intervention, it was necessary to create the /var/run/motion directory and copy the motion-dist.conf configuration file to /usr/local/etc/motion.conf. The config file was modified to define a USB camera, the camera's default resolution was defined and the destination directory for images was set. The framerate parameter was changed to 2 seconds to slow down the rate of accumulation of image files.

A Kensington Model 67015 VideoCAM VGA USB camera was plugged into the computer. It is a good idea to run a real-time video monitoring application such as xawtv or EffecTV (in DumbTV mode) to adjust the camera's focus, brightness and contrast settings. Running Motion was simply a matter of typing "motion" on the command line. The program takes about 25 seconds to start recording movement, presumably most of this time is spent learning the contents of the video. After this delay, the software would output a line of text and create one .jpg file for each movement it detected. The images were inspected with the Mirage image viewer and a changing sequence of static images was observed.

Motion has a wide variety of capabilities and configurable parameters. The Motion Guide and Config File Options are a good place to read about the various capabilities and the FAQ gives answers to common questions.

One can imagine a number of uses for Motion, cube farm denizens could find out what is causing their pens to disappear at night, people in high crime areas could use it to catch vandals and thieves in the act. The on_picture_save configuration directive can execute a script on motion detection, this could be used