The state of KDE4

Kicking a software release out the door is always a chaotic time; the developers never feel like the software is ready while project management is trying to put a box around something that can be shipped. KDE 4, which is the first major K Desktop Environment release since KDE 3.5 in 2005, is currently in this stage. Questions are being asked about changing deadlines, beta versus alpha quality software, whether a stable release can be delivered on time, and so on. The questions reflect an underlying concern for the quality of the final release.

Part of the problem, it seems, is that earlier beta versions of KDE 4 did not get widespread use. There were enough fundamental problems that users, even those expecting a rough ride, were unable to get things going enough to start filing bugs. Torsten Rahn reported on some feedback he received at a show:

It is a difficult problem, a beta must be usable enough for people to work with it. But if it worked correctly all the time, it would already be released. For KDE 4, part of the difficulty is that the platform and underlying libraries are in pretty good shape, "release candidate" worthy, but the workspace (Plasma) and applications have lagged. Those are things that users see first, of course; fundamental bugs will turn them away from testing further.

One project that is helping get more beta testers is the KDE Four Live CD which is an openSUSE-based live distribution with KDE 4 as the desktop. Beta testers will not have to build everything from source, nor have to install anything on their machines. Screenshots from the Beta 4 release version of the live CD accompany this article.

There were some discussions on the kde-core-devel mailing list regarding whether to call things release candidates, betas, or alphas, with some advocating recognizing the quality of the code and sticking with a beta label. In response, Aaron Seigo makes a good point about the difference between software releases in the volunteer-driven free software world as opposed to paid developers in more traditional development shops:

The distinction is not quite as stark as Seigo paints it, "lines in the sand" are important tools for any project, volunteer or otherwise. It is the tension between getting it right and getting it into the hands of users that cause these conflicts as releases approach. There are an awful lot of "I can deal with that later" problems that come due. Determining which are critical and which can be deferred further is difficult and error-prone. It is much easier for developers to decide that they are all critical and the release needs to slip.

Sticking to previously established schedules can help by forcing things out the door, even if they aren't quite ready. Betas and release candidates are meant to be used, regardless of what they are called – they are also expected to have bugs. The flip side, of course, is that if a release is unusable, it will be largely ignored. This is what the early betas of KDE 4 seem to have run into.

In a casual test drive of the Beta 4 release, there were plenty of problems to be found, but, by and large, it worked. It doesn't seem ready to run as a day-to-day desktop, yet, but testers and others interested should be able to assist in finding and tracking down bugs. The existence of a live CD should help, as will integration with existing installations; imagine being able to boot the live CD and have it work with the existing users, preferences, filesystems, etc. on the underlying system. If a showstopping bug comes along, reboot to the installed OS and file a bug. Perhaps KDE Four Live does some of this integration with an installed openSUSE, but it did not with the Kubuntu installed on the test machine.

Conflicts in the development process often come to a head when a release is imminent, KDE 4 is hardly alone in seeing these kinds of disagreements. Software is difficult to develop and is even harder to release, opinions will differ on the best approach. In general, though, everyone has an interest in the same end result – a solid, working final release – keeping that in mind can help defuse things. The release of KDE 4 is planned for December 2007, we look forward to seeing it.

Comments (12 posted)

GNOME and OOXML

The OOXML document standard being pushed by Microsoft has caused a certain amount of stress within both the development and commercial sides of the free software community. In some quarters it is seen as the latest attempt by a monopolistic firm to co-opt free software and the move to more free file formats; they would like to limit our involvement to opposition to the adoption of OOXML as a standard. Others see it as an attempt by Microsoft to come to terms with the demand for more open formats and to promote, in its own special way, interoperability. Few people really think we need this particular format, but many feel that, given that it will exist, it might as well be documented and, to the extent possible, its development should be encouraged to go in relatively useful directions.

This debate returned to the foreground recently with the publication of this open letter to the GNOME Foundation on a site named, for better or worse, "Fanatic Attack." This letter begins:

The letter is long and strongly-worded, but it is rather short on information about just what the GNOME Foundation's participation in this process actually is. It turns out that the letter's author never asked that question, but LWN did. One answer can be found in this response posted by GNOME Foundation board member Jeff Waugh:

There is also an explanation from Jody Goldberg posted on the Foundation's mailing list:

It's worth noting that Mr. Goldberg does support the standardization of the OOXML format.

This episode has inspired a certain amount of complaint on the Foundation mailing list. The problem is not the participation in the committee, which appears to be relatively uncontroversial there, but the fact that this particular controversy was not anticipated and addressed ahead of time. Had the Foundation issued a press release at the outset explaining what it was doing, it would not have to be engaging in a damage control effort now. As it is, said press release appears to be under construction, but it will likely be less effective than it could have been.

In any case, this response will not satisfy everybody. There appears to be a fundamental difference of opinion in the community over how we should deal with the OOXML effort. While nobody seems to really like this standard (OK, almost nobody), not everybody dislikes it in the same way. To some, OOXML is characterized by patent problems, extreme complexity, opaque binary blobs, and the questionable tactics of its corporate backer. For those people, any engagement with the standardization process other than outright opposition is an unacceptable compromise. They see no good that can come from recognizing this standard in any way when we already have a standardized open document format which needs support.

On the other hand, the truth of the matter is that this format exists and is in use. Free software will end up supporting this format, not (just) because certain companies want to sell services into corporate environments, but because interoperability has always been a high priority in our community. If, some day, we as a community decree that we are strong enough that we do not need to support formats we don't like, we will have lost something important.

One should not overlook another important component in this situation: the fact that OpenDocument is not the final answer to document formats. Instead, it seems that the level of criticism of this format is growing, and that development of document formats will have to continue into the future. We do not, in other words, have all the answers in this area.

So, assuming that we, as a community, do intend to interoperate with the OOXML format, it makes sense to take advantage of the opportunities presented by the standardization process to ensure that the format is (1) not completely irrational, and (2) documented as completely as it can be. Participation in the process at this level has the potential to save a lot of work and interoperability hassles in the coming years.

Once upon a time, the free software community would have had no influence over a major manufacturer's file formats. We have succeeded in changing the world to the point where such formats are expected to be open, and where our comments on those formats have to be taken seriously. To refuse to wield that influence would, in essence, be a decision to go back to the days of the early 1990's, when our thoughts were mostly confined to a few small mailing lists and went generally unheard. That would not be a step forward for our community.

That said, participation in groups like standards bodies should be done the way we do almost everything else: in full openness. The GNOME Foundation exists to represent the community of GNOME developers, many of whom, it seems, were unaware that the Foundation was representing them in this particular forum. What form this representation has taken, and what has been accomplished by it, is still somewhat unclear; this lack of transparency has made the recent flames possible. The GNOME Foundation board, presumably, knows what positions are being taken by its representative on the ECMA committee; it would behoove that board to be more active in communicating that information to the Foundation's members.

Comments (29 posted)

Memory part 6: More things programmers can do

[Editor's note: this is part 6 of Ulrich Drepper's "What every programmer should know about memory"; this part contains the second half of section 6, covering the optimization of multi-threaded code. The first half of this section was published in part 5; please see part 1 for pointers to the other sections.]

6.4 Multi-Thread Optimizations

When it comes to multi-threading, there are three different aspects of cache use which are important:

• Concurrency

• Atomicity

• Bandwidth

These aspects also apply to multi-process situations but, because multiple processes are (mostly) independent, it is not so easy to optimize for them. The possible multi-process optimizations are a subset of those available for the multi-thread scenario. So we will deal exclusively with the latter here.

Concurrency in this context refers to the memory effects a process experiences when running more than one thread at a time. A property of threads is that they all share the same address space and, therefore, can all access the same memory. In the ideal case, the memory regions used by the threads are distinct, in which case those threads are coupled only lightly (common input and/or output, for instance). If more than one thread uses the same data, coordination is needed; this is when atomicity comes into play. Finally, depending on the machine architecture, the available memory and inter-processor bus bandwidth available to the processors is limited. We will handle these three aspects separately in the following sections—although they are, of course, closely linked.

6.4.1 Concurrency Optimizations

Initially, in this section, we will discuss two separate issues which actually require contradictory optimizations. A multi-threaded application uses common data in some of its threads. Normal cache optimization calls for keeping data together so that the footprint of the application is small, thus maximizing the amount of memory which fits into the caches at any one time.

There is a problem with this approach, though: if multiple threads write to a memory location, the cache line must be in ‘E’ (exclusive) state in the L1d of each respective core. This means that a lot of RFO requests are sent, in the worst case one for each write access. So a normal write will be suddenly very expensive. If the same memory location is used, synchronization is needed (maybe through the use of atomic operations, which is handled in the next section). The problem is also visible, though, when all the threads are using different memory locations and are supposedly independent.

Figure 6.10: Concurrent Cache Line Access Overhead

Figure 6.10 shows the results of this “false sharing”. The test program (shown in Section 9.3) creates a number of threads which do nothing but increment a memory location (500 million times). The measured time is from the program start until the program finishes after waiting for the last thread. The threads are pinned to individual processors. The machine has four P4 processors. The blue values represent runs where the memory allocations assigned to each thread are on separate cache lines. The red part is the penalty occurred when the locations for the threads are moved to just one cache line.

The blue measurements (when using individual cache lines) match what one would expect. The program scales without penalty to many threads. Each processor keeps its cache line in its own L1d and there are no bandwidth issues since not much code or data has to be read (in fact, it is all cached). The measured slight increase is really system noise and probably some prefetching effects (the threads use sequential cache lines).

The measured overhead, computed by dividing the time needed when using one cache line versus a separate cache line for each thread, is 390%, 734%, and 1,147% respectively. These large numbers might be surprising at first sight but, when thinking about the cache interaction needed, it should be obvious. The cache line is pulled from one processor's cache just after it has finished writing to the cache line. All processors, except the one which has the cache line at any given moment, are delayed and cannot do anything. Each additional processor will just cause more delays.

It is clear from these measurements that this scenario must be avoided in programs. Given the huge penalty, this problem is, in many situations, obvious (profiling will show the code location, at least) but there is a pitfall with modern hardware. Figure 6.11 shows the equivalent measurements when running the code on a single processor, quad core machine (Intel Core 2 QX 6700). Even with this processor's two separate L2s the test case does not show any scalability issues. There is a slight overhead when using the same cache line more than once but it does not increase with the number of cores. {I cannot explain the lower number when all four cores are used but it is reproducible.} If more than one of these processors were used we would, of course, see results similar to those in Figure 6.10. Despite the increasing use of multi-core processors, many machines will continue to use multiple processors and, therefore, it is important to handle this scenario correctly, which might mean testing the code on real SMP machines.

Figure 6.11: Overhead, Quad Core

There is a very simple “fix” for the problem: put every variable on its own cache line. This is where the conflict with the previously mentioned optimization comes into play, specifically, the footprint of the application would increase a lot. This is not acceptable; it is therefore necessary to come up with a more intelligent solution.

What is needed is to identify which variables are used by only one thread at a time, those used by only one thread ever, and maybe those which are contested at times. Different solutions for each of these scenarios are possible and useful. The most basic criterion for the differentiation of variables is: are they ever written to and how often does this happen.

Variables which are never written to and those which are only initialized once are basically constants. Since RFO requests are only needed for write operations, constants can be shared in the cache (‘S’ state). So, these variables do not have to be treated specially; grouping them together is fine. If the programmer marks the variables correctly with const, the tool chain will move the variables away from the normal variables into the .rodata (read-only data) or .data.rel.ro (read-only after relocation) section {Sections, identified by their names are the atomic units containing code and data in an ELF file.} No other special action is required. If, for some reason, variables cannot be marked correctly with const, the programmer can influence their placement by assigning them to a special section.

When the linker constructs the final binary, it first appends the sections with the same name from all input files; those sections are then arranged in an order determined by the linker script. This means that, by moving all variables which are basically constant but are not marked as such into a special section, the programmer can group all of those variables together. There will not be a variable which is often written to between them. By aligning the first variable in that section appropriately, it is possible to guarantee that no false sharing happens. Assume this little example:

  int foo = 1;
  int bar __attribute__((section(".data.ro"))) = 2;
  int baz = 3;
  int xyzzy __attribute__((section(".data.ro"))) = 4;

If compiled, this input file defines four variables. The interesting part is that the variables foo and baz, and bar and xyzzy are grouped together respectively. Without the attribute definitions the compiler would allocate all four variables in the sequence in which they are defined in the source code the a section named .data. {This is not guaranteed by the ISO C standard but it is how gcc works.} With the code as-is the variables bar and xyzzy are placed in a section named .data.ro. The section name .data.ro is more or less arbitrary. A prefix of .data. guarantees that the GNU linker will place the section together with the other data sections.

The same technique can be applied to separate out variables which are mostly read but occasionally written. Simply choose a different section name. This separation seems to make sense in some cases like the Linux kernel.

If a variable is only ever used by one thread, there is another way to specify the variable. In this case it is possible and useful to use thread-local variables (see [mytls]). The C and C++ language in gcc allow variables to be defined as per-thread using the __thread keyword.

  int foo = 1;
  __thread int bar = 2;
  int baz = 3;
  __thread int xyzzy = 4;

The variables bar and xyzzy are not allocated in the normal data segment; instead each thread has its own separate area where such variables are stored. The variables can have static initializers. All thread-local variables are addressable by all other threads but, unless a thread passes a pointer to a thread-local variable to those other threads, there is no way the other threads can find that variable. Due to the variable being thread-local, false sharing is not a problem—unless the program artificially creates a problem. This solution is easy to set up (the compiler and linker do all the work), but it has its cost. When a thread is created, it has to spend some time on setting up the thread-local variables, which requires time and memory. In addition, addressing thread-local variables is usually more expensive than using global or automatic variables (see [mytls] for explanations of how the costs are minimized automatically, if possible).

One drawback of using thread-local storage (TLS) is that, if the use of the variable shifts over to another thread, the current value of the variable in the old thread is not available to new thread. Each thread's copy of the variable is distinct. Often this is not a problem at all and, if it is, the shift over to the new thread needs coordination, at which time the current value can be copied.

A second, bigger problem is possible waste of resources. If only one thread ever uses the variable at any one time, all threads have to pay a price in terms of memory. If a thread does not use any TLS variables, the lazy allocation of the TLS memory area prevents this from being a problem (except for TLS in the application itself). If a thread uses just one TLS variable in a DSO, the memory for all the other TLS variables in this object will be allocated, too. This could potentially add up if TLS variables are used on a large scale.

In general the best advice which can be given is

1. Separate at least read-only (after initialization) and read-write variables. Maybe extend this separation to read-mostly variables as a third category.

2. Group read-write variables which are used together into a structure. Using a structure is the only way to ensure the memory locations for all of those variables are close together in a way which is translated consistently by all gcc versions..

3. Move read-write variables which are often written to by different threads onto their own cache line. This might mean adding padding at the end to fill a remainder of the cache line. If combined with step 2, this is often not really wasteful. Extending the example above, we might end up with code as follows (assuming bar and xyzzy are meant to be used together):

     int foo = 1;
     int baz = 3;
     struct {
       struct al1 {
         int bar;
         int xyzzy;
       };
       char pad[CLSIZE - sizeof(struct al1)];
     } rwstruct __attribute__((aligned(CLSIZE))) =
       { { .bar = 2, .xyzzy = 4 } };
   

   Some code changes are needed (references to bar have to be replaced with rwstruct.bar, likewise for xyzzy) but that is all. The compiler and linker do all the rest. {This code has to be compiled with -fms-extensions} on the command line.}

4. If a variable is used by multiple threads, but every use is independent, move the variable into TLS.

6.4.2 Atomicity Optimizations

If multiple threads modify the same memory location concurrently, processors do not guarantee any specific result. This is a deliberate decision made to avoid costs which are unnecessary in 99.999% of all cases. For instance, if a memory location is in the ‘S’ state and two threads concurrently have to increment its value, the execution pipeline does not have to wait for the cache line to be available in the ‘E’ state before reading the old value from the cache to perform the addition. Instead it reads the value currently in the cache and, once the cache line is available in state ‘E’, the new value is written back. The result is not as expected if the two cache reads in the two threads happen simultaneously; one addition will be lost.

To assure this does not happen, processors provide atomic operations. These atomic operations would, for instance, not read the old value until it is clear that the addition could be performed in a way that the addition to the memory location appears as atomic. In addition to waiting for other cores and processors, some processors even signal atomic operations for specific addresses to other devices on the motherboard. All this makes atomic operations slower.

Processor vendors decided to provide different sets of atomic operations. Early RISC processors, in line with the ‘R’ for reduced, provided very few atomic operations, sometimes only an atomic bit set and test. {HP Parisc still does not provide more…} At the other end of the spectrum, we have x86 and x86-64 which provide a large number of atomic operations. The generally available atomic operations can be categorized in four classes:

Bit Test

These operations set or clear a bit atomically and return a status indicating whether the bit was set before or not.

Load Lock/Store Conditional (LL/SC)

{Some people use “linked” instead of “lock”, it is all the same.}

These operations work as a pair where the special load instruction is used to start an transaction and the final store will only succeed if the location has not been modified in the meantime. The store operation indicates success or failure, so the program can repeat its efforts if necessary.

Compare-and-Swap (CAS)

This is a ternary operation which writes a value provided as a parameter into an address (the second parameter) only if the current value is the same as the third parameter value;

Atomic Arithmetic

These operations are only available on x86 and x86-64, which can perform arithmetic and logic operations on memory locations. These processors have support for non-atomic versions of these operations but RISC architectures do not. So it is no wonder that their availability is limited.

An architecture supports either the LL/SC or the CAS instruction, not both. Both approaches are basically equivalent; they allow the implementation of atomic arithmetic operations equally well, but CAS seems to be the preferred method these days. All other operations can be indirectly implemented using it. For instance, an atomic addition:

  int curval;
  int newval;
  do {
    curval = var;
    newval = curval + addend;
  } while (CAS(&var, curval, newval));

The result of the CAS call indicates whether the operation succeeded or not. If it returns failure (non-zero value), the loop is run again, the addition is performed, and the CAS call is tried again. This repeats until it is successful. Noteworthy about the code is that the address of the memory location has to be computed in two separate instructions. {The CAS opcode on x86 and x86-64 can avoid the load of the value in the second and later iterations but, on this platform, we can write the atomic addition in a simpler way, with a single addition opcode.} For LL/SC the code looks about the same.

  int curval;
  int newval;
  do {
    curval = LL(var);
    newval = curval + addend;
  } while (SC(var, newval));

Here we have to use a special load instruction (LL) and we do not have to pass the current value of the memory location to SC since the processor knows if the memory location has been modified in the meantime.

The big differentiators are x86 and x86-64 where we have the atomic operations and, here, it is important to select the proper atomic operation to achieve the best result. Figure 6.12 shows three different ways to implement an atomic increment operation.

for (i = 0; i < N; ++i) __sync_add_and_fetch(&var,1); 1. Add and Read Result

for (i = 0; i < N; ++i) __sync_fetch_and_add(&var,1); 2. Add and Return Old Value

for (i = 0; i < N; ++i) { long v, n; do { v = var; n = v + 1; } while (!__sync_bool_compare_and_swap(&var, v, n)); } 3. Atomic Replace with New Value

Figure 6.12: Atomic Increment in a Loop

All three produce different code on x86 and x86-64 while the code might be identical on other architectures. There are huge performance differences. The following table shows the execution time for 1 million increments by four concurrent threads. The code uses the built-in primitives of gcc (__sync_*).

1. Exchange Add	2. Add Fetch	3. CAS
0.23s	0.21s	0.73s

The first two numbers are similar; we see that returning the old value is a little bit faster. The important piece of information is the highlighted field, the cost when using CAS. It is, unsurprisingly, a lot more expensive. There are several reasons for this: 1. there are two memory operations, 2. the CAS operation by itself is more complicated and requires even conditional operation, and 3. the whole operation has to be done in a loop in case two concurrent accesses cause a CAS call to fail.

Now a reader might ask a question: why would somebody use the complicated and longer code which utilizes CAS? The answer to this is: the complexity is usually hidden. As mentioned before, CAS is currently the unifying atomic operation across all interesting architectures. So some people think it is sufficient to define all atomic operations in terms of CAS. This makes programs simpler. But as the numbers show, the results can be everything but optimal. The memory handling overhead of the CAS solution is huge. The following illustrates the execution of just two threads, each on its own core.

Thread #1	Thread #2	var Cache State
v = var		‘E’ on Proc 1
n = v + 1	v = var	‘S’ on Proc 1+2
CAS(var)	n = v + 1	‘E’ on Proc 1
	CAS(var)	‘E’ on Proc 2

We see that, within this short period of execution, the cache line status changes at least three times; two of the changes are RFOs. Additionally, the second CAS will fail, so that thread has to repeat the whole operation. During that operation the same can happen again.

In contrast, when the atomic arithmetic operations are used, the processor can keep the load and store operations needed to perform the addition (or whatever) together. It can ensure that concurrently-issued cache line requests are blocked until the atomic operation is done. Each loop iteration in the example therefore results in, at most, one RFO cache request and nothing else.

What all this means is that it is crucial to define the machine abstraction at a level at which atomic arithmetic and logic operations can be utilized. CAS should not be universally used as the unification mechanism.

For most processors, the atomic operations are, by themselves, always atomic. One can avoid them only by providing completely separate code paths for the case when atomicity is not needed. This means more code, a conditional, and further jumps to direct execution appropriately.

For x86 and x86-64 the situation is different: the same instructions can be used in both atomic and non-atomic ways. To make them atomic, a special prefix for the instruction is used: the lock prefix. This opens the door for atomic operations to avoid the high costs if the atomicity requirement in a given situation is not needed. Generic code in libraries, for example, which always has to be thread-safe if needed, can benefit from this. No information is needed when writing the code, the decision can be made at runtime. The trick is to jump over the lock prefix. This trick applies to all the instructions which the x86 and x86-64 processor allow to prefix with lock.

      cmpl $0, multiple_threads
      je   1f
      lock
  1:  add  $1, some_var

If this assembler code appears cryptic, do not worry, it is simple. The first instruction checks whether a variable is zero or not. Nonzero in this case indicates that more than one thread is running. If the value is zero, the second instruction jumps to label 1. Otherwise, the next instruction is executed. This is the tricky part. If the je instruction does not jump, the add instruction is executed with the lock prefix. Otherwise it is executed without the lock prefix.

Adding a relatively expensive operation like a conditional jump (expensive in case the branch prediction fails) seems to be counter productive. Indeed it can be: if multiple threads are running most of the time, the performance is further decreased, especially if the branch prediction is not correct. But if there are many situations where only one thread is in use, the code is significantly faster. The alternative of using an if-then-else construct introduces an additional unconditional jump in both cases which can be slower. Given that an atomic operation costs on the order of 200 cycles, the cross-over point for using the trick (or the if-then-else block) is pretty low. This is definitely a technique to be kept in mind. Unfortunately this means gcc's __sync_* primitives cannot be used.

6.4.3 Bandwidth Considerations

When many threads are used, and they do not cause cache contention by using the same cache lines on different cores, there still are potential problems. Each processor has a maximum bandwidth to the memory which is shared by all cores and hyper-threads on that processor. Depending on the machine architecture (e.g., the one in Figure 2.1), multiple processors might share the same bus to memory or the Northbridge.

The processor cores themselves run at frequencies where, at full speed, even in perfect conditions, the connection to the memory cannot fulfill all load and store requests without waiting. Now, further divide the available bandwidth by the number of cores, hyper-threads, and processors sharing a connection to the Northbridge and suddenly parallelism becomes a big problem. Programs which are, in theory, very efficient may be limited by the memory bandwidth.

In Figure 3.32 we have seen that increasing the FSB speed of a processor can help a lot. This is why, with growing numbers of cores on a processor, we will also see an increase in the FSB speed. Still, this will never be enough if the program uses large working sets and it is sufficiently optimized. Programmers have to be prepared to recognize problems due to limited bandwidth.

The performance measurement counters of modern processors allow the observation of FSB contention. On Core 2 processors the NUS_BNR_DRV event counts the number of cycles a core has to wait because the bus is not ready. This indicates that the bus is highly used and loads from or stores to main memory take even longer than usual. The Core 2 processors support more events which can count specific bus actions like RFOs or the general FSB utilization. The latter might come in handy when investigating the possibility of scalability of an application during development. If the bus utilization rate is already close to 1.0 then the scalability opportunities are minimal.

If a bandwidth problem is recognized, there are several things which can be done. They are sometimes contradictory so some experimentation might be necessary. One solution is to buy faster computers, if there are some available. Getting more FSB speed, faster RAM modules, and possibly memory local to the processor, can—and probably will—help. It can cost a lot, though. If the program in question is only needed on one (or a few machines) the one-time expense for the hardware might cost less than reworking the program. In general, though, it is better to work on the program.

After optimizing the program itself to avoid cache misses, the only option left to achieve better bandwidth utilization is to place the threads better on the available cores. By default, the scheduler in the kernel will assign a thread to a processor according to its own policy. Moving a thread from one core to another is avoided when possible. The scheduler does not really know anything about the workload, though. It can gather information from cache misses etc but this is not much help in many situations.

Figure 6.13: Inefficient Scheduling

One situation which can cause big FSB usage is when two threads are scheduled on different processors (or cores which do not share a cache) and they use the same data set. Figure 6.13 shows such a situation. Core 1 and 3 access the same data (indicated by the same color for the access indicator and the memory area). Similarly core 2 and 4 access the same data. But the threads are scheduled on different processors. This means each data set has to be read twice from memory. This situation can be handled better.

Figure 6.14: Efficient Scheduling

In Figure 6.14 we see how it should ideally look like. Now the total cache size in use is reduced since now core 1 and 2 and core 3 and 4 work on the same data. The data sets have to be read from memory only once.

This is a simple example but, by extension, it applies to many situations. As mentioned before, the scheduler in the kernel has no insight into the use of data, so the programmer has to ensure that scheduling is done efficiently. There are not many kernel interfaces available to communicate this requirement. In fact, there is only one: defining thread affinity.

Thread affinity means assigning a thread to one or more cores. The scheduler will then choose among those cores (only) when deciding where to run the thread. Even if other cores are idle they will not be considered. This might sound like a disadvantage, but it is the price one has to pay. If too many threads exclusively run on a set of cores the remaining cores might mostly be idle and there is nothing one can do except change the affinity. By default threads can run on any core.

There are a number of interfaces to query and change the affinity of a thread:

#define _GNU_SOURCE
#include <sched.h>

int sched_setaffinity(pid_t pid, size_t size, const cpu_set_t *cpuset);
int sched_getaffinity(pid_t pid, size_t size, cpu_set_t *cpuset);

These two interfaces are meant to be used for single-threaded code. The pid argument specifies which process's affinity should be changed or determined. The caller obviously needs appropriate privileges to do this. The second and third parameter specify the bitmask for the cores. The first function requires the bitmask to be filled in so that it can set the affinity. The second fills in the bitmask with the scheduling information of the selected thread. The interfaces are declared in <sched.h>.

The cpu_set_t type is also defined in that header, along with a number of macros to manipulate and use objects of this type.

#define _GNU_SOURCE
#include <sched.h>

#define CPU_SETSIZE
#define CPU_SET(cpu, cpusetp)
#define CPU_CLR(cpu, cpusetp)
#define CPU_ZERO(cpusetp)
#define CPU_ISSET(cpu, cpusetp)
#define CPU_COUNT(cpusetp)

CPU_SETSIZE specifies how many CPUs can be represented in the data structure. The other three macros manipulate cpu_set_t objects. To initialize an object CPU_ZERO should be used; the other two macros should be used to select or deselect individual cores. CPU_ISSET tests whether a specific processor is part of the set. CPU_COUNT returns the number of cores selected in the set. The cpu_set_t type provide a reasonable default value for the upper limit on the number of CPUs. Over time it certainly will prove too small; at that point the type will be adjusted. This means programs always have to keep the size in mind. The above convenience macros implicitly handle the size according to the definition of cpu_set_t. If more dynamic size handling is needed an extended set of macros should be used:

#define _GNU_SOURCE
#include <sched.h>

#define CPU_SET_S(cpu, setsize, cpusetp)
#define CPU_CLR_S(cpu, setsize, cpusetp)
#define CPU_ZERO_S(setsize, cpusetp)
#define CPU_ISSET_S(cpu, setsize, cpusetp)
#define CPU_COUNT_S(setsize, cpusetp)

These interfaces take an additional parameter with the size. To be able to allocate dynamically sized CPU sets three macros are provided:

#define _GNU_SOURCE
#include <sched.h>

#define CPU_ALLOC_SIZE(count)
#define CPU_ALLOC(count)
#define CPU_FREE(cpuset)

The CPU_ALLOC_SIZE macro returns the number of bytes which have to be allocated for a cpu_set_t structure which can handle count CPUs. To allocate such a block the CPU_ALLOC macro can be used. The memory allocated this way should be freed with CPU_FREE. The functions will likely use malloc and free behind the scenes but this does not necessarily have to remain this way.

Finally, a number of operations on CPU set objects are defined:

#define _GNU_SOURCE
#include <sched.h>

#define CPU_EQUAL(cpuset1, cpuset2)
#define CPU_AND(destset, cpuset1, cpuset2)
#define CPU_OR(destset, cpuset1, cpuset2)
#define CPU_XOR(destset, cpuset1, cpuset2)
#define CPU_EQUAL_S(setsize, cpuset1, cpuset2)
#define CPU_AND_S(setsize, destset, cpuset1, cpuset2)
#define CPU_OR_S(setsize, destset, cpuset1, cpuset2)
#define CPU_XOR_S(setsize, destset, cpuset1, cpuset2)

These two sets of four macros can check two sets for equality and perform logical AND, OR, and XOR operations on sets. These operations come in handy when using some of the libNUMA functions (see Section 12).

A process can determine on which processor it is currently running using the sched_getcpu interface:

#define _GNU_SOURCE
#include <sched.h>
int sched_getcpu(void);

The result is the index of the CPU in the CPU set. Due to the nature of scheduling this number cannot always be 100% correct. The thread might have been moved to a different CPU between the time the result was returned and when the thread returns to userlevel. Programs always have to take this possibility of inaccuracy into account. More important is, in any case, the set of CPUs the thread is allowed to run on. This set can be retrieved using sched_getaffinity. The set is inherited by child threads and processes. Threads cannot rely on the set to be stable over the lifetime. The affinity mask can be set from the outside (see the pid parameter in the prototypes above); Linux also supports CPU hot-plugging which means CPUs can vanish from the system—and, therefore, also from the affinity CPU set.

In multi-threaded programs, the individual threads officially have no process ID as defined by POSIX and, therefore, the two functions above cannot be used. Instead <pthread.h> declares four different interfaces:

#define _GNU_SOURCE
#include <pthread.h>

int pthread_setaffinity_np(pthread_t th, size_t size,
                           const cpu_set_t *cpuset);
int pthread_getaffinity_np(pthread_t th, size_t size, cpu_set_t *cpuset);
int pthread_attr_setaffinity_np(pthread_attr_t *at,
                                size_t size, const cpu_set_t *cpuset);
int pthread_attr_getaffinity_np(pthread_attr_t *at, size_t size, 
                                cpu_set_t *cpuset);

The first two interfaces are basically equivalent to the two we have already seen, except that they take a thread handle in the first parameter instead of a process ID. This allows addressing individual threads in a process. It also means that these interfaces cannot be used from another process, they are strictly for intra-process use. The third and fourth interfaces use a thread attribute. These attributes are used when creating a new thread. By setting the attribute, a thread can be scheduled from the start on a specific set of CPUs. Selecting the target processors this early—instead of after the thread already started—can be of advantage on many different levels, including (and especially) memory allocation (see NUMA in Section 6.5).

Speaking of NUMA, the affinity interfaces play a big role in NUMA programming, too. We will come back to that case shortly.

So far, we have talked about the case where the working set of two threads overlaps such that having both threads on the same core makes sense. The opposite can be true, too. If two threads work on separate data sets, having them scheduled on the same core can be a problem. Both threads fight for the same cache, thereby reducing each others effective use of the cache. Second, both data sets have to be loaded into the same cache; in effect this increases the amount of data that has to be loaded and, therefore, the available bandwidth is cut in half.

The solution in this case is to set the affinity of the threads so that they cannot be scheduled on the same core. This is the opposite from the previous situation, so it is important to understand the situation one tries to optimize before making any changes.

Optimizing for cache sharing to optimize bandwidth is in reality an aspect of NUMA programming which is covered in the next section. One only has to extend the notion of “memory” to the caches. This will become ever more important once the number of levels of cache increases. For this reason, the solution to multi-core scheduling is available in the NUMA support library. See the code samples in Section 12 for ways to determine the affinity masks without hardcoding system details or diving into the depth of the /sys filesystem.

6.5 NUMA Programming

For NUMA programming everything said so far about cache optimizations applies as well. The differences only start below that level. NUMA introduces different costs when accessing different parts of the address space. With uniform memory access we can optimize to minimize page faults (see Section 7.5) but that is about it. All pages are created equal.

NUMA changes this. Access costs can depend on the page which is accessed. Differing access costs also increase the importance of optimizing for memory page locality. NUMA is inevitable for most SMP machines since both Intel with CSI (for x86,x86-64, and IA-64) and AMD (for Opteron) use it. With an increasing number of cores per processor we are likely to see a sharp reduction of SMP systems being used (at least outside data centers and offices of people with terribly high CPU usage requirements). Most home machines will be fine with just one processor and hence no NUMA issues. But this a) does not mean programmers can ignore NUMA and b) it does not mean there are not related issues.

If one thinks about generalizations to NUMA one quickly realizes the concept extends to processor caches as well. Two threads on cores using the same cache will collaborate faster than threads on cores not sharing a cache. This is not a fabricated case:

• early dual-core processors had no L2 sharing.

• Intel's Core 2 QX 6700 and QX 6800 quad core chips, for instance, have two separate L2 caches.

• as speculated early, with more cores on a chip and the desire to unify caches, we will have more levels of caches.

Caches form their own hierarchy, and placement of threads on cores becomes important for sharing (or not) of caches. This is not very different from the problems NUMA is facing and, therefore, the two concepts can be unified. Even people only interested in non-SMP machines should therefore read this section.

In Section 5.3 we have seen that the Linux kernel provides a lot of information which is useful—and needed—in NUMA programming. Collecting this information is not that easy, though. The currently available NUMA library on Linux is wholly inadequate for this purpose. A much more suitable version is currently under construction by the author.

The existing NUMA library, libnuma, part of the numactl package, provides no access to system architecture information. It is only a wrapper around the available system calls together with some convenience interfaces for commonly used operations. The system calls available on Linux today are:

mbind

Select binding for specified memory pages to nodes.

set_mempolicy

Set the default memory binding policy.

get_mempolicy

Get the default memory binding policy.

migrate_pages

Migrate all pages of a process on a given set of nodes to a different set of nodes.

move_pages

Move selected pages to given node or request node information about pages.

These interfaces are declared in <numaif.h> which comes along with the libnuma library. Before we go into more details we have to understand the concept of memory policies.

6.5.1 Memory Policy

The idea behind defining a memory policy is to allow existing code to work reasonably well in a NUMA environment without major modifications. The policy is inherited by child processes, which makes it possible to use the numactl tool. This tool can be used to, among other things, start a program with a given policy.

The Linux kernel supports the following policies:

MPOL_BIND

Memory is allocated only from the given set of nodes. If this is not possible allocation fails.

MPOL_PREFERRED

Memory is preferably allocated from the given set of nodes. If this fails memory from other nodes is considered.

MPOL_INTERLEAVE

Memory is allocated equally from the specified nodes. The node is selected either by the offset in the virtual memory region for VMA-based policies, or through a free-running counter for task-based policies.

MPOL_DEFAULT

Choose the allocation based on the default for the region.

This list seems to recursively define policies. This is half true. In fact, memory policies form a hierarchy (see Figure 6.15).

Figure 6.15: Memory Policy Hierarchy

If an address is covered by a VMA policy then this policy is used. A special kind of policy is used for shared memory segments. If no policy for the specific address is present, the task's policy is used. If this is also not present the system's default policy is used.

The system default is to allocate memory local to the thread requesting the memory. No task and VMA policies are provided by default. For a process with multiple threads the local node is the “home” node, the one which first ran the process. The system calls mentioned above can be used to select different policies.

6.5.2 Specifying Policies

The set_mempolicy call can be used to set the task policy for the current thread (task in kernel-speak). Only the current thread is affected, not the entire process.

#include <numaif.h>

long set_mempolicy(int mode, 
                   unsigned long *nodemask, 
		   unsigned long maxnode);

The mode parameter must be one of the MPOL_* constants introduced in the previous section. The nodemask parameter specifies the memory nodes to use and maxnode is the number of nodes (i.e., bits) in nodemask. If MPOL_DEFAULT is used the nodemask parameter is ignored. If a null pointer is passed as nodemask for MPOL_PREFERRED the local node is selected. Otherwise MPOL_PREFERRED uses the lowest node number with the corresponding bit set in nodemask.

Setting a policy does not have any effect on already-allocated memory. Pages are not automatically migrated; only future allocations are affected. Note the difference between memory allocation and address space reservation: an address space region established using mmap is usually not automatically allocated. The first read or write operation on the memory region will allocate the appropriate page. If the policy changes between accesses to different pages of the same address space region, or if the policy allows allocation of memory from different nodes, a seemingly uniform address space region might be scattered across many memory nodes.

6.5.3 Swapping and Policies

If physical memory runs out, the system has to drop clean pages and save dirty pages to swap. The Linux swap implementation discards node information when it writes pages to swap. That means when the page is reused and paged in the node which is used will be chosen from scratch. The policies for the thread will likely cause a node which is close to the executing processors to be chosen, but the node might be different from the one used before.

This changing association means that the node association cannot be stored by a program as a property of the page. The association can change over time. For pages which are shared with other processes this can also happen because a process asks for it (see the discussion of mbind below). The kernel by itself can migrate pages if one node runs out of space while other nodes still have free space.

Any node association the user-level code learns about can therefore be true for only a short time. It is more of a hint than absolute information. Whenever accurate knowledge is required the get_mempolicy interface should be used (see Section 6.5.5).

6.5.4 VMA Policy

To set the VMA policy for an address range a different interface has to be used:

#include <numaif.h>

long mbind(void *start, unsigned long len,
           int mode,
           unsigned long *nodemask,
           unsigned long maxnode,
           unsigned flags);

This interface registers a new VMA policy for the address range [start, start + len). Since memory handling operates on pages the start address must be page-aligned. The len value is rounded up to the next page size.

The mode parameter specifies, again, the policy; the values must be chosen from the list in Section 6.5.1. As with set_mempolicy, the nodemask parameter is only used for some policies. Its handling is identical.

The semantics of the mbind interface depends on the value of the flags parameter. By default, if flags is zero, the system call sets the VMA policy for the address range. Existing mappings are not affected. If this is not sufficient there are currently three flags to modify this behavior; they can be selected individually or together:

MPOL_MF_STRICT

The call to mbind will fail if not all pages are on the nodes specified by nodemask. In case this flag is used together with MPOL_MF_MOVE and/or MPOL_MF_MOVEALL the call will fail if any page cannot be moved.

MPOL_MF_MOVE

The kernel will try to move any page in the address range allocated on a node not in the set specified by nodemask. By default, only pages used exclusively by the current process's page tables are moved.

MPOL_MF_MOVEALL

Like MPOL_MF_MOVE but the kernel will try to move all pages, not just those used by the current process's page tables alone. This has system-wide implications since it influences the memory access of other processes—which are possibly not owned by the same user—as well. Therefore MPOL_MF_MOVEALL is a privileged operation (CAP_NICE capability is needed).

Note that support for MPOL_MF_MOVE and MPOL_MF_MOVEALL was added only in the 2.6.16 Linux kernel.

Calling mbind without any flags is most useful when the policy for a newly reserved address range has to be specified before any pages are actually allocated.

void *p = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0);
if (p != MAP_FAILED)
  mbind(p, len, mode, nodemask, maxnode, 0);

This code sequence reserve an address space range of len bytes and specifies that the policy mode referencing the memory nodes in nodemask should be used. Unless the MAP_POPULATE flag is used with mmap, no memory will have been allocated by the time of the mbind call and, therefore, the new policy applies to all pages in that address space region.

The MPOL_MF_STRICT flag alone can be used to determine whether any page in the address range described by the start and len parameters to mbind is allocated on nodes other than those specified by nodemask. No allocated pages are changed. If all pages are allocated on the specified nodes, the VMA policy for the address space region will be changed according to mode.

Sometimes the rebalancing of memory is needed, in which case it might be necessary to move pages allocated on one node to another node. Calling mbind with MPOL_MF_MOVE set makes a best effort to achieve that. Only pages which are solely referenced by the process's page table tree are considered for moving. There can be multiple users in the form of threads or other processes which share that part of the page table tree. It is not possible to affect other processes which happen to map the same data. These pages do not share the page table entries.

If both MPOL_MF_STRICT and MPOL_MF_MOVE are passed to mbind the kernel will try to move all pages which are not allocated on the specified nodes. If this is not possible the call will fail. Such a call might be useful to determine whether there is a node (or set of nodes) which can house all the pages. Several combinations can be tried in succession until a suitable node is found.

The use of MPOL_MF_MOVEALL is harder to justify unless running the current process is the main purpose of the computer. The reason is that even pages that appear in multiple page tables are moved. That can easily affect other processes in a negative way. This operation should thus be used with caution.

6.5.5 Querying Node Information

The get_mempolicy interface can be used to query a variety of facts about the state of NUMA for a given address.

#include <numaif.h>
long get_mempolicy(int *policy,
             const unsigned long *nmask,
             unsigned long maxnode,
             void *addr, int flags);

When get_mempolicy is called without a flag set in flags, the information about the policy for address addr is stored in the word pointed to by policy and in the bitmask for the nodes pointed to by nmask. If addr falls into an address space region for which a VMA policy has been specified, information about that policy is returned. Otherwise information about the task policy or, if necessary, system default policy will be returned.

If the MPOL_F_NODE flag is set in flags, and the policy governing addr is MPOL_INTERLEAVE, the value stored in the word pointed to by policy is the index of the node on which the next allocation is going to happen. This information can potentially be used to set the affinity of a thread which is going to work on the newly-allocated memory. This might be a less costly way to achieve proximity, especially if the thread has yet to be created.

The MPOL_F_ADDR flag can be used to retrieve yet another completely different data item. If this flag is used, the value stored in the word pointed to by policy is the index of the memory node on which the memory for the page containing addr has been allocated. This information can be used to make decisions about possible page migration, to decide which thread could work on the memory location most efficiently, and many more things.

The CPU—and therefore memory node—a thread is using is much more volatile than its memory allocations. Memory pages are, without explicit requests, only moved in extreme circumstances. A thread can be assigned to another CPU as the result of rebalancing the CPU loads. Information about the current CPU and node might therefore be short-lived. The scheduler will try to keep the thread on the same CPU, and possibly even on the same core, to minimize performance losses due to cold caches. This means it is useful to look at the current CPU and node information; one only must avoid assuming the association will not change.

libNUMA provides two interfaces to query the node information for a given virtual address space range:

#include <libNUMA.h>

int NUMA_mem_get_node_idx(void *addr);
int NUMA_mem_get_node_mask(void *addr,
                           size_t size,
                           size_t __destsize,
                           memnode_set_t *dest);

NUMA_mem_get_node_mask sets in dest the bits for all memory nodes on which the pages in the range [addr, addr+size) are (or would be) allocated, according to the governing policy. NUMA_mem_get_node only looks at the address addr and returns the index of the memory node on which this address is (or would be) allocated. These interfaces are simpler to use than get_mempolicy and probably should be preferred.

The CPU currently used by a thread can be queried using sched_getcpu (see Section 6.4.3). Using this information, a program can determine the memory node(s) which are local to the CPU using the NUMA_cpu_to_memnode interface from libNUMA:

#include <libNUMA.h>

int NUMA_cpu_to_memnode(size_t cpusetsize,
                        const cpu_set_t *cpuset,
                        size_t memnodesize,
                        memnode_set_t *
                        memnodeset);

A call to this function will set (in the memory node set pointed to by the fourth parameter) all the bits corresponding to memory nodes which are local to any of the CPUs in the set pointed to by the second parameter. Just like CPU information itself, this information is only correct until the configuration of the machine changes (for instance, CPUs get removed and added).

The bits in the memnode_set_t objects can be used in calls to the low-level functions like get_mempolicy. It is more convenient to use the other functions in libNUMA. The reverse mapping is available through:

#include <libNUMA.h>

int NUMA_memnode_to_cpu(size_t memnodesize,
                        const memnode_set_t *
                        memnodeset,
                        size_t cpusetsize,
                        cpu_set_t *cpuset);

The bits set in the resulting cpuset are those of the CPUs local to any of the memory nodes with corresponding bits set in memnodeset. For both interfaces, the programmer has to be aware that the information can change over time (especially with CPU hot-plugging). In many situations, a single bit is set in the input bit set, but it is also meaningful, for instance, to pass the entire set of CPUs retrieved by a call to sched_getaffinity to NUMA_cpu_to_memnode to determine which are the memory nodes the thread ever can have direct access to.

6.5.6 CPU and Node Sets

Adjusting code for SMP and NUMA environments by changing the code to use the interfaces described so far might be prohibitively expensive (or impossible) if the sources are not available. Additionally, the system administrator might want to impose restrictions on the resources a user and/or process can use. For these situations the Linux kernel supports so-called CPU sets. The name is a bit misleading since memory nodes are also covered. They also have nothing to do with the cpu_set_t data type.

The interface to CPU sets is, at the moment, a special filesystem. It is usually not mounted (so far at least). This can be changed with

     mount -t cpuset none /dev/cpuset

Of course the mount point /dev/cpuset must exist. The content of this directory is a description of the default (root) CPU set. It comprises initially all CPUs and all memory nodes. The cpus file in that directory shows the CPUs in the CPU set, the mems file the memory nodes, the tasks file the processes.

To create a new CPU set one simply creates a new directory somewhere in the hierarchy. The new CPU set will inherit all settings from the parent. Then the CPUs and memory nodes for new CPU set can be changed by writing the new values into the cpus and mems pseudo files in the new directory.

If a process belongs to a CPU set, the settings for the CPUs and memory nodes are used as masks for the affinity and memory policy bitmasks. That means the program cannot select any CPU in the affinity mask which is not in the cpus file for the CPU set the process is using (i.e., where it is listed in the tasks file). Similarly for the node masks for the memory policy and the mems file.

The program will not experience any errors unless the bitmasks are empty after the masking, so CPU sets are an almost-invisible means to control program execution. This method is especially efficient on large machines with lots of CPUs and/or memory nodes. Moving a process into a new CPU set is as simple as writing the process ID into the tasks file of the appropriate CPU set.

The directories for the CPU sets contain a number of other files which can be used to specify details like behavior under memory pressure and exclusive access to CPUs and memory nodes. The interested reader is referred to the file Documentation/cpusets.txt in the kernel source tree.

6.5.7 Explicit NUMA Optimizations

All the local memory and affinity rules cannot help out if all threads on all the nodes need access to the same memory regions. It is, of course, possible to simply restrict the number of threads to a number supportable by the processors which are directly connected to the memory node. This does not take advantage of SMP NUMA machines, though, and is therefore not a real option.

If the data in question is read-only there is a simple solution: replication. Each node can get its own copy of the data so that no inter-node accesses are necessary. Code to do this can look like this:

void *local_data(void) {
  static void *data[NNODES];
  int node =
    NUMA_memnode_self_current_idx();
  if (node == -1)
    /* Cannot get node, pick one.  */
    node = 0;
  if (data[node] == NULL)
    data[node] = allocate_data();
  return data[node];
}
void worker(void) {
  void *data = local_data();
  for (...)
    compute using data
}

In this code the function worker prepares by getting a pointer to the local copy of the data by a call to local_data. Then it proceeds with the loop, which uses this pointer. The local_data function keeps a list of the already allocated copies of the data around. Each system has a limited number of memory nodes, so the size of the array with the pointers to the per-node memory copies is limited in size. The NUMA_memnode_system_count function from libNUMA returns this number. If the pointer for the current node, as determined by the NUMA_memnode_self_current_idx call, is not yet known a new copy is allocated.

It is important to realize that nothing terrible happens if the threads get scheduled onto another CPU connected to a different memory node after the sched_getcpu system call. It just means that the accesses using the data variable in worker access memory on another memory node. This slows the program down until data is computed anew, but that is all. The kernel will always avoid gratuitous rebalancing of the per-CPU run queues. If such a transfer happens it is usually for a good reason and will not happen again for the near future.

Things are more complicated when the memory area in question is writable. Simple duplication will not work in this case. Depending on the exact situation there might a number of possible solutions.

For instance, if the writable memory region is used to accumulate results, it might be possible to first create a separate region for each memory node in which the results are accumulated. Then, when this work is done, all the per-node memory regions are combined to get the total result. This technique can work even if the work never really stops, but intermediate results are needed. The requirement for this approach is that the accumulation of a result is stateless, i.e., it does not depend on the previously collected results.

It will always be better, though, to have direct access to the writable memory region. If the number of accesses to the memory region is substantial, it might be a good idea to force the kernel to migrate the memory pages in question to the local node. If the number of accesses is really high, and the writes on different nodes do not happen concurrently, this could help. But be aware that the kernel cannot perform miracles: the page migration is a copy operation and as such it is not cheap. This cost has to be amortized.

6.5.8 Utilizing All Bandwidth

The numbers in Figure 5.4 show that access to remote memory when the caches are ineffective is not measurably slower than access to local memory. This means a program could possibly save bandwidth to the local memory by writing data it does not have to read again into memory attached to another processor. The bandwidth of the connection to the DRAM modules and the bandwidth of the interconnects are mostly independent, so parallel use could improve overall performance.

Whether this is really possible depends on many factors. One really has to be sure that caches are ineffective since otherwise the slowdown related to remote accesses is measurable. Another big problem is whether the remote node has any needs for its own memory bandwidth. This possibility must be examined in detail before the approach is taken. In theory, using all the bandwidth available to a processor can have positive effects. A family 10h Opteron processor can be directly connected to up to four other processors. Utilizing all that additional bandwidth, perhaps coupled with appropriate prefetches (especially prefetchw) could lead to improvements if the rest of the system plays along.

Comments (13 posted)

OpenID 2.0 closing in on acceptance

A very common complaint about using the web today is the proliferation of user IDs and associated passwords, people would much rather see a "single sign-on" (SSO) system. There are many proposed solutions for SSO, but OpenID is one of the simpler and most widespread; it also has the advantage of not being tied to a specific vendor, with open specifications and freely available libraries. Currently the OpenID 2.0 specification is closing in on acceptance with just the "intellectual property" rights (IPR) policy standing in its way.

One of the nicest features of OpenID is its user-centric nature – users can have as much control as they want over their identity. Unlike other solutions, there is no central authority required to store identities or process authentication requests. Users can run their own server or pick one of the available providers to get a free ID. An overview of OpenID appeared on this page last year.

OpenID 2.0 adds a number of features that will be quite useful for both users and websites that implement OpenID ("relying parties" or RPs in OpenID terminology). The Attribute Exchange extension is one that could solve a common problem by allowing users to associate additional information with their identity, sharing and, more importantly, updating that information at multiple sites more or less transparently. If a user moves or changes email addresses, that information could be updated at multiple sites.

OpenID 2.0 also provides support for additional extensions to the protocol, allowing functionality beyond what is currently envisioned, while adding namespaces to avoid name collisions between those extensions. Directed identities takes the delegation idea from OpenID 1.1 one step further, allowing users to specify the URL of the OpenID provider (OP), rather than their user-specific URL, as their ID. The OP can then resolve the user's URL through some means (such as a login screen) and provide that back to the RP. As James Henstridge points out in his weblog, this would allow an OP like AOL to allow "aol.com" as the OpenID for millions of users. Perhaps not the OpenID of choice for everyone, but it does offer a pretty simple ID to remember.

There are other improvements included in OpenID 2.0, including interfacing with other identity solutions, security improvements, and allowing for arbitrary length of protocol messages, rather than being limited by the URL-length limits of browsers. There are freely available implementations of OpenID 2.0 for PHP, Python, and Java (at least), all of which interoperate.

A recent discussion on the specs mailing list would appear to pave the way for the most recent draft (Draft 12) to gain acceptance. According to David Recordon, there are no technical barriers to acceptance:

The only barrier is a legal one, the IPR policy needs to be agreed upon, then each contributor needs to sign a "non-assertion statement" that promises not to sue any implementer of the standard for patent infringement. This allows anyone to implement the standard without fear of lawsuits or having to pay royalties, at least to the companies that have signed. Other companies or, worse yet, patent trolls are, of course, free to sue.

OpenID still suffers from a lack of sites that accept it, though many big players are flirting with it: AOL and Microsoft for example. AOL is an OpenID provider, all AOL screen names have an OpenID if they wish to use it, but you cannot log in to AOL using it. Also, there is rampant speculation that Google's recently announced OpenSocial API will provide OpenID support eventually. So far, though, other than the LiveJournal blogging sites (where OpenID originated) and Digg, there just aren't that many sites where OpenID can be used. Perhaps finalizing and accepting the 2.0 specification will turn the tide.

Comments (8 posted)

cups: buffer overflow

Package(s):	cups	CVE #(s):	CVE-2007-4351
Created:	October 31, 2007	Updated:	November 19, 2007
Description:	The CUPS code charged with dealing with TCP-based Internet Printer Protocol connections suffers from a buffer overflow which could possibly be exploitable remotely. The vulnerability is only present if remote hosts are allowed to connect to the IPP port, which is usually not the default setting.
Alerts:	Debian DSA-1407-1 cupsys 2007-11-18 Gentoo 200711-16 cups 2007-11-12 Mandriva MDKSA-2007:204-1 cups 2007-11-12 Fedora FEDORA-2007-2982 cups 2007-11-08 Ubuntu USN-539-1 cupsys 2007-11-06 Fedora FEDORA-2007-740 cups 2007-11-05 Slackware SSA:2007-305-01 cups 2007-11-02 Fedora FEDORA-2007-2715 cups 2007-11-01 Mandriva MDKSA-2007:204 cups 2007-11-01 rPath rPSA-2007-0227-1 cups 2007-10-31 SuSE SUSE-SA:2007:058 cups 2007-10-31 Red Hat RHSA-2007:1020-01 CUPS 2007-10-31

Comments (none posted)

mldonkey: privilege escalation

Package(s):	mldonkey	CVE #(s):
Created:	October 25, 2007	Updated:	October 31, 2007
Description:	The MLDonkey peer-to-peer filesharing client can be used to add a user to the system with a valid login shell and no password. This can be used for the escalation of privilege.
Alerts:	Gentoo 200710-25 mldonkey 2007-10-24

Comments (none posted)

python: integer overflows

Package(s):	python	CVE #(s):	CVE-2007-4965
Created:	October 30, 2007	Updated:	July 30, 2009
Description:	Multiple integer overflows in the imageop module in Python 2.5.1 and earlier allow context-dependent attackers to cause a denial of service (application crash) and possibly obtain sensitive information (memory contents) via crafted arguments to (1) the tovideo method, and unspecified other vectors related to (2) imageop.c, (3) rbgimgmodule.c, and other files, which trigger heap-based buffer overflows.
Alerts:	CentOS CESA-2009:1176 python 2009-07-29 Red Hat RHSA-2009:1176-01 python 2009-07-27 Mandriva MDVSA-2009:036 python 2009-02-12 Mandriva MDVSA-2008:164 python 2008-08-07 Mandriva MDVSA-2008:163 python 2007-08-07 Debian DSA-1620-1 python2.5 2008-07-27 Gentoo 200807-01 python 2008-07-01 Debian DSA-1551-1 python2.4 2008-04-19 Ubuntu USN-585-1 python2.4/2.5 2008-03-11 Foresight FLEA-2008-0002-1 python 2008-02-11 SuSE SUSE-SR:2008:003 java, nss_ldap, cairo, geronimo, moodle, SDL_image, python, mysql, nx, xemacs 2008-02-07 Mandriva MDVSA-2008:013 python 2007-01-14 Mandriva MDVSA-2008:012 python 2008-01-14 Red Hat RHSA-2007:1076-02 python 2007-12-10 rPath rPSA-2007-0254-1 idle python 2007-11-30 Gentoo 200711-07 python 2007-11-07 Fedora FEDORA-2007-2663 python 2007-10-29

Comments (none posted)

subversion: possible information leak

Package(s):	subversion	CVE #(s):	CVE-2007-2448
Created:	October 30, 2007	Updated:	February 1, 2011
Description:	Subversion 1.4.3 and earlier does not properly implement the "partial access" privilege for users who have access to changed paths but not copied paths, which allows remote authenticated users to obtain sensitive information (revision properties) via svn (1) propget, (2) proplist, or (3) propedit.
Alerts:	Ubuntu USN-1053-1 subversion 2011-02-01 rPath rPSA-2007-0264-1 subversion 2007-12-12 Fedora FEDORA-2007-2635 subversion 2007-10-29

Comments (none posted)

xen-utils: insecure temp files

Package(s):	xen-utils	CVE #(s):	CVE-2007-3919
Created:	October 25, 2007	Updated:	May 16, 2008
Description:	The xen-utils collection of XEN administrative tools uses temporary files insecurely. Local users can use this to truncate arbitrary files.
Alerts:	CentOS CESA-2008:0194 xen 2008-05-16 Red Hat RHSA-2008:0194-01 xen 2008-05-13 Fedora FEDORA-2007-737 xen 2007-11-05 Debian DSA-1395-1 xen-utils 2007-10-25

Comments (none posted)

Kernel release status

The current 2.6 prepatch remains 2.6.24-rc1. Fixes have been flowing into the mainline repository (along with a Japanese translation of the SubmittingPatches document), but -rc2 has not been released as of this writing.

For older kernels: the 2.6.22.11 stable update is in the review process now; it should be released sometime on or after November 2. It contains 26 patches addressing a number of problems. This is likely to be the last update to 2.6.22.

2.6.16.56-rc2 was released on October 29; it adds a handful of fixes.

Comments (none posted)

Quotes of the week

-- Casey Schaufler

-- Andrew Morton

-- Linus Torvalds

Comments (2 posted)

The chained scatterlist API

When asked which of the changes in 2.6.24 was most likely to create problems, an informed observer might well point at the i386/x86_64 merger. As it happens, that large patch set has gone in with relatively few hitches, but a rather smaller change has created quite a bit of fallout. The change in question is the updated API for the management of scatterlists, which are used in scatter/gather I/O. This work broke a number of in-tree drivers, so it seems likely to affect a lot of out-of-tree code as well.

Scatter/gather I/O allows the system to perform DMA I/O operations on buffers which are scattered throughout physical memory. Consider, for example, the case of a large (multi-page) buffer created in user space. The application sees a continuous range of virtual addresses, but the physical pages behind those addresses will almost certainly not be adjacent to each other. If that buffer is to be written to a device in a single I/O operation, one of two things must be done: (1) the data must be copied into a physically-contiguous buffer, or (2) the device must be able to work with a list of physical addresses and lengths, grabbing the right amount of data from each segment. Scatter/gather I/O, by eliminating the need to copy data into contiguous buffers, can greatly increase the efficiency of I/O operations while simultaneously getting around the problem that the creation of large, physically-contiguous buffers can be problematic in the first place.

Within the kernel, a buffer to be used in a scatter/gather DMA operation is represented by an array of one or more scatterlist structures, defined in <linux/scatterlist.h>. This array has traditionally been constrained to fit within a single page, which imposes a maximum length on scatter/gather operations. That limit has proved to be a bottleneck on high-end systems, which could otherwise benefit from transferring very large buffers (usually to and from disk devices). As a result, there has been a search for ways to get around that limit; the large block size patches which occasionally surface on the mailing lists are one approach. But the solution which has made it into the 2.6.24 kernel is to remove the limit on the length of scatter/gather lists by allowing them to be chained.

A chained scatter/gather list can be made up of more than one page, and those pages, too, are likely to be scattered throughout physical memory. When this chaining is done, a couple of low-order bits in the buffer pointer are used to mark chain entries and the end of the list. This usage is not something which driver code needs to worry about, but the existence of special bits and chain pointers forces some changes to how drivers work with scatterlists.

Drivers which do not perform chaining will allocate their scatterlist arrays in the usual way - usually through a call to kcalloc() or some such. Prior to 2.6.23, there was no initialization step required, beyond, perhaps, zeroing the entire array. That has changed, however; drivers should now initialize a scatterlist array with:

    void sg_init_table(struct scatterlist *sg, unsigned int nents);

Here, sg points to the allocated array, and nents is the number of allocated scatter/gather entries.

As before, a driver should loop through the segments of the buffer, setting one scatterlist entry for each. It is no longer possible to set the page pointer directly, however: that pointer does not exist in 2.6.24. Instead, the usual way to set a scatterlist entry will be with one of:

    void sg_set_page(struct scatterlist *sg, struct page *page,
		     unsigned int len, unsigned int offset);

    void sg_set_buf(struct scatterlist *sg, const void *buf,
	      	    unsigned int buflen);

2.6.24 scatterlists also require that the end of the list be explicitly marked. This marking is performed when sg_init_table() is called, so drivers will not normally have to mark the end explicitly. Should the I/O operation not use all of the entries which were allocated in the list, though, the driver should mark the final segment with:

    void sg_mark_end(struct scatterlist *sg, unsigned int nents);

Where nents is the number of valid entries in the scatterlist.

After the scatterlist has been mapped (with a function like dma_map_sg()), the driver will need to program the resulting DMA addresses into the hardware. The old approach of just stepping through the array will no longer work; instead, a driver should move on to the next entry in a scatterlist with:

    struct scatterlist *sg_next(struct scatterlist *sg);

The return value will be the next entry to process - or NULL if the end of the list has been reached. There is also a for_each_sg() macro which can be used to iterate through an entire scatterlist; it will typically be used in code which looks like:

    int i;
    struct scatterlist *list, *sgentry;

    /* Fill in list and pass it to dma_map_sg().  Then... */
    for_each_sg(i, list, sgentry, nentries) {
	program_hw(device, sg_dma_address(sgentry), sg_dma_len(sgentry));
    }

Drivers which wish to take advantage of the chaining feature must do just a little more work. Each piece of the scatterlist must be allocated independently, then those pieces must be chained together with:

    void sg_chain(struct scatterlist *prv, unsigned int prv_nents,
		  struct scatterlist *next);

This call turns the scatterlist entry prv[nents] into a chain link to next. If the chaining is done while the list is being filled, prv should have no more than prv_nents-1 segments stored into it. Alternatively, a driver can chain together the pieces of the list ahead of time (remembering to allocate one entry for each chain link), then use sg_next() to fill the list without the need to worry about where the chain links are.

As of this writing, this API is still evolving in response to issues which have come up with in-tree drivers. It seems unlikely that any more substantial changes will be made before the 2.6.24 release, but surprises are always possible.

Comments (none posted)

Fixing CAP_SETPCAP

Linux capabilities have been around for almost ten years now – they were originally merged into a 2.1 kernel – but they haven't gotten a lot of use in that time. One pretty basic missing feature, support for associating capabilities with files, has been merged for 2.6.24. This allows a longstanding hack, which redefines the proper usage of CAP_SETPCAP, to be fixed; this too has been merged into 2.6.24.

A bit of review is probably in order. Capabilities are a way to separate individual privileges that are normally all granted to the root user. There are currently 31 different capabilities defined (in linux/capability.h), but there are efforts underway to allow for expansion. The idea is that a program should be able to set the system time, for example, without needing the entire set of privileges that come with a setuid(0) program.

Capabilities originally came from a proposed POSIX standard that was eventually not adopted, but, in the meantime, got included into Linux. The feature has languished since, for a number of reasons, but perhaps the largest was that there was no way to associate executable programs with a set of capability bits. Now that capability bits can be stored in the extended attributes of files, the process can get the proper capabilities when the program is invoked. Standard UNIX permissions still apply – users can only execute programs they have an x bit for.

In order to use capabilities at all, prior to being able to store them with files, a method was needed to set the capabilities of a running process. The CAP_SETPCAP capability was co-opted for this purpose. A process with this capability, which, in practice, meant root processes could set the capabilities of another process. If that process was meant to be able to do the same – something that needs to be carefully considered – it could get the CAP_SETPCAP bit as well.

This could really only be used to add capabilities to long running processes that were not run as root (which has all of the capabilities), or to remove some capabilities from daemons run as root. Other schemes using setuid wrappers for utility programs that needed some privileges could also be imagined, but distributions or tools that use capabilities are not widespread.

CAP_SETPCAP was never meant to have this behavior, so the recent patch restores it its original meaning. As odd as it might seem at first, CAP_SETPCAP is only meant to allow changes to a process's own capabilities; in fact, with this patch applied, there is no way for a process to change a running process's capabilities. That is probably the biggest user-visible change.

Capabilities are not a single set of bits, but are instead, three sets of bits representing the effective, permitted, and inheritable capabilities of a process. Files, similarly, have three capability sets which are combined with those of the process executing the file using the "capability rules" (described in the patch and in an LWN article from a year ago) to determine the three sets for the process created.

For processes, the effective set contains those capabilities currently enabled – a process might drop some that it is allowed once it has performed the corresponding privileged operation – while the permitted set is a superset of the effective set, including all capabilities allowed to that process. The inheritable set are those that are passed on to a new program started by an exec() call, which is where the new CAP_SETPCAP comes into play; a process with this capability can change its inheritable set to include any capability, including those that are not in their permitted set.

This allows processes to bestow privileges that they do not possess upon their children, which provides for some interesting uses. It helps further partition privileges by not requiring a process to have a particular capability simply to pass it on to children. The example provided in the patch illustrates this nicely: the login program does not require many privileges, but through some policy mechanism (pam_cap for example) could allow certain users to have extra capabilities. Because the login process does not itself possess those extra capabilities, this could limit the damage an exploit of login could do.

It is unclear whether these recent additions to the capability feature set will result in more capability users. There is a lot of work in the kernel security space right now as kernel hackers and security folks try to come up with sensible security solutions for Linux. The complexity of SELinux, along with the fact that many administrators disable it rather than try to figure it out, seems to have the community casting about for other solutions. It is possible that capabilities might be a part of another solution, though its complexities are far from trivial. Though most of the major distributions have already made their security model choice, a capabilities-based distribution would be interesting to see; it might make a nice project for a smaller, up-and-coming, distribution to try.

Comments (10 posted)

Notes from a container

"Containers" are a form of lightweight virtualization as represented by projects like OpenVZ. While virtualization creates a new virtual machine upon which the guest system runs, containers implementations work by making walls around groups of processes. The result is that, while virtualized guests each run their own kernel (and can run different operating systems than the host), containerized systems all run on the host's kernel. So containers lack some of the flexibility of full virtualization, but they tend to be quite a bit more efficient.

As of 2.6.23, virtualization is quite well supported on Linux, at least for the x86 architecture. Containers lag a little behind, instead. It turns out that, in many ways, containers are harder to implement than virtualization is. A container implementation must wrap a namespace layer around every global resource found in the kernel, and there are a lot of these resources: processes, filesystems, devices, firewall rules, even the system time. Finding ways to wrap all of these resources in a way which satisfies the needs of the various container projects out there, and which also does not irritate kernel developers who may have no interest in containers, has been a bit of a challenge.

Full container support will get quite a bit closer once the 2.6.24 kernel is released. The merger of a number of important patches in this development cycle fills in some important pieces, though a certain amount of work remains to be done.

Once upon a time, there was a patch set called process containers. The containers subsystem allows an administrator (or administrative daemon) to group processes into hierarchies of containers; each hierarchy is managed by one or more "subsystems." The original "containers" name was considered to be too generic - this code is an important part of a container solution, but it's far from the whole thing. So containers have now been renamed "control groups" (or "cgroups") and merged for 2.6.24.

Control groups need not be used for containers; for example, the group scheduling feature (also merged for 2.6.24) uses control groups to set the scheduling boundaries. But it makes sense to pair control groups with the management of the various namespaces and resource management in general to create a framework for a containers implementation.

The management of control groups is straightforward. The system administrator starts by mounting a special cgroup filesystem, associating the subsystems of interest with the filesystem at mount time. There can be more than one such filesystem mounted, as long as each subsystem appears on at most one control group. So the administrator could create one cgroup filesystem to manage scheduling and a completely different one to associate processes with namespaces.

Once the filesystem is mounted, specific groups are created by making directories within the cgroup filesystem. Putting a process into a control group is a simple matter of writing its process ID into the tasks virtual file in the cgroup directory. Processes can be moved between control groups at will.

The concept of a process ID has gotten more complicated, though, since the PID namespace code was also merged. A PID namespace is a view of the processes on the system. On a "normal" Linux system, there is only the global PID namespace, and all processes can be found there. On a system with PID namespaces, different processes can have very different views of what is running on the system. When a new PID namespace is created, the only visible process is the one which created that namespace; it becomes, in essence, the init process for that namespace. Any descendants of that process will be visible in the new namespace, but they will never be able to see anything running outside of that namespace.

Virtualizing process IDs in this way complicates a number of things. A process which creates a namespace remains visible to its parent in the old namespace - and it may not have the same process ID in both namespaces. So processes can have more than one ID, and the same process ID may be found referring to different processes in different namespaces. For example, it is fairly common in containers implementations to have the per-namespace init process have ID 1 in its namespace.

[PULL QUOTE: What all of this means is that process IDs only make sense when placed into a specific context. That, in turn, sets a trap for any kernel code which works with process IDs. END QUOTE] What all of this means is that process IDs only make sense when placed into a specific context. That, in turn, sets a trap for any kernel code which works with process IDs; any such code must take care to maintain the association between a process ID and the namespace in which it is defined. To make life easier (and safer), the containers developers have been working for some time to eliminate (to the greatest extent possible) use of process IDs within the kernel itself. Kernel code should use task_struct pointers (which are always unambiguous) to refer to specific processes; a process ID, instead, has become a cookie for communication with user space, and not much more.

This job of cleaning up PID use is not complete at this point. In fact, the process ID namespace work has a great many loose ends in general, to the point that some of the developers do not think that it is really ready to be used yet. In particular, there is concern that some of the management APIs could change, breaking code which is written for the 2.6.24 API. Adding new user-space APIs is always problematic in this regard: getting an API right is hard, and getting it right the first time is even harder. But user-space APIs are supposed to stay constant once they are merged; there is no provision for any sort of stabilization period where things can change. For PID namespaces, what's likely to happen is that the feature will be marked "experimental" in the hope that nobody will use it in its 2.6.24 form.

Also merged for 2.6.24 is the network namespace patch. The idea behind this code is to allow processes within each namespace to have an entirely different view of the network stack. That includes the available interfaces, routing tables, firewall rules, and so on. These patches are in a relatively early state; they add the infrastructure to track different namespaces, but not a whole lot more. Quite a few internal networking APIs have been changed to take a namespace parameter, but, in most cases, the code simply fails any operation which is attempted in anything other than the default, root namespace. There is a new "veth" virtual network device which can be used to create tunnels between namespaces.

The PID and network namespace patches have added a couple of lines to <linux/sched.h>:

    #define CLONE_NEWPID	0x20000000	/* New pid namespace */
    #define CLONE_NEWNET	0x40000000	/* New network namespace */

These entries highlight an interesting problem: the CLONE_ flags are passed to the kernel as a 32-bit value. As of this writing, there are only two bits left for new flags. So the containers developers are going to run out of flags; how they plan to deal with that problem is not clear at this point.

These developers are also working on the management of containers, and, in particular, how to move between them. One of the things likely to come out of that work in the near future is a proposal for a new system call:

    int hijack(unsigned long clone_flags, int which, int id);

This system call behaves much like clone() in that it creates a new process, but with an interesting twist. The new process created by clone() takes all of its resources - including namespaces - from the calling process; these resources will be copied or shared as directed by the clone_flags argument. A call to hijack(), instead, obtains all of those resources from the process whose ID is given in the id parameter. So it is possible to write a little program which forks via a hijack() call and runs a shell in the resulting child process; that shell will be running with all of the namespaces of the hijacked process.

To make life easier for people working with containers, the which parameter was added in recent versions of this API. If which is passed as 1, the call treats id as a process ID, as described above. A value of 2, instead, says that id is actually an open file descriptor for the tasks file in a cgroup control directory. In this case, hijack() finds the lead process for that control group and obtains resources from there.

This system call is new, and it has not seen a whole lot of review outside of the containers mailing list. So chances are that some changes will be requested once it becomes more widely visible; among other things, a name change might be called for. In general, there is a lot yet to be done with the containers code, but progress is visibly being made. There will come a point where the mainline kernel comes equipped with complete container capabilities.

Comments (11 posted)

Parting is such sweet sorrow - The Optimistic Contributor's review of Parted Magic

Like many in the IT profession, I have amassed an arsenal of hardware and software tools to assist in my daily duties. Rarely do I let the opportunity pass by to show off a newly found treasure to a friend or colleague. One of my recently discovered gold nuggets is the Parted Magic LiveCD/USB/PXE distribution. From the project's website the mission of the project is stated quite succinctly: "Parted Magic is a Linux LiveCD/USB/PXE with its elemental purpose being to partition hard drives."

From a base Linux OS created from scratch (okay, he has a few init scripts from Linux From Scratch), the project originator and main developer, Patrick Verner, has assembled a collection of tools and utilities that are brought together in a cohesive manner with a high level of polish.

After downloading and burning the tiny 37 megabyte ISO image (version 1.9 was released on 10/26) to a CD, I booted a test PC and was greeted with a boot menu that provides just about any option necessary to make the system work without having to memorize boot parameters or dig through the documentation. After a short time-out, the default settings take effect and a simple XFCE desktop appears.

Exploring the XFCE panel, I found a logically organized arrangement. The first launcher starts the GParted application, the primary tool included to partition hard drives. GParted is a graphical front-end to the GNU parted utility. It offers all of the features of the command line version of parted, but they are wrapped in an easy to use GUI.

Verner has actually extended this application with a few patches of his own, one of which being the addition of the ability to create HFS+ partitions for those wishing to prepare a hard drive for use with Mac OS X.

The second launcher is that of the Thunar file manager, a light-weight system for browsing disks that is the default with any standard XFCE desktop installation. It was impressive just how fast the file manager (and the rest of the desktop) responded even on my old test laptop. Verner has obviously made a wise selection to use XFCE as the default environment. Although aesthetically very pleasing, it is not very resource intensive, providing a nice balance between form and function.

The remaining launchers continue the logical progression previously set forth: one for a shell prompt, one for a tool for taking screenshots (which is very handy to have for making documentation), one for the other utilities present (such as the very useful TestDisk recovery program), and finally one for documentation. For a full list of all the programs included, please see here.

As you may have already surmised, I like this project. It has become my 'go to' tool when dealing with any hard drive related issue, whether trying to setup multiple operating systems on a PC, or trying to recover some pictures for a friend that he accidentally deleted. It would take a long time to delve into each feature, so I would recommend you check out the website for more details.

My optimism started to fade recently when Verner posted on his web site that, due to the overwhelming amount of time that the project was consuming, (by his own estimate he has invested over 1,000 hours into the project), and the general lack of support from the community (in the form of donations, patches, etc.) that version 1.9 of the project would be its last. I was chagrined to say the least, but could understand. After reading through the project's web forum it was obvious that Verner was growing weary of fielding support requests from people who had not read the documentation or seemed to be demanding help rather than asking for it.

Since I had not contributed to the effort in any way, I sent a small donation with a note of thanks to Verner. He replied and it led to me questioning him about details of the project via email. It made me think about Free Software and what it means.

There are any number of reasons why a free software project might fail. It seems that (at least in this instance) too many people think of free software as in free beer, not as in freedom. When I advocate the use of free software such as Linux, I always tend to think of the freedom to make changes, the freedom not to be locked in. What I forgot is the old adage that freedom is not free. Along with that freedom comes the responsibility of the community at large to do what they can to help.

This help can come in any form, whether it is writing documentation, helping to moderate a web forum, or just simply sending a thank you email to the developer(s). In Parted Magic's case, the primary developer is a family man with an unrelated day job. He had hoped to be able to work on the project full time if given enough support. Because of the low amount of involvement of the community, a unique LiveCD project is going to cease further development.

I can only wonder how many other projects in the free software world have met the same fate. What great application or idea is lying dormant in Google's cache or the Internet Archive? I know what you are thinking, if we are dealing with open source software, why doesn't someone else just pick up where the original developer left off? The simple answer may be that people with the time, skills or inclination to scratch the same itch that brought a project to fruition are few and far between. Quite frankly, why would someone want to, knowing that they might meet the same fate as Patrick Verner?

The power of the concept that makes free software great lies in one area alone, the community. If we truly believe in the principles we espouse, we must each do what we can to help the foster work the community puts out there. My thanks go out to Patrick Verner, and all the people that did help him (they are listed on his web site), as well as all the other free software developers out there. I will try my best to do my part for software freedom. I hope (always the optimist) that other people do the same.

Comments (5 posted)

Bits from the Security Team

Debian's security team has some noteworthy changes that Debian users should be aware of. Click below to see more about the Debian Security Tracker, embedded code copies, architecture versatility, a separate queue for unembargoed security problems, security mirrors, the request tracker, Debtags for the scope of security support, and more.

Full Story (comments: none)

Unofficial FAQ Updated for Fedora 7!

The Unofficial FAQ has been updated for Fedora 7. "I've overhauled all of the questions to be up-to-date with Fedora 7. I've also re-worked the yum configuration a neat way, so that every package is available to you from every repository, without any cross-repo conflicts!"

Full Story (comments: none)

Fedora Board Recap

Click below for a quick look at the October 30, 2007 meeting of the Fedora Board.

Full Story (comments: none)

Fedora is breaking X in rawhide

Regular users of Fedora Rawhide will have already noticed the disruption. "We're working on a lot of stuff that's going into upstream X really soon. It's going to be quite disruptive, many drivers will fail to launch, etc. Good stuff, but disruptive." Expect at least a month of pain here.

Full Story (comments: 1)

Cooker Wishlist is open

Mandriva has a wishlist page available on the wiki. Mandriva users are encouraged to enter any suggestions or wishes for Mandriva 2008.1.

Full Story (comments: none)

Hardy Heron open for uploads

Fast on the heels of the newly release Gutsy Gibbon, the Ubuntu project begins work on Hardy Heron. "The doors are now open for uploads to Hardy Heron, the next in the Ubuntu line of releases, due for release in the first half of 2008. We are ready and waiting for your contributions to what is certain to be our best release yet!"

Full Story (comments: 16)

Universe QA for Hardy

The Hardy Heron is the name of Ubuntu's next release and it's a Long Term Support (LTS) release. The Masters of the Universe (MOTU) are already at work making this a quality release that can be supported for the next five years. Click below for a lengthy email on the Quality Assurance work that is expected for Hardy. There is a lot to do and help is always appreciated.

Full Story (comments: 1)

Fedora Weekly News Issue 107

The latest Fedora Weekly News, number 107, is out. It has updates on the status of the Fedora 8 release, including blocker bugs and testing needed for the ALSA kernel. Lots of other information of interest to the Fedora community is also included, click below for the issue.

Full Story (comments: none)

full circle #6 is out

full circle, the Ubuntu community magazine, has released issue #6 with articles on upgrading Feisty to Gutsy, using Photoshop plugins in the Gimp, an interview with John Phillips about Open Font Library, and more.

Comments (none posted)

Ubuntu Weekly News: Issue #63

The Ubuntu Weekly Newsletter for October 27, 2007 covers new MOTU Team members and MOTU council changes, the release of Full Circle Magazine #6, the release of Launchpad 1.1.10, the Ubunteros Tribe on TribalWars, Ubuntu Forum News, and much more.

Full Story (comments: none)

DistroWatch Weekly, Issue 226

The DistroWatch Weekly for October 29, 2007 is out. "As the Ubuntu Developer Summit gets under way in Boston later today, it is clear that the project's recently released version 7.10 is a resounding success - certainly one of the most user-friendly desktop Linux distributions ever delivered to the computing world. We take a look at both Ubuntu and Kubuntu 7.10 and look forward to the project's next release - "Hardy Heron". In other news, FreeBSD gears up for a flurry of development releases prior to the 6.3 and 7.0 versions, Mandriva starts collecting ideas for 2008.1, Russia's ALT Linux revels in the success of Linux on the domestic market, and the popular GNU Image Manipulation Program hits version 2.4. Finally, don't miss the statistical piece analysing the DistroWatch web logs, with a brief note explaining why these data aren't as useful in measuring distro market share as some readers might believe."

Comments (none posted)

Interview with Dimitris Glezos on Transifex

Jonathan Roberts interviews Fedora developer Dimitris Glezos. "Free software is used all around the world, and as such it needs to be translated to all kinds of different locales. Fedora has a very active translation community, and they decided it was time that some better tools existed for contributing translations and integrating with upstream. To find out more about this, I talked with Dimitris Glezos, discussing the new Transifex project, what it was like to work on a Google Summer of Code Project, and much more..."

Comments (1 posted)

PulseAudio in Fedora 8

The Fedora project, claiming to have the first distribution to use PulseAudio by default, has posted an interview with Lennart Poettering. "A lot of things have changed. For example, you can now change the volume of every playback stream seperately. Then, we have better hotplug support: Just plug in your USB speaker and it will appear in your mixer... You can move streams during playback between output devices. With a single click in our 'paprefs' tool you can aggregate all local audio devices into a virtual one, which distributes audio to all outputs, and deals with the small frequency deviations in the sound card's quartzes -- and that code even deals with hotplugging/unplugging."

Comments (18 posted)

Mini Linux: DSL 4.0 is ready (heise online)

heise online covers the release of Damn Small Linux 4.0. "The developers of Damn Small Linux (DSL) have released Version 4.0 of the mini distribution with a graphical user interface. In addition to numerous bugfixes DSL 4.0 comes with a new tool for configuring networks, for printing and new software. In the current version the displaying of icons on the desktop is the responsibility of the Desktop File Manager (dfm); thus the graphic emelfm tool replaces Midnight Commander as file manager. With the help of sudo normal users can now also make use of WLAN cards with Prism2 chipsets. The developers have updated the kernel to Version 2.4.31."

Comments (none posted)

Review: NimbleX 2007 (Raiden's Realm)

Raiden's Realm has a review of NimbleX 2007. "NimbleX is a Linux distribution built on the idea that "fast is best". It comes complete with a lot of great tools to help you do a wide variety of tasks. Even though it's setup as a desktop distribution, the uses for NimbleX are quite extensive. It is designed to run from a CD, a USB pen drive or even from the network rather than a hard drive. Not many distributions combine such a selection of boot methods. Most use one or the other, but never all three in combination."

Comments (none posted)

Firefox Extension Issues

The Firefox 2 web browser is undoubtedly one of the most important applications running on the Linux desktop. Your author has been running Firefox for many years now. It is generally user friendly, the features that it has are useful, normally it doesn't get in the way of the user and crashes are rare. The Linux desktop would clearly not be the same without it.

One exception to this generally happy situation involves the handling of non-standard audio and video formats. Take, for example, the extremely common mp3 audio format. For reference, we'll be working with the Firefox version 2.0.0.8 on Ubuntu 7.10, both current releases. Firefox on Ubuntu is set up to use Totem, a GNOME movie player for playing mp3 files. Unfortunately, across quite a few releases of Ubuntu, your author has never had any luck getting Totem to play an mp3 file, clicking on an mp3 link causes Totem to fire up, then it simply freezes.

If you don't mind having a bit of closed-source software on your machine, RealPlayer 10 is a basic mp3 player with a simple GUI control panel that can be connected to Firefox. Here's how the installation was performed: the RealPlayer10GOLD.bin file was downloaded to a user directory, the downloaded file was executed, then the realplay command was executed manually in order to answer the installation and license questions. The libstdc++5 package had to be installed for realplay to run. Once realplay was initialized, things got more complicated. It was necessary to become root, visit the /usr/lib/firefox/plugins/ directory, remove the default libtotem files and restart Firefox. Downloading an mp3 file caused Firefox to display a popup window that prompted the user to select an appropriate player. The appropriate player was selected and things now worked. This process is easy the second time around, but a lot of digging through documentation was required initially.

Now, lets say you want to watch a new Hot Tuna video on You Tube. This case is a bit easier than setting Firefox up to play mp3 files. You Tube directed the browser to the Adobe Flash Player Download Center. The software was downloaded, unzipped and extracted with tar. The flashplayer-installer command was executed and it put a copy of libflashplayer.so in the ~/.mozilla/plugins directory, the plugins directory may require manual creation. Firefox was restarted and Hot Tuna played.

Another example of a common browser plugin is Java. It can be interesting to look at weather radar on the US NEXRAD network. If you click on the Loop buttons, Firefox will tell you that it needs to have Java installed. Unlike older versions, this version of Firefox/Ubuntu brought up a menu for choosing Sun's Java or GCJ. GCJ was chosen and seemed to install correctly, but was not able to display the radar movie. Once installed, removal of the faulty GCJ became a mystery. Installing the Sun Java manually seemed to overwrite the correct links, although the GCJ files are still sitting on the system in some unknown location, taking up disk space. The new magic only seems to work the first time Java needs to be installed.

The Java software was found on the Sun Microsystems Download Center for Java(TM) SE Runtime Environment 6 Update 3. Java was downloaded and the jre-6u3-linux-i586.bin file was executed. The installation/license questions had to be answered and the software was installed. Again, it was necessary to go to the ~/.mozilla/plugins directory and make a symbolic link back to the installed jre/jre1.6.0_03/plugin/i386/ns7/libjavaplugin_oji.so file. Not something your grandmother would want to do. Firefox was restarted and the radar movies work.

These examples may not be the most optimal solutions, but they were effective for achieving the desired results. To get the above three plugins running, it was necessary to modify either the system-wide or user-specific plugin directories. In one case, symbolic links were used to point to the installed libraries, in another case the library was copied directly. There does not seem to be any kind of standard technique in use.

Firefox has an internal about:plugins URL to display the plugin list. On one test machine, the plugin list was missing any entry for realplayer, but the player was installed and functioning. Unlike the about:config URL, there is no way to modify anything shown in about:plugins.

It seems like the adding of plugins should be possible using the Firefox menus. Clicking on Edit->Preferences->Content->File Types [Manage] brings up the Download Actions window, but that window seems to be crippled. There is no "Add" button, only a "Change Action" button that works on a limited number of pre-defined file extensions. There is no MP3 or JAVA extension to be found. Again, the list of plugins does not show everything installed.

Some of the plugin confusion is likely the result of different methods used by the various plugin software writers. However, that is likely caused by having too many ways to do one thing. This section of Firefox really looks like it could use a code review. Some work on simplifying the interface and the addition of some basic features would go a long way toward improving the end user experience. Managing plugins under Firefox really should be a lot easier to do.

Comments (24 posted)

PostgreSQL 8.3 Beta 2 released

Version 8.3 Beta 2 of the PostgreSQL DBMS has been announced. "Our first two weeks of testing were extremely fruitful, finding several bugs in Beta 1, which we've now fixed and are ready for you to re-test version 8.3. If you weren't able to get Beta 1 working on your system, it should be working now ... so try it out and tell us what you find!"

Full Story (comments: none)

SQuirreL SQL Client version 2.6.1 released (SourceForge)

Version 2.6.1 of SQuirreL SQL Client is available with several important bug fixes. "SQuirreL SQL Client is a graphical SQL client written in Java that will allow you to view the structure of a JDBC compliant database, browse the data in tables, issue SQL commands etc."

Comments (none posted)

Setting Up Master-Master Replication With MySQL 5 On Debian Etch (HowtoForge)

HowtoForge presents a tutorial on MySQL 5 Master-Master replication. "Since version 5, MySQL comes with built-in support for master-master replication, solving the problem that can happen with self-generated keys. In former MySQL versions, the problem with master-master replication was that conflicts arose immediately if node A and node B both inserted an auto-incrementing key on the same table. The advantages of master-master replication over the traditional master-slave replication are that you don't have to modify your applications to make write accesses only to the master, and that it is easier to provide high-availability because if the master fails, you still have the other master."

Comments (none posted)

Postgres Weekly News

The October 28, 2007 edition of the Postgres Weekly News is online with the latest PostgreSQL DBMS articles and resources.

Full Story (comments: none)

Django security fix released

A security fix release of the Django web platform has been announced. "Today we're releasing a fix for a security vulnerability discovered in Django's internationalization framework. The complete details are below, but the executive summary is that you should updated to a fixed version of Django immediately."

Comments (none posted)

opentaps 1.0.0-Preview6 released (SourceForge)

Version 1.0.0-Preview6 of opentaps has been announced. Opentaps is an: "ERP and CRM suite, including eCommerce, inventory, warehouse, order, customer management, general ledger, MRP, POS. Database independent service-oriented architecture (SOA). The opentaps Open Source ERP + CRM application suite released version 1.0.0 Preview 6 today. There have been over 500 commits since the release of Preview 6 at the end of September."

Comments (none posted)

PLplot release 5.8.0-RC1

Release 5.8.0-RC1 of PLplot, a scientific plotting library, has been announced. "This is a release candidate 1 for a stable release of PLplot. It represents the ongoing efforts of the community to improve the PLplot plotting package. Development releases in the 5.9.x series will be available every few months. The next full release will be 5.10.0."

Comments (none posted)

GNOME Software Announcements

The following new GNOME software has been announced this week:

• Accerciser 1.0.1 (first development release)
• Anjuta DevStudio 2.2.2 (bug fixes)
• cairo snapshot 1.5.2 (new features and bug fixes)
• Empathy 0.21.1 (new features and bug fixes)
• Eye of GNOME 2.20.2 (bug fixes and translation work)
• Eye of GNOME 2.21.1 (new features, bug fixes and translation work)
• gcalctool 5.21.1 (new features, bug fixes and translation work)
• Gimmie 0.2.8 (new features and bug fixes)
• gnome-games 2.21.1 (new features, bug fixes and translation work)
• gnome-phone-manager 0.30 (new features and bug fixes)
• Gnome Subtitles 0.7 (new features, bug fixes and translation work)
• Gnumeric 1.7.13 (new features and bug fixes)
• GtkIEEmbed 0.9.0 (initial release)
• Libgda/Libgnomedb 3.1.2 (new features, bug fixes and translation work)
• libgnomekbd 2.21.1 (translation work)
• Mergeant 0.67 (new features, bug fixes and translation work)
• Orca 2.21.1 (bug fixes and translation work)
• Pango-1.19.0 (new features and bug fixes)
• Strongwind 0.9 (unspecified)
• Tomboy 0.9.0 (new features and bug fixes)

You can find more new GNOME software releases at gnomefiles.org.

Comments (none posted)

KDE 4.0 Beta 4 and Development Platform RC1 Released

The KDE project has announced the releases of the fourth KDE 4.0 beta and the first development platform release candidate. The project is strongly interested in more feedback from testers so that the final 4.0 release can be as solid as possible.

Full Story (comments: 5)

KDE Commit-Digest for 28th October 2007 (KDE.News)

The October 28, 2007 edition of the KDE Commit-Digest has been announced. The content summary says: "Further XMP tag support in Digikam. Beginnings of a Plasma lock/logout applet and a weather applet, to display data from the existing weather data engine. Continued work on the new Plasma-based KNewsTicker applet. Continued work and development ideas in Parley. More various developments and optimisations in KHTML. Jamendo album download support in Amarok 2.0..."

Comments (none posted)

KDE Software Announcements

The following new KDE software has been announced this week:

• HypnoQuit 1.0 (initial release)
• K Raid Monitor 0.1 (initial release)
• KBibTeX 0.2 (new features and bug fixes)
• KDE Priority handler 1.0 RC2 (new features and bug fixes)
• KPowersave v0.7.3-unstable (new features, bug fixes and translation work)
• Krsync 0.1-Alpha (initial release)
• KSystemLog 0.3.2 (new features and bug fixes)
• mpdemu 0.1 (initial release)
• PokerTH 0.6-beta (new features)
• PySMSsend 1.25 (new features and bug fixes)
• Qosmic 1.1.1 (build fixes)
• Qosmic 1.1.2 (new feature and bug fix)
• Qtractor 0.0.7.752 (new features and bug fixes)
• Strigi 0.5.7 (new features and bug fixes)
• Zhu3D 3.1.0 (bug fixes)

You can find more new KDE software releases at kde-apps.org.

Comments (none posted)

Wine 0.9.48 released

Version 0.9.48 of Wine has been announced. "What's new in this release: Still more fixes for regression test failures. Much more complete cryptnet implementation. WIDL is now able to generate the oleaut32 proxy code. Lots of bug fixes."

Comments (none posted)

a2jmidid version 2 released

Version 2 of a2jmidid has been announced. "a2jmidid is daemon for exposing legacy ALSA sequencer applications in JACK MIDI system. It is based on jack-alsamidi-0.5 (jackd alsa seq midi backend) by Dmitry Baikov. The main purpose is to ease usage of legacy, not JACK-ified apps, in JACK MIDI enabled systems. New in this release is addition of configure script (autotools) that enables compatibility with different JACK MIDI API variants."

Full Story (comments: none)

horgand 1.14 released

Version 1.14 of horgand, an organ synthesizer, is out. Changes include: "Hundreds of new presets (Strings, Brass, ... ) availables on the website. Five new waveforms. All the waveforms are available for LFO and DSP effects. Back to ALSA if JACK is not available. Code improved for less CPU usage. Some minor bugs fixed."

Full Story (comments: none)

The OpenOffice.org Impress presenter screen

While OpenOffice.org's Impress is a reasonable presentation program, it lacks one often-requested feature: the provision of a separate screen for the presenter which would contain notes, an indication of what the next slide is, etc. So it is encouraging that the OOo developers have just announced that this feature is now in development; see this page for some description of how it is expected to work. "Implementation of the Presenter Screen extension has begun and there is an early extension that shows its basic capabilities. The look, layout, and detailed behaviour of the controls, however, are far from final. This is where you, dear reader, come into play. You can help us develop this extension by giving feedback when you try out the developer snapshot, by telling us what would help you most giving a presentation, or by joining us in implementing it..."

Comments (10 posted)

OpenOffice.org Newsletter

The October, 2007 edition of the OpenOffice.org Newsletter is out with the latest OO.o office suite articles and events.

Full Story (comments: none)

Mozilla Links Newsletter

The October 25, 2007 edition of the Mozilla Links Newsletter is online, take a look for the latest news about the Mozilla browser and related projects.

Full Story (comments: none)

The beta version of CommonDesktop Infrastructure is released

The first beta release of the CommonDesktop Infrastructure has been announced. "The programs provide the functions which convert protocol of QtDBUS, UNO, and SOAP. So, you can call Qt program from UNO program vice versa. And, you can also call SOAP service from Qt or UNO program. The code generators which generate protocol conversion codes from IDL are included. So, you can use Common Desktop Infrastructure by only writing IDL. In short, you can seamlessly connect KDE4/Qt4, OpenOffice/UNO, and SOAP."

Full Story (comments: none)

KnowledgeTree OSS STABLE 3.5 is now available (SourceForge)

Stable version 3.5 of KnowledgeTree, a cross-platform document management system, has been announced. "This release, the first of the 3.5 series, presents some major updates to KnowledgeTree. Some of the highlights are... - KnowledgeTree is now licensed under the GPLv3; - KnowledgeTree has moved to PHP5 and MySQL 5; - A brand new powerful search system has been implemented."

Full Story (comments: none)

md5deep version 2.0.1 released (SourceForge)

Version 2.0.1 of md5deep has been announced. The software: "Computes the MD5, SHA-1, SHA-256, Tiger, or Whirlpool message digest for any number of files while optionally recursively digging through the directory structure. Can also match input files against lists of known hashes in a variety of formats. I've published version 2.0.1 to fix a compilation bug on older Linux systems. The new code required a newer kernel to work; this version is backwards compatible."

Comments (none posted)

GCC 4.3.0 Status Report

The GCC 4.3.0 Status Report has been published. "We're still in Stage 3 for GCC 4.3. As before, I think a reasonable target for creating the release branch is less than 100 open regressions. At present, we're at 184 -- and, of those, 36 are P1."

Full Story (comments: none)

Caml Weekly News

The October 30, 2007 edition of the Caml Weekly News is out with new Caml language articles.

Full Story (comments: none)

Haskell Weekly News

The October 25, 2007 edition of the Haskell Weekly News is online. "It has been a huge month for the Haskell community, with the Haskell Workshop, ICFP and CUFP conferences, the second international Haskell Hackathon, and 63 libraries and tools uploaded to hackage! A round of applause to everyone involved!" (Thanks to Don Stewart).

Comments (none posted)

SBCL 1.0.11 has been released

Version 1.0.11 of SBCL has been announced. "Steel Bank Common Lisp 1.0.11 has been released on 25 October 2007. This version adds a semaphore interface, improves stack allocation of lists, removes locking from hash table accessors, and fixes some bugs."

Full Story (comments: none)

Perl 6 Design Meeting Minutes for 24 October 2007 (use Perl)

The minutes from the October 24, 2007 Perl 6 Design Meeting have been published. "The Perl 6 design team met by phone on 24 October 2007. Larry, Allison, Patrick, Jerry, Will, Jesse, and chromatic attended."

Comments (none posted)

PyDirectio 1.0.2 announced (SourceForge)

Version 1.0.2 of PyDirectio has been announced. "A Python interface to open/read/write on a direct I/O context. This is an interface to open(), read(), write() and close() on a direct I/O context (O_DIRECT flag) from the Python programming language. For those of you who lurk around Cheeseshop from time to time I am happy to bring you the first updated version of directio for Python on Linux in over a year! This release fixes several major problems with the first public edition so if you are an older edition make sure you update to this newer edition that fixes major bugs!"

Comments (none posted)

Python-URL! - weekly Python news and links

The October 29, 2007 edition of the Python-URL! is online with a new collection of Python article links.

Full Story (comments: none)

Tcl-URL! - weekly Tcl news and links

The October 30, 2007 edition of the Tcl-URL! is online with new Tcl/Tk articles and resources.

Full Story (comments: none)

Data Sources as Web Services (XML.com)

Kyle Gabhart introduces WS02 on on O'Reilly's XML.com. "Kyle Gabhart describes WS02's Data Services, a new feature in WS02 that allows for rapid creation of web services wrapping relational, Excel, CSV, and JNDI data sources quickly and easily."

Comments (none posted)

monotone 0.37 released

Version 0.37 of monotone, a version controle system, has been released. "Time for a new, and currently not so regular release. This is version[] 0.37, with quite a number of changes that I imagine would be interesting for quite a lot of people."

Full Story (comments: none)

Maybe UCANN school ICANN on whois (Linux Journal)

Doc Searls covers an ICANN debate on the future of the whois command, the comment period closes at the end of October. "Raise your hand if you use whois every day. Even if your hand isn't up, and you just regard whois as am essential sysadmin tool, this post is for you. Because if you're interested in keeping whois working for the those it was made for in the first place, you need to visit the battlefield where whois' future is being determined right now. That is, you must be Beowulf to the Grendel that is the Intellectual Property Community. Worse, you must confront him in the vast cave that is ICANN."

Comments (9 posted)

Itty-bitty, teeny-weeny Linux PCs (DesktopLinux)

DesktopLinux looks at some small yet full-featured Linux-powered PCs. "When I say full-featured, don't mean Internet tablets, like the Nokia N800, or PDAs (personal digital assistants), such as Palm's Foleo mobile companion. No, what I wanted to see were real desktops or laptops that I could fit into a coat pocket. Here's what I found."

Comments (15 posted)

Second KOffice Sprint in Berlin focuses on release, polish (KDE.News)

KDE.News covers the KOffice Sprint. "This weekend, ten KOffice hackers congregated once again in the hospitable Berlin KDAB headquarters. KOffice has come a long way in six months: all the groundwork has been laid for the new version, KOffice 2.0. From Krita to KPresenter, KWord to KSpread, KChart to Karbon, KPlato to Kexi, and from KFormula to Kivio, the big underlying frameworks are ready. This meeting was called to decide on common look & feel issues and a release plan and schedule."

Comments (none posted)

SCO claims it has Unix business buyer (Linux-Watch)

Linux-Watch reports that York Capital Management is interested in buying parts of the bankrupt SCO. "No one would buy this plot element in a TV drama like Boston Legal, but The SCO Group claims it has a buyer lined up, a subsidiary of York Capital Management that wants to buy its Unix business and associated Linux lawsuits. One might well ask, "What business?" SCO is in danger of being delisted from the Nasdaq stock exchange; it's filed for Chapter 11 bankruptcy; it's lost all claims to the Unix IP (intellectual property) to Linux rival Novell; and its Unix business continues to decline and lose money. Who would want to buy such a company's assets?"

Comments (11 posted)

Novell cries Wolfe (Linux-Watch)

Linux-Watch reports on Novell's hiring of Tim Wolfe. "Novell appointed Tim Wolfe as president of Novell Americas. In this position, he'll be responsible for the execution of Novell's strategy across the Americas. Before this job, Wolfe, who brings nearly three decades of software, technology and consulting leadership experience to the role, most recently held the position of vice president and general manager of Novell's East region in the United States. He is expected to play a key role in Novell's transition to a greater focus on customers and partners in implementing the company's go-to-market strategy."

Comments (none posted)

Is Linux really losing market share to Windows? (Linux-Watch)

Linux-Watch takes a look at an IDG server report that shows Linux losing in the server market. "Let's look closer at what IDG is really doing. First, the actual number of Linux servers is still increasing. What's "decreased" is its rate of increase. Despite the impression you may get from Microsoft ads, almost no one is turning in Linux servers for Windows servers."

Comments (19 posted)

New Desktop Face-Off: Gnome 2.20 vs KDE 3.5 (O'ReillyNet)

Judith Myerson compares the latest versions of KDE and GNOME on O'Reilly. "With the new features that Gnome and KDE (K Desktop Environment) are adding, each desktop environment is challenging the other for a larger share of the market. If Linux-like operating systems come with one desktop environment, the user has the option to add to the other. Because of the ever-increasing sophistication of the new features, some latest versions of the operating system are including packages for both desktop environments, allowing users to have the option of switching from one desktop environment to another. In this article I will briefly talk about the new features of both Gnome and KDE, and then look at some similarities and important differences between the two desktop environments."

Comments (38 posted)

Asus Eee PC 701 review (CNET)

CNET has a review of the Asus Eee laptop, which is a small, lightweight system running Linux. "The Eee PC doesn't use a Microsoft operating system, which is part of the reason it's so inexpensive. Instead, Asus supplies its own Linux-based graphical user interface. The laptop also ships with some 40 applications, which is arguably more than you'd get with a standard Windows laptop. It includes Firefox for browsing the Web, Skype, OpenOffice and SMPlayer for video playback."

Comments (18 posted)

Managing Linux servers made easy with Linux q-Status Rev 5 (Linux-Watch)

Linux-Watch takes a look LogiQwest's Linux q-Status Server Analysis and Configuration software. "q-Status supports summary reports of all servers in the data center. A new feature is dynamic Disk summary reporting. This summarizes not just the total disk space used, on both individual servers and across the server farm. It also enables administrators to track storage space that's available with dynamic file system type filtering (e.g. root, data, var). "We are very proud of the new disk summary report as it easily provides answers IT and finance have long searched for," said Michael Barto, LogiQwest's lead product evangelist. "It identifies which servers have the most free space available and which servers have the most disk space used.""

Comments (none posted)

Linux-powered PMP sports 7-inch touchscreen (LinuxDevices)

LinuxDevices looks at the iRiver Unit2. "iRiver is readying a Linux-based media recorder/player comprised of a detachable mobile unit and tethered docking station. The Unit2's base station offers a DVD/CD player, TV tuner, and PC-style I/O, while the detachable display features a 7-inch WVGA (800x480) touchscreen, WiFi, and USB."

Comments (2 posted)

Wubi = super easy Linux installation (Crunch Gear)

Crunch Gear has a short review of Wubi, which runs Ubuntu from within Windows. "I don’t have the patience to go through a full install of Ubuntu but I also don’t think that running the Live CD does the operating system justice. If only there were an easy-to-use Windows installer that didn’t "require you to modify the partitions of your PC or to use a different bootloader." Enter Wubi. Here’s some more information from Wubi’s FAQs… "How does Wubi work? Wubi adds an entry to the Windows boot menu which allows you to run Linux. Ubuntu is installed within a file in the windows file system (c:\wubi\disks\system.virtual.disk), this file is seen by Linux as a real hard disk. Is this running Ubuntu within a virtual environment or something similar? No. This is a real installation, the only difference is that Ubuntu is installed within a file as opposed to being installed within its own partition."

Comments (none posted)

EFF wins reexamination of bogus patent

The Electronic Frontier Foundation has announced that a bogus patent will be reexamined by the U.S. Patent and Trademark Office. "NeoMedia Technologies, Inc., claims to own rights to all systems that provide information over computer networks using database-like lookup procedures that rely on scanned inputs, such as a barcode. NeoMedia has used these claims to threaten and sue innovators in the mobile information space. But EFF's reexamination request, filed in conjunction with Paul Grewal and James Czaja of Day Casebeer Madrid & Batchelder, showed that the functionality covered by NeoMedia's bad patent was repeatedly included as part of prior patent applications from other companies."

Full Story (comments: none)

Book Auction to Benefit the FreeBSD Foundation

No Starch Press is holding an auction to benefit the FreeBSD Foundation. "Open source software depends on community support and the efforts of countless non-profits, like The FreeBSD Foundation. In an enlightened example of a for-profit company acting to benefit a non-profit organization, book publisher No Starch Press of San Francisco, California will auction the first copy of the second edition of "Absolute FreeBSD" to the highest bidder. All proceeds will benefit The FreeBSD Foundation."

Full Story (comments: none)

Monsoon Multimedia GPL lawsuit settled

The Software Freedom Law Center has announced the end of the Monsoon Multimedia GPL compliance lawsuit. "As a result of the plaintiffs agreeing to dismiss the lawsuit and reinstate Monsoon Multimedia's rights to distribute BusyBox under the GPL, Monsoon Multimedia has agreed to appoint an Open Source Compliance Officer within its organization to monitor and ensure GPL compliance, to publish the source code for the version of BusyBox it previously distributed on its Web site, and to undertake substantial efforts to notify previous recipients of BusyBox from Monsoon Multimedia of their rights to the software under the GPL. The settlement also includes an undisclosed amount of financial consideration paid by Monsoon Multimedia to the plaintiffs."

Comments (29 posted)

Nigerian education selects Mandriva Linux

Mandriva has announced a deployment in Nigeria. "Mandriva today announced that the Nigerian government has selected Intel-powered classmate PCs running on Mandriva Linux for educational use in nationwide pilot in Nigeria. Mandriva is working with Intel Corporation and Technology Support Center Ltd. to provide 17,000 Intel-powered classmate PC. The aim of this project is to improve the quality of technology delivered to students, and to help teachers and parents."

Full Story (comments: 1)

Eben Moglen on NetApp v. Sun

The Software Freedom Law Center has landed firmly in Sun's corner in its patent dispute with NetApp. ""NetApp, in bringing this litigation, has announced that it wishes to prevent Sun from sharing ZFS with the community. This conduct is a misuse of questionable patents to prevent the spread of valuable technology. Using patent threats and litigation against free software and open source communities is an abuse of the public interest the law is supposed to serve."

Comments (47 posted)

Opera introduces Opera Link

Opera has announced the launch of Opera Link. "Wherever you are, whatever Opera browser you use, or whichever device you use Opera on, you can instantly access your bookmarks, Speed Dial, and personal bar. Say goodbye to the tedious hassle of entering text in your mobile phone to visit your top sites and say hello to a new level of convenience from device to device."

Full Story (comments: none)

The Portland Group announces PGI Release 7.1 Optimizing Compilers

The Portland Group has announced release 7.1 of its suite of Fortran, C and C++ compilers and development tools. "PGI(R) compilers and tools are used widely in high-performance computing (HPC), the field of technical computing engaged in the modeling and simulation of complex phenomena, such as ocean modeling, weather forecasting, seismic analysis, bioinformatics and other areas. PGI compilers, which convert software programs into the binary instructions that a computer can understand, are recognized in the HPC community for delivering world-class performance across a wide spectrum of applications and benchmarks, and they are referenced regularly as the industry standard for performance and reliability."

Comments (none posted)

Slashdot Turns Ten

Slashdot has announced its 10 year anniversary. "The tech community news site, started in 1997 by Rob "CmdrTaco" Malda with Jeff "Hemos" Bates, has grown to an Internet phenomenon in its 10 year run. Slashdot features stories submitted by readers and posted by a dedicated Slashdot editorial board. The site serves as a water cooler for a generation of technophiles and established the model for today's changing media landscape."

Full Story (comments: none)

The Yellow Dog Drives Autonomous Jeep at DARPA Challenge

Terra Soft's Yellow Dog Linux will be used for an autonomous vehicle controller. "In early October, Terra Soft Solutions was engaged by Axion Racing to assist with the integration of a Sony PlayStation3 running Yellow Dog Linux into "Spirit", Axion's fully autonomous Jeep Grand Cherokee which is this week competing in the qualifying rounds for the DARPA autonomous vehicle challenge in Victorville, California. Integral to the on-board, realtime image processing system, the YDL PS3 rides atop a set of 1U rackmount servers inside Spirit which drives itself through simulated city traffic and obstacles during the qualifying rounds, currently underway."

Full Story (comments: none)

Network Security Assessment--New from O'Reilly Media

O'Reilly has published the book Network Security Assessment by Chris McNab.

Full Story (comments: none)

The PHP Anthology, 2nd Edition--New from SitePoint

SitePoint has published the book The PHP Anthology, 2nd Edition by Ben Balbo, Harry Fuecks, Davey Shafik, Ligaya Turmelle and Matthew Weler O'Phinney.

Full Story (comments: none)

First ever ModSecurity public training at OWASP/WASC conf in SJ

A public ModSecurity training event has been announced. "As part of the upcoming OWASP/WASC AppSec 2007 conference in San Jose, Ryan Barnett is going to give a two day ModSecurity Boot-Camp Training course on Nov 12th and 13th. For those of you who don't know Ryan, he is ModSecurity Community Manager and Director of Application Security Training at Breach Security, and one of the best ModSecurity experts out there. As an additional bonus, Ivan Ristic, The creator of ModSecurity will also be in attendance for portions of the class."

Full Story (comments: none)

Gaming Miniconf at LCA2008 - Call for Participation

A Call for Participation has gone out for the Gaming Miniconf at LCA2008. The Miniconf will be held on January 29, 2008 in Melbourne, Australia. "All speakers **must** submit their submissions before the '''15th of November'''. Speaker slots are limited and will fill up fast so don't leave your submission to the last moment!"

Full Story (comments: none)

The 2008 Linux Storage and Filesystem Workshop

The 2008 Linux storage and filesystem workshop will be held in San Jose, California next February 25 and 26. There is a call for proposals out there for those who would like to participate. "This year we're trying to concentrate on more problem solving sessions, short term projects and joint sessions."

Full Story (comments: none)

RailsConf 2008: Call For Participation Open

A call for participation has gone out for RailsConf 2008, submissions are due by December 13. "RailsConf 2008 is returning to the Oregon Convention Center, Portland, Oregon, from May 29-June 1, 2008. RailsConf, co-produced by Ruby Central, Inc. and O'Reilly Media, Inc., will gather over 1,600 attendees for four days of keynotes, sessions, tutorials, panels, and events to explore the latest Rails developments."

Full Story (comments: none)

BlogWorld Expo - Blogging Trend Story Ideas

The 2007 BlogWorld Expo will be held on November 8-9, 2007 in Las Vegas, NV. "The first and only industry-wide tradeshow, conference, and media event dedicated to promoting the dynamic industry of blogging and new media. In addition to the only industry-wide exhibition, BlogWorld will feature the largest blogging conference in the world including more than 50 seminars, panel discussions and keynotes from iconic personalities on the leading-edge of online technology and internet-savvy business."

Full Story (comments: none)

The 2007 Chicago Perl Hackathon (use Perl)

The 2007 Chicago Perl Hackathon has been announced. "This three-day event will be held at Hosteling International Chicago, which is located in downtown Chicago at the corner of Congress and Wabash. The hackathon runs from December 14th through December 16th."

Comments (none posted)

Speaker list for FOSS.IN 2007 announced

The first cut at the speaker list for FOSS.IN has been announced. The conference will be held in Bangalore, India, December 4-8. "This year, I simply do not have the energy to make a separate post about all the famous names who are coming to FoSS.IN, but let me wave to Rusty Russel, Mitchell Baker, James Morris, Jim Grisanzio, Thomas Gleixner, Harald Welte, Sam Hocevar, Danese Cooper, Andrew Cooper Cowie, Josh Berkus, Till Adam, Ulrich Drepper and many others who are coming this year." Click below for more details.

Full Story (comments: none)

Events: November 8, 2007 to January 7, 2008

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
November 5 November 9	Python Bootcamp with Dave Beazley	Atlanta, USA
November 8 November 9	Blog World Expo	Las Vegas, NV, USA
November 10 November 11	Linuxtage	Essen, NRW, Germany
November 11 November 17	Large Installation System Administration Conference	Dallas, TX, USA
November 12 November 16	Ruby on Rails Bootcamp with Charles B. Quinn	Atlanta, USA
November 12 November 15	OWASP & WASC AppSec 2007 Conference	San Jose, USA
November 12 November 16	ApacheCon US 2007	Atlanta, GA, USA
November 13 November 14	IV Latin American Free Software Conference	Foz do Iguacu, Brazil
November 15 November 18	Piksel07	Bergen, Norway
November 15	Alfresco European Community Conference	Paris, France
November 16 November 18	aKademy-es 2007	Zaragoza, Spain
November 20 November 23	DeepSec ISDC 2007	Vienna, Austria
November 22 November 23	Conferencia Rails Hispana	Madrid, Spain
November 24	LinuxDay in Vorarlberg (Deutschland, Schweiz, Liechtenstein und Österreich)	Dornbirn, Austria
November 26 November 29	Open Source Developers' Conference	Brisbane, Australia
November 28 November 30	Mono Summit 2007	Madrid, Spain
November 29 November 30	PacSec 2007	Tokyo, Japan
December 1	Django Worldwide Sprint	Online, World
December 1	London Perl Workshop 2007	London, UK
December 4 December 8	FOSS.IN 2007	Bangalore, India
December 7 December 8	Free Software Conference Scandinavia	Gotherburg, Sweden
December 7 December 8	PGCon Brazil	Sao Paulo, Brazil
December 10	Paris on Rails (2nd Edition)	Paris, France
December 11 December 12	3rd DoD Open Conference: Deployment of Open Technologies and Architectures within Military Systems	Vienna, VA, USA
December 15 December 22	Unix Meeting 2007	IRC, Worldwide
December 27 December 30	24th Chaos Communication Congress	Berlin, Germany
December 31	Israeli Perl Workshop	Ramat Efal, Israel

If your event does not appear here, please tell us about it.