LWN.net Weekly Edition for November 19, 2009

Btrfs for Rawhide users?

It is worth noting at the outset that Fedora is not, yet, considering using Btrfs in Rawhide by default. What has been proposed, instead, is the implementation of a "system rollback" feature for Rawhide users who are crazy enough to install on Btrfs despite its young and immature state. If this feature works out, it could remove much of the risk of tracking Rawhide and begin the exploration of a new capability which could prove highly useful for Linux users in general in the future.

One of the many features provided by Btrfs is copy-on-write snapshots. At any time, it is possible to freeze an image of the state of the filesystem. Snapshots are cheap - at creation time, their cost is almost zero. As changes are made to the filesystem, copies will be made of modified blocks while the snapshot remains unchanged. One can certainly fill a filesystem through use of the snapshot facility - and filling Btrfs filesystems remains a bit of a hazardous thing to do - but Btrfs will share data between snapshots for as long as possible.

The value of snapshots to system administrators is fairly obvious: a snapshot can be taken immediately prior to an operating system upgrade. Should that upgrade turn out to be less of a step forward than had been hoped, the filesystem can simply be reverted back to its pre-upgrade state. The days of digging around for older versions of a broken packages - perhaps with the assistance of a rescue disk - should be long gone.

That said, there are a number of details which need to be worked out before snapshots can be made ready even for Rawhide users, much less the wider user community. Perhaps the biggest problem is that Btrfs snapshots cover the entire filesystem, so reverting to an older state will lose all changes made to the filesystem in the meantime. If a system update fails to boot, dumping the update seems like a straightforward choice - there will be no other changes to lose. But going back to a snapshot after the system has been running for a while could lose a fair amount of work, log data, etc. along with the unwelcome changes. One can always cherry-pick changed files after reverting to the snapshot, but that would be a tedious and error-prone process.

There are a lot of user interface details to take care of as well. Tools need to be created to allow administrators to look at existing snapshots, mount them for examination, clean them up, and so on. Btrfs will probably have to be extended with a concept of a user-selectable "default" snapshot for each filesystem. Grub needs some work for boot-time snapshot selection. There is also talk of eventually adding snapshot-browsing support to Nautilus as well.

Snapshots will clearly be a useful feature for Linux in the future. Back in your editor's system administration days, backup tapes were occasionally used to recover from disk disasters, but much more frequently used to help users recover from "fat-finger" incidents. Snapshots are not true backups, but they should certainly be useful as a quick error-recovery mechanism. Your editor is looking forward to the day when his system always supports a series of snapshots allowing the recent state of the filesystem to be recovered.

A snapshot is a heavyweight tool for dealing with system upgrade problems, though. In the longer term, it would make sense to have better rollback support built into the package management system itself. Interestingly, Yum and RPM have had some rollback support in the past, but that feature does not seem to be well supported now. Providing rollback support at this level is a hard problem, to say the least, but solving that problem would put a powerful tool into the hands of Linux system administrators.

In the absence of this feature, filesystem-level snapshots will have to do; certainly they are a major improvement over what we have now. In the short term, potential users should remain aware that Btrfs is a very young filesystem, and that snapshots may not be a viable recovery mechanism if the filesystem itself gets corrupted. In the longer term, though, there will be a day when we will wonder how we ever used our systems without this feature. The work being done by the Fedora developers is an important step in that direction.

Comments (22 posted)

Reducing HTTP latency with SPDY

Google unveiled an experimental open source project in early November aimed at reducing web site load times. SPDY, as it is called, is a modification to HTTP designed to target specific, real-world latency issues without altering GET, POST, or any other request semantics, and without requiring changes to page content or network infrastructure. It does this by implementing request prioritization, stream multiplexing, and header compression. Results from tests on a SPDY-enabled Chrome and a SPDY web server show a reduction in load times of up to 60%.

SPDY is part of Google's "Let's make the web faster" initiative that also includes projects targeting JavaScript speed, performance benchmarking, and analysis tools. Mike Belshe and Roberto Peon announced SPDY on November 11 on both the Chromium and Google Research blogs, noting that "HTTP is an elegantly simple protocol that emerged as a web standard in 1996 after a series of experiments. HTTP has served the web incredibly well. We want to continue building on the web's tradition of experimentation and optimization, to further support the evolution of websites and browsers."

Finding the latency in HTTP

The SPDY white paper details the group's analysis of web latency, beginning with the observation that although page requests and responses rely on both HTTP as the application-layer protocol and TCP as the transport-layer protocol, it would be infeasible to implement changes to TCP. Experimenting on HTTP, on the other hand, requires only a compliant browser and server and can be tested on real network conditions.

The group found four factors to be HTTP's biggest sources of latency. First, relying on a single request per HTTP connection makes inefficient use of the TCP channel and forces browsers to open multiple HTTP connections to send requests, adding overhead. Second, the size of uncompressed HTTP headers, which comprise a significant portion of HTTP traffic because of the large number of HTTP requests in a single page. Third, the sending of redundant headers — such as User-Agent and Host — that remain the same for a session. Finally, the exclusive reliance on the client to initiate all HTTP requests, when there are cases where the server knows that related content will be requested, but cannot push it to the client.

SPDY tackles these weaknesses by multiplexing an unlimited number of concurrent streams over a single TCP connection, by allowing the client to assign priorities to HTTP requests in order to avert channel congestion, and by compacting HTTP request and response headers with gzip compression and omitting the redundant transmission of headers. The SPDY draft specification also includes options for servers to initiate content delivery. The available methods are "server push," in which the server initiates transmission of a resource via an X-Associated-Content header, and "server hint," in which the server only suggests related resources to the client with X-Subresources.

In addition, SPDY is designed to run on top of SSL, because the team decided it was wiser to build security into its implementation now than to add it later. Also, because SPDY requires agents to support gzip compression for headers, it compresses the HTTP data with gzip too.

The important thing to note is that SPDY's changes affect only the manner in which data is sent over the wire between the client and the server; there are no changes to the existing HTTP protocol that a web page owner would notice. Thus, SPDY is not a replacement for HTTP so much as a set of possible enhancements to it.

Comments on the blog posts indicate that although most readers see the value in header compression and request prioritization, some are skeptical of the need to multiplex HTTP requests over a single TCP connection. Other alternatives have been tried in the past, notably HTTP pipelining and the Stream Control Transmission Protocol (SCTP).

The white paper addresses both. SCTP, it says, is a transport-layer protocol designed to replace TCP, and although it may offer some improvements, it would not fix the problems with HTTP itself, which SPDY attempts to do. Implementing SCTP would also require large changes to client and server networking stacks and web infrastructure. The latter is also true for similar transport-layer solutions like Structured Stream Transport (SST), intermediate-layer solutions like MUX, and HTTP-replacements like Blocks Extensible Exchange Protocol (BEEP).

The problem with pipelining, it says, is that even when multiple requests are pipelined into one HTTP connection, the entire connection remains first-in-first-out, so a lost packet or delay in processing one request results in the delay of every subsequent request in the pipeline. On top of that, HTTP pipelining is difficult for web proxies to implement, and remains disabled by default in most browsers. The fully multiplexed approach taken by SPDY, however, allows multiple HTTP requests and responses to be interleaved in any order, more efficiently filling the TCP channel. A lost packet would still be retransmitted, but other requests could continue to be filled without pausing to wait for it. A request that requires server-side processing would form a bottleneck in an HTTP pipeline, but SPDY can continue to answer requests for static data over the channel while the server works on the slower request.

Implementation and test results

The development team wrote a SPDY web server and added client support in a branch of the Chrome browser, then ran tests serving up "top 100" web site content over simulated DSL and cable home Internet connections. The test included SSL and non-SSL runs, single-domain and multiple-domain runs, and server push and server hint runs. The resulting page load times were smaller in every case, ranging from 27.93% to 63.53% lower.

The team's stated goal is a 50% reduction in load time; the average of the published tests in all of their variations is 48.76%. Though it calls the initial results promising, the team also lists several problems — starting with the lack of well-understood models for real world packet loss behavior.

SPDY remains an experiment, however, and the team solicits input on a number of open questions, including dealing with the latency introduced by SSL handshakes, recovering from a lost TCP connection, and how best to implement the server-side logic to truly take advantage of server push and server hint. Interested people are encouraged to join the mailing list and download the code.

So far, only the modified Chrome client code is available, and that from the public Subversion repository, not binary downloads. Peon said that the server release is coming soon, and the project page says that the test suite and benchmarking code used in Google's test will be released under an open source license as well.

A 50% reduction in page load times is nothing to sneer at, particularly when all of the gains come from tweaking HTTP's connection and data transfer behavior. Header compression alone gives noticeable savings; the white paper states that it resulted in an "~88% reduction in the size of request headers and an ~85% reduction in the size of response headers." The future of the web may indeed include new protocols like SCTP and BEEP, but SPDY is already demonstrating that there is plenty of room for improvement without drastically altering the protocol stack.

Comments (61 posted)

Notes from the LF End User Summit

The End User Summit draws a different crowd than any other event. Well-known Linux developers are present, certainly, but they do not form the majority of the crowd; they are, instead, strongly outnumbered by representatives of banks, insurance companies, and financial firms. Old conference T-shirts are far outnumbered by suits and ties in this crowd. The End User Summit, in other words, caters to enterprise distribution customers and others who are using Linux in high-stakes situations - even a major stock exchange which has based its operation on Gentoo. It makes for an interesting combination of people and a unique set of conversations.

One speaker was Brian Clark from the New York Stock Exchange. NYSE's systems run under high pressure and tight constraints. They process some three billion transactions per day - more than Google does - and those transactions need to execute in less than one millisecond. Customers can switch to competing exchanges instantly and for almost no cost, so if NYSE's systems are not performing, its customers will vanish. A typical trading day involves the processing of 1.5TB of data; some 8 petabytes of data are kept online. And this whole operation runs on Linux.

NYSE is highly concerned with software quality and security; they are subject to thousands of attacks every day. Downtime is to be limited to 90 seconds per year. All told, Linux has worked very well in this setting. NYSE had some requests, though, including the increasingly common desire for a way to move everything except a specific application off of a given core. Brian requested a way to lock a process's memory in place - a functionality which mlock() would appear to have provided for many years. He would also like a non-disruptive way to measure latencies, especially in the network stack.

In the end, he says, NYSE likes Linux because of the community which stands behind it - an interesting position given NYSE's rather low profile in that community. One place where it was suggested NYSE could help would be to advise the developers on the best placement of tracepoints into the network stack to yield the sort of latency measurements they would like to see.

Al Gillen of IDC is a common presence at this sort of event; he gave a chart-heavy talk on how IDC expects things to go in the server marketplace. The outlook for Linux server shipments would appear to be bright. One interesting tidbit from the talk: Linux server shipments will be growing strongly in the coming years, while Unix will be declining. That means that, in 2013, the Linux market looks likely to reach half the revenue value of the Unix server market. Unix may be suffering, but there's still a lot of money being spent on it.

Anthony Golia of Morgan Stanley discussed the use of Linux there; Morgan Stanley has been heavily using the operating system for several years now, and is running it on tens of thousands of systems. It was, he says, a bit of a rough start, but Morgan Stanley learned that the community "lends itself well to partnership." The company figured out how to send fixes back upstream and has experience the "warm fuzzy feeling" that comes with getting fixes merged. In recent times they are finding far fewer bugs and are quite happy with the choice to go with Linux.

Anthony had some requests too, beginning with support for TCP offload engines. What Morgan Stanley really needs, though, is shorter network latencies. Trades are dependent on getting orders in quickly in response to events, and latencies work against that goal. They would like a way to generate long-term statistics of a process's memory use, mostly as a way of knowing whether it's safe to load more work onto a specific server. There was also a request for better coordination between distributors and hardware manufacturers, yielding support for new hardware as soon as that hardware is available.

Jeffrey Birnbaum of the Bank of America led a session on shortcomings he sees with Linux at this time. In particular, Jeffrey anticipates a future dominated by increasing availability of fast CPUs and the growing influence of solid-state storage devices. The world is changing, and he worries that Linux is not changing quickly enough to keep up with it. Technology is improving quickly, he says, and the kernel is holding users back.

Specific problems include latency in the network stack and the ability of networking to make use of large numbers of CPUs. TCP, he says, is not scalable, but it wasn't clear where the problems are. One request that was clear was a means by which messages could be sent to multiple destinations with a single system call - something akin to the proposed sendmmsg() system call. He suggested that the time has come to move beyond POSIX interfaces - he is a fan of Ulrich Drepper's event interface proposal - and that the use of protocols like SATA to talk to solid-state storage is a mistake. There was also some discussion about difficulties getting a scalability problem with the epoll_wait() system call fixed.

Perhaps the clearest point to emerge from this session is that users like Jeffrey need a solid channel to communicate with the development community about their needs and frustrations. One would think that this would be an ideal role for enterprise distribution vendors to fill; indeed, in the following session, Novell's Carlos Montero-Luque described the session as a great advertisement for commercial distributions. But, for whatever reason, those distributions do not appear to be filling that role in this case.

Carlos, along with Red Hat's Brian Stevens, talked about the future as the distributors see it. There was lots of talk on the value of Linux on mainframes, which seems to be of great interest to this user community currently. Interestingly, Brian noted that Red Hat is not entirely sure that the success which has been achieved with Linux can be replicated at other levels; the JBoss development community, for example, is nearly 100% Red Hat employees.

On the subject of unpaid Linux, Brian claimed that these deployments were "fantastic." Anything which grows the overall market can only be good for the participants therein. Carlos had some darker comments about how unpaid Linux is not "free," and that it will always be paid for in some other way.

[PULL QUOTE: Everybody was afraid of being sued and ending up on the front page of the Wall Street Journal, so outright prohibitions on the use of open source were common. END QUOTE] Tim Golden is a manager at a high-profile American bank; in his talk on "the changing role of enterprise open source," though, he was clear to point out that he was speaking only for himself. This talk started with the relatively early days, when companies like banks saw open source as being far too risky to use. Everybody was afraid of being sued and ending up on the front page of the Wall Street Journal, so outright prohibitions on the use of open source were common.

There were a couple of intermediate steps, including one where managers came to the radical conclusion that the submission of bug fixes did not deprive a company of its Valuable Intellectual Property. During this time, fears about the use of open source faded considerably, and companies increasingly decided that they could tolerate whatever risk remained - at least in "high value" situations.

The current situation is heavily affected by the financial crisis; financial companies have realized that they must find a way to be competitive with far less money. This understanding has helped to usher in the "open source software as a strategy" era, with companies setting up formalized management programs for open source. An interesting thing is happening in some companies as they go through this process, though: executives are figuring out that it's hard to drive open-source projects from the back seat. They are also coming to the conclusion that participation in development projects is not as disruptive as they had once thought.

So now these companies are beginning to dip their toes in the water and look at ways to participate. There are lots of options, ranging from simple cash contributions - which don't create any real linkage with the community - through to investments in companies and "intellectual property contributions." Eventually, says Tim, we'll start to see something that was once unthinkable: development projects being run by end users.

That last statement maybe reveals something about how these companies see free software. To them, projects run by end users are a new, scary, and exotic thing. But your editor would submit that almost every development project of interest is run by end users. The developers who came together to create the Linux kernel weren't working for others. The group that pulled together their patches and released "a patchy" server were planning to deploy that server (now "Apache") themselves. As end users in the financial industry start to run projects aimed at meeting their own needs, some of those projects, at least, should prove equally successful.

There is no need to convince the financial industry that free software can benefit its operation; they have understood that for a few years now. Convincing this industry that contributing to the software it uses makes sense has been somewhat harder. It would appear that this message is starting to be heard, and companies in this industry are beginning to look for ways to reach out to the development community. Events like the End User Summit seem like an ideal way to facilitate communication between the existing development community and its future members; it is a learning experience for everybody involved.

Comments (36 posted)

The LWN.net Weekly Edition will be early next week

Comments (1 posted)

Page editor: Jonathan Corbet

Security

TLS renegotiation vulnerability

Transport Layer Security (TLS), and its predecessor Secure Sockets Layer (SSL), are commonly used protocols for encrypting internet traffic, so TLS vulnerabilities can potentially affect a wide range of internet services. A recently disclosed flaw in the TLS protocol—though there is some dispute whether TLS is at fault—allows an "injected plaintext" attack against an encrypted session. This allows a "man in the middle" (MITM) attacker to prefix a victim's request with their own data, which gets interpreted by the server as if it came from the victim.

The flaw was disclosed on the Internet Engineering Task Force (IETF) TLS mailing list by Martin Rex of SAP on November 4, but it had actually been discovered two months earlier. Marsh Ray of PhoneFactor, Inc. discovered the problem in early August, but kept it quiet while alerting the IETF and various TLS developers and vendors. Those organizations came together as "Project Mogul" to find a solution to the problem. But once Rex had posted publicly about a possible MITM attack—independently discovered—against TLS (specifically Microsoft's IIS), Ray and his colleague Steve Dispensa released additional information showing that it was a pervasive problem for TLS (including Apache web servers, OpenSSL, GNUTLS, and others).

TLS allows clients and servers to renegotiate various session parameters within the TLS connection. When the renegotiation is done, however, TLS applications still accept data that came in before the renegotiation as if it were in the new security context. That hole allows a MITM attack. By arranging that the last data received is from the attacker, then causing a renegotiation with the victim, the attack effectively prepends the attacker's payload to the victim's request.

While it may sound rather theoretical, there is already a proof of concept (PoC) program available, and that has been adapted into a way to steal Twitter credentials. Other kinds of attacks are possible as well, some of which may exploit other internet protocols that use TLS, for example: SMTP (email), FTP, XMPP (Jabber), and some VPN implementations. Exploiting the vulnerability requires the attacker to control some node in between the client and server, which is not that difficult to do these days because of pervasive WiFi. Users are accustomed to using any available access point—especially if they use encrypted connections for their web/email/etc. connections—and that access point could be compromised.

Injecting the attacker's text into an existing TLS session may not seem like a particularly potent attack, but there are several examples of how it could work. Rex's original disclosure was related to web servers configured to require client certificates for some resources, and not for others, which will often lead to renegotiation. That led some to believe that this relatively rare configuration was required, but subsequent examples and explanations make it clear that there are other ways to cause, and exploit, renegotiation.

To attack a web-based application, the attacker typically would send their prefix to the server, then cause the renegotiation to occur. That renegotiation would actually be done between the victim's client and the server (with the MITM attacker just proxying the traffic). Due to the bug, the server would process the prefix in the new security context that gets established via the renegotiation. So, neither the client nor the server have any idea that this has occurred, and the attacker gets to insert his payload into the the client's secure session.

Eric Rescorla is one of those working on a long-term fix, but he also has a fairly straightforward example of the plaintext injection:

    GET /pizza?toppings=pepperoni;address=attackersaddress HTTP/1.1 
    X-Ignore-This:

    GET /pizza?toppings=sausage;address=victimssaddress HTTP/1.1 
    Cookie: victimscookie

    GET /pizza?toppings=pepperoni;address=attackersaddress HTTP/1.1 
    X-Ignore-This: GET /pizza?toppings=sausage;address=victimssaddress HTTP/1.1 
    Cookie: victimscookie

The Twitter attack uses a similar scheme, but exploits the REST-based API for Twitter. In that case, the attacker's Twitter feed shows the victim's server request, which includes the base64-encoded username and password of the victim. In Thierry Zoller's "TLS/SSLv3 renegotiation vulnerability explained" [PDF]—a comprehensive guide to the problem—there is an example of hijacking email traffic using a similar technique: the attacker prepends SMTP commands to send an email to himself, then leaves the DATA command unterminated, effectively capturing the victim's outgoing email message.

Shortly after the initial disclosure, OpenSSL released a workaround that disabled renegotiation. According to Rescorla, almost all servers could disable renegotiation without impacting any needed functionality, but that doesn't fix the problem on the client side. Clients cannot detect whether the server allows renegotiation because the attacker could always fake a rejection of the renegotiation request.

A more long-term fix has been proposed—that site, ironically, has an invalid SSL certificate—by Rescorla, Ray, Dispensa, and Nasko Oskov of Microsoft. Their proposal would extend TLS to add information to each renegotiation about the previous renegotiation. In that way, clients could detect that their connection is not being negotiated for the first time, which would mean that a MITM had done a TLS negotiation already.

While TLS is the right place to make the fix, Adam Langley is unhappy that TLS is being blamed for the problem. His contention is that the applications are making unwarranted assumptions about what TLS provides. While that may be true, "every single protocol that is layered on top of TLS got this wrong", as Ben Laurie points out.

Zoller's paper also has a good summary of other protocols and programs that use TLS, along with their vulnerability status. It may be a hard vulnerability to exploit, because it requires a MITM position along with an ability to predict the client's traffic, but it is rather serious. As Rescorla and others note, though, MITM attacks simply using an attacker's certificate are generally fairly successful because users are willing to accept any certificate presented to them. It really isn't necessary to go to the lengths that are required to exploit this vulnerability until that changes.

[ Thanks to Tzvetan Mikov who suggested this as a topic shortly after it was made public. ]

Comments (23 posted)

New vulnerabilities

apache-conf: cross-site scripting

Comments (none posted)

asterisk: access control violation

Comments (none posted)

cups: denial of service

Comments (none posted)

ffmpeg: denial of service

Comments (none posted)

gimp: integer overflow

Comments (none posted)

java: remote file creation and modification

Comments (none posted)

java: multiple vulnerabilities

Comments (none posted)

kernel: denial of service

Comments (none posted)

openjdk: arbitrary code execution

Comments (none posted)

openldap: man in the middle attack

Comments (none posted)

qt: multiple vulnerabilities

Comments (none posted)

texlive: buffer overflow

Comments (none posted)

wordpress: multiple vulnerabilities

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The 2.6.32-rc7 regression list shows a total of 41 unresolved regressions - a high number for this stage in the development cycle. So we may be a couple weeks away from the final 2.6.32 release yet.

Comments (none posted)

Quotes of the week

Comments (2 posted)

Some approaches to parallelism avoidance

    int sched_wait_block(pid_t pid, struct timespec *uts);

This call would put the process to sleep until the process indicated by pid blocked, at which point the calling process would go back onto the run queue. It would thus allow a sort of "only run me when process pid is sleeping" semantic.

Ingo Molnar responded with a suggestion for a very different approach; to him, this problem is another nail for the "perf events" hammer. An interested process could sign up for "parallelism" events, then receive notifications when specific processes sleep or become runnable. He sees some real benefits from such a capability:

Linus, though, had a very different suggestion: rather than create this whole framework, just add a relatively stupid "only run one of this group of threads at a time" mode to the scheduler. This mode, which could be specified with a new clone() flag, seems like it could solve most of the problems in this area without adding a new set of complicated interfaces.

As of this writing, only sched_wait_block() has an actual patch associated with it, and nobody has committed to writing any others. So the eventual outcome - if any - from this conversation is unclear at best, but it's an interesting exploration of approaches in any case.

Comments (4 posted)

eclone()

The latest proposal is called eclone(); it looks like this:

    int eclone(u32 flags_low, struct clone_args *args, int args_size,
	       pid_t *pids);

The flags_low argument corresponds to the flags argument to the existing clone() call, which is running out of space for new flags. The pids argument is an optional list of PIDs to apply to the new child process, one for each namespace in which the process appears. Everything else goes into args:

    struct clone_args {
	u64 clone_flags_high;
	u64 child_stack_base;
	u64 child_stack_size;
	u64 parent_tid_ptr;
	u64 child_tid_ptr;
	u32 nr_pids;
	u32 reserved0;
	u64 reserved1;
    };

A number of these fields (child_stack_base, child_stack_size, parent_tid_ptr, child_tid_ptr) correspond to existing clone() arguments. clone_flags_high allows the addition of more flags; no new flags are defined in the eclone() proposal, though. The length of the pids array is given by nr_pids, and the reserved fields are there for future expansion.

Comments on the new proposal have been scarce; it may be that the development community has gotten a little tired of seeing these patches over and over. The silence could also mean that there are no objections to this proposal. One big obstacle could remain to the merging of this system call, though: it is there to support the checkpoint/restart facility, which is definitely not ready for merging into the mainline. Getting checkpoint/restart to a completed and maintainable state is likely to take some time; until then, there may be reluctance to add a new system call which does not, yet, have any real-world users.

Comments (9 posted)

Van de Ven: Some PowerTOP updates

Comments (1 posted)

Kernel development news

High-order GFP_ATOMIC allocation trouble

Since the beginning of the 2.6.31 development cycle, some users have been complaining about an increase in kernel memory allocation failures, leading to log messages, failed applications, and the occasional unwelcome appearance of the out-of-memory killer. Various bugs have been filed (see #14141 and #14265, for example) and a fair amount of head-scratching has gone on. But few developers really know where to start when looking at this kind of problem, and, of those who do, some have been content to write off the problem as being caused by higher-order allocations. So progress has been slow.

High-order (multi-page) allocations are a perennial problem on Linux systems; as memory fragments, it gets increasingly hard to find groups of physically-contiguous pages to satisfy higher-order allocation requests. Whenever possible, kernel code is written to avoid high-order allocations, but there are times when that is difficult. Many of the recently-reported problems seemingly have to do with certain not-top-of-the-line wireless network adapters which require contiguous memory chunks to operate. Fixing the problem is important - users of cheap network interfaces want to run Linux too - but there are also reports of single-page allocation failures.

Fortunately, Mel Gorman is not afraid to wander into that part of the kernel; he has been putting some serious time into reproducing the problem and trying to understand what has gone wrong since 2.6.30. Mel has posted a five-part patch series which tries to make allocation failures less likely again. Looking at what Mel has done provides a good lesson on just how subtle this kind of programming can be.

When looking at this code, it's worth bearing in mind that the kernel has two fundamental mechanisms for recovering memory when it is needed for new allocations. Direct reclaim is active memory cleaning done at allocation time; when an allocation falls short, the process trying to allocate the memory will go off and try to free some memory elsewhere in the system. Direct reclaim has the advantages of immediacy - reclaim work happens right away when memory pressure hits - and of dumping the work into processes which are allocating memory, but there are limits to how long any one process can spend reclaiming memory without introducing unacceptable latencies. So more extensive cleaning is pushed off to the kswapd kernel thread, which is dedicated to that task.

Current mainline kernels do not wake up kswapd from the direct reclaim code if the direct reclaim operation fails to get the job done. But if memory is that tight, kswapd should be running, especially if high-order allocations are needed. So the first patch in Mel's series is a simple one-liner which causes kswapd to be waked on direct allocation failure and, perhaps, to work harder on recovering higher-order chunks as well. That change brings behavior back to something closer to what older kernels did.

Patch #2 is a simple tweak which keeps realtime interrupt handlers from driving the memory allocation code too hard. Again, this is a reversion to behavior seen back in the 2.6.30 days.

The third patch is a bit more subtle. Direct reclaim will, if it is successful, result in the creation of I/O operations to write dirty pages to their backing store. There are limits to the number of block I/O operations which can be outstanding, though; once that limit is hit the underlying device is said to be "congested" and the task performing reclaim is forced to wait until things clear out a bit. This "congestion wait" keeps the system from filling up with pending I/O operations and serves to throttle processes performing memory allocations.

As it happens, there are actually two "wait for congestion" queues - one each for synchronous and asynchronous requests. "Synchronous" requests are those for which a process is actively waiting - read requests, usually - while asynchronous requests are those which do not have active waiters. In current kernels, direct reclaim waits on the asynchronous queue, while older kernels used the synchronous queue instead. Moving back to the synchronous queue makes a number of problems go away, but Mel sees that fix as being workload-specific. Instead, he has changed the direct reclaim code to make it wait for congestion to clear on both queues.

Why does this help? It seems to be a matter of letting kswapd get its job done. Kswapd, too, must wait when queues become congested; if direct reclaimers are frequently filling the I/O queues, kswapd will stall more often. It turns out that better results are had if kswapd is allowed to run for longer periods of time. Making direct reclaimers wait until both queues have cleared allows kswapd to get some real work done once it gets going. That is good for the creation of high-order chunks and the performance of the system in general.

Patch #4 also relates to kswapd's duty cycle. Kswapd will stop working and go to sleep once it decides that it has done enough; one definition of "enough" is when the amount of free memory reaches an upper watermark value. But if kswapd is running, chances are good that there is unmet demand for memory in the system; in that situation, the amount of free memory may not stay above the high watermark for very long. Mel's patch has kswapd start with a catnap rather than a real sleep; after 0.1 sec., kswapd wakes back up and reassesses the situation. If the amount of free memory has fallen below the high watermark in that time, kswapd goes back to work; otherwise it goes to sleep for real. In this way, kswapd will continue to work to free memory if the system is consuming it quickly.

The final patch touches on another aspect of waiting for congestion. When block devices become congested, kswapd waits for things to clear. But, Mel notes, that may not be the right thing to do in all situations:

In the original version of the patch, kswapd would become increasingly resistant to waiting for congestion as the situation got worse. Motohiro Kosaki suggested an alternative approach, though, wherein kswapd simply refuses to wait as long as the high watermark is not reached, and Mel adopted it.

Mel's patch posting includes a fair amount of information on how he has tested it and what the results are. With the patch set applied, allocation failures are fewer, and system throughput improves as well. The sad truth about memory management patches, though, is that a change which improves one workload may worsen another. So these changes really need some widespread testing, especially since there is some interest in getting them into 2.6.32.

Comments (none posted)

Receive packet steering

From the operating system's point of view, distributing the work of outgoing data across CPUs is relatively straightforward. The processes generating data will naturally spread out across the system, so the networking stack does not need to think much about it, especially now that multiple transmit queues are supported. Incoming data is harder to distribute, though, because it is coming from a single source. Some network interfaces can help with the distribution of incoming packets; they have multiple receive queues and multiple interrupt lines. Others, though, are equipped with a single queue, meaning that the driver for that hardware must deal with all incoming packets in a single, serialized stream. Parallelizing such a stream requires some intelligence on the part of the host operating system.

Tom's patch provides that intelligence by hooking into the receive path - netif_rx() and netif_receive_skb() - right when the driver passes a packet into the networking subsystem. At that point, it creates a hash from the relevant protocol data (IP addresses and port numbers, in particular) and uses it to pick a CPU; the packet is then enqueued for the target CPU's attention. By default, any CPU on the system is fair game for network processing, but the list of target CPUs for any given interface can be configured explicitly by the administrator if need be.

The code is relatively simple, but it succeeds in distributing the load of receive processing across the system. The use of the hash is important: it ensures that packets for the same stream of data end up on the same processor, increasing cache locality (and, thus, performance). This scheme is also nice in that it requires no driver changes at all, so it can be deployed quickly and with minimal disruption.

There is one place where drivers can help, though. The calculation of the hash requires accessing data from the packet header. That access will necessarily involve one or more cache misses on the CPU running the steering code - that data was just put there by the network interface and thus cannot be in any CPU's cache. Once the packet has been passed over to the CPU which will be doing the real work, that cache miss overhead is likely to be incurred again. Unnecessary cache misses are the bane of high-speed network processing; quite a bit of work has been done to eliminate them wherever possible. Adding a new cache miss for every packet in the steering code would be counterproductive.

It turns out that a number of network interfaces can, themselves, calculate a hash value for incoming packets. That processing comes for free, and it could eliminate the need to calculate that hash (and suffer the overhead of accessing the data) on the dispatching processor. To take advantage of this capability, the RPS patch adds a new rxhash field to the sk_buff (SKB) structure. Drivers which are able to obtain hash values from the hardware can place them in the SKB; the network stack will then skip the calculation of its own hash value. That should keep the packet's data out of the dispatching CPU's cache entirely, speeding processing.

How well does this work? The patch included some benchmark results using the netperf tool. An 8-core server with a tg3-based network interface went from 90,000 transactions per second to 285,000; an e1000-based adapter on the same system went from 90,000 to 292,000. Similar results are obtained for nForce and bnx2x chipsets on 16-core servers. It would appear that this patch does succeed in making networking processing faster on multi-core systems.

The patch, incidentally, comes from Google, which has a bit of experience with network processing. It has, evidently, been running on Google's production servers for a while. So the RPS patch is, hopefully, an early component of what will be a broad stream of contributions from Google as that company tries to work more closely with the mainline. It seems like a good start.

Comments (6 posted)

SamyGO: replacing television firmware

While it is quite common for consumer electronics—TVs, DVRs, and the like—to be running Linux these days, it is less common to see projects geared towards replacing and upgrading the Linux firmware in that class of devices. But that is exactly what the SamyGO project is doing for Samsung televisions. By using the source provided by Samsung, along with quite a bit of ingenuity, SamyGO allows users to telnet into their television—an amusing concept—but also to enable functionality beyond that which ships with the device.

The SamyGO wiki lists several modifications that can be made to the TV firmware. One of the main modifications seems to be enabling NFS or SMB/CIFS support so that media files from servers on the network can be played. The TVs already support getting media from the local network using Digital Living Network Alliance (DLNA) protocols, but there are restrictions on the audio and video formats and some playback functionality (pause, forward, rewind) depending on the DLNA server. By using NFS or CIFS, all of the formats and features available for USB-based playback are also available across the network.

Obviously, these are fairly high-end TVs, with both Ethernet connectivity and USB ports. The devices "supported" by SamyGO are LCD models in the LE-32-55Bxxx series and LED models from the UE-xx-B70xx series. The USB ports are available for viewing/playing additional media or for games. Using the "Games" menu with programs stored on a USB stick is one of the ways to run programs on the TV.

The USB ports are also used for a Samsung-branded WiFi "dongle" that owners can buy to avoid the wiring hassle of Ethernet. But, Linux supports far more wireless devices than just the Samsung devices, so SamyGO developers are working to enable others as well. In fact, the Ralink rt73 and rt2870 drivers have been modified in the kernel source supplied by Samsung to remove many additional device IDs, so that only the Samsung devices will work. There are now drivers available without that restriction.

The early efforts have been to get telnet working so that the TV filesystem could be explored. This is done by patching the firmware binaries provided by Samsung and then using the TV's firmware upgrade mechanism to install them on the device. The aptly named "Warning : Read Me First or Brick Your TV!" message in the SamyGO forum outlines the dangers of upgrading the firmware. For those that just want to try this all out, without upgrading any firmware, a safer method is also described, which masquerades as a game on a USB stick to enable telnet.

The kernel is 2.6.18-based with the addition of Samsung's Robust FAT File System (RFS), which is a filesystem for NAND flash devices. As the name would indicate, it is also FAT compatible. It is not in the mainline, however, nor have the SamyGO developers gotten it working for desktop distributions. For that reason, they have resorted to binary patching of the firmware.

Samsung has also released RFS source, along with a Linux porting guide that should be helpful in those efforts. Once RFS can be built for recent kernels, or a utility to create RFS images is made, developers will be able to build their own firmware images for these TVs. [ Update: see the comments below, there is no source RFS release. ]

The kernel source is available, but the project has not yet released any kernels built from it. The Ralink drivers were rebuilt after modifying the device IDs, though, so they can be inserted into the system. The kernel itself has been patched, adding OMAP architecture and sound support among other things, but there has been no mention of binary drivers on the forum, so it should be possible to build the released kernel—or something more recent.

So far, Samsung doesn't seem to have reacted to the project, either positively or negatively. Some concern has been expressed in the forum that working around the WiFi restrictions might raise the company's ire. But one would guess that the number of folks willing to risk bricking an expensive TV in order to use a cheaper WiFi dongle is relatively small—likely to go unnoticed by Samsung.

In the meantime, if the SamyGO hackers add other functionality that might be interesting to customers—there has been talk of web browsers for example—Samsung might just adopt it themselves. Either way, the code is out there for those who might want to give it a try.

Comments (46 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.32-rc7 . (November 13, 2009)

Build system

• Luis R. Rodriguez: scripts: add gen-linux-conf for remote kernel configuration . (November 13, 2009)

Core kernel code

• Stefani Seibold: kfifo: new API v0.6 . (November 16, 2009)

• Tejun Heo: workqueue: prepare for concurrency managed workqueue . (November 17, 2009)

• Michel Lespinasse: futex: add FUTEX_SET_WAIT operation . (November 17, 2009)

• Bharata B Rao: CFS Hard limits - v4 . (November 17, 2009)

• Stijn Devriendt: Initial prototype version of sched_wait_block . (November 17, 2009)

Development tools

• Hitoshi Mitake: perf bench: Add new subsystem "mem" and new suite "memcpy" . (November 13, 2009)

• Hitoshi Mitake: Measuring term of acquiring spinlock . (November 16, 2009)

• Mathieu Desnoyers: [RELEASE] LTTng 0.173, ltt-control 0.76, lttv 0.12.22, trace format 2.5 . (November 18, 2009)

• Masami Hiramatsu: tracepoint: Add signal events . (November 18, 2009)

Device drivers

• Samu Onkalo: Driver for ami305 magnetometer . (November 12, 2009)

• Luis R. Rodriguez: Stable compat-wireless 2.6.32-rc7 released . (November 13, 2009)

• Meng, Chen: GPIO driver for ST Microelectronics ConneXt (STA2X11/STA2X10), PCIe I/O Hub . (November 13, 2009)

• Rafael J. Wysocki: PCI run-time PM support . (November 16, 2009)

• Wolfram Sang: net/can: add driver for mscan family & mpc52xx_mscan . (November 16, 2009)

• Christoph Hellwig: libata: add TRIM support . (November 16, 2009)

• Arjan van de Ven: libata: Add ALPM power state accounting to the AHCI driver . (November 16, 2009)

• Wan ZongShun: ARM: Add spi controller driver support for NUC900 . (November 17, 2009)

• Matthew Garrett: [RFC] Add support for uevents on block device idle changes . (November 18, 2009)

• Laurent Pinchart: V4L core cleanups . (November 18, 2009)

Filesystems and block I/O

• Christoph Hellwig: XFS status update for October 2009 . (November 12, 2009)

• Vivek Goyal: Block IO Controller V2 . (November 13, 2009)

• Corrado Zoccolo: cfq: autotuning support . (November 17, 2009)

• Jan Blunck: BKL pushdown from do_new_mount() to the filesystems (v3) . (November 18, 2009)

Janitorial

• Arnd Bergmann: compat-ioctl.c diet . (November 16, 2009)

Memory management

• Andi Kleen: Allow memory hotplug and hibernation in the same kernel . (November 13, 2009)

• KAMEZAWA Hiroyuki: speculative page fault . (November 16, 2009)

• Lee Schermerhorn: Numa: Use Generic Per-cpu Variables for numa_*_id() . (November 16, 2009)

• Andrea Arcangeli: Transparent Hugepage support #1 . (November 16, 2009)

• Mel Gorman: Reduce GFP_ATOMIC allocation failures, candidate fix V3 . (November 17, 2009)

Architecture-specific

• Martin Schwidefsky: s390 patches for the next merge window (2.6.33) . (November 16, 2009)

• Wu Zhangjin: ftrace for MIPS . (November 16, 2009)

• Tony Lindgren: New omap boards and rx51/n900 updates for 2.6.33 . (November 18, 2009)

• Marek Szyprowski: Add Samsung S5PC110 SoC support . (November 18, 2009)

Virtualization and containers

• Avi Kivity: KVM updates for the 2.6.33 merge window (batch 1/2) . (November 16, 2009)

• Adam Litke: virtio: Add memory statistics reporting to the balloon driver (V2) . (November 18, 2009)

Benchmarks and bugs

• Rafael J. Wysocki: 2.6.32-rc7-git1: Reported regressions from 2.6.31 . (November 17, 2009)

• Rafael J. Wysocki: 2.6.32-rc7-git1: Reported regressions 2.6.30 -> 2.6.31 . (November 17, 2009)

Miscellaneous

• Jon Masters: ANNOUNCE: LKML Podcast . (November 13, 2009)

• André Goddard Rosa: introduce skip_spaces(), reducing code size plus some clean-ups . (November 16, 2009)

• Jozsef Kadlecsik: ipset 4.1 released . (November 16, 2009)

Page editor: Jonathan Corbet

Distributions

News and Editorials

openSUSE 11.2

Right behind Mandriva and Ubuntu, openSUSE 11.2 arrived as scheduled after almost a year of development. This incremental version increase has enough new features to warrant a closer look. Sometimes adding so many new features between minor versions can backfire. Is that what happened this time with openSUSE 11.2? Or are we seeing some early effects of the newly sanctioned community contributions?

So What's New?

The first thing anyone might notice is the new theme. Developed by KDE's Nuno Pinheiro, it has a very "Airy" look and feel to it. I say theme, but I really mean the background and application graphics because KDE still hasn't offered a decent window decoration for KDE 4 and openSUSE 11.2 is using KDE's default. I don't want to spend too much time on appearances, but while most have found the desktop unattractive, I have seen a few positive remarks for the olive and peridot greens.

Speaking of KDE 4, KnetworkManager recently received an overhaul and the bulk of the work was done by openSUSE developer Will Stephenson, with contributions from Fedora, Kubuntu, and other developers. This rewrite has been in the works for quite a while and it is still not fully completed now. Struggles with the interface plagued development and improvements sit at the top of the todo list for future releases. As it is, it's built with QT4 and fits in with the KDE Plasma desktop adequately. The current interface hides most available wireless connections detected from the default view, instead only listing those previously used. Other little goodies include setting custom icons on a per connection basis, configurable tooltips, one-click connects, and the option of forcing password input on each connect for users.

The YaST Control Center has been ported to QT4 for the KDE desktop as well. It still resembles the GTK version for GNOME quite a bit and is said to be more consistent with KDE System Settings, but somehow it doesn't feel very polished. The fonts are atrocious, and I had to configure some of my hardware (scanner and TV card) more than once for it to actually take effect. This is very uncharacteristic for openSUSE and I have to wonder what happened. Also new this time is WebYaST, an easy-to-use remote management and administration interface.

The YaST2 Software Manager also saw a few tweaks for this release. The GUI features a new View tab that's actually a drop down menu containing various package viewing choices. Underneath, Zypper now functions a bit faster and includes the option of downloading only. Also new for this release is openSUSE's equivalent to a dist-upgrade. A live update can be performed from the GUI or command line, although reports have stated that the command line route is more reliable at this point.

The Firefox KDE integration has received quite a bit of publicity during the development cycle. Basically, that consists of setting Firefox as the default KDE browser and calling KDE applications when a trigger is clicked. For example, Firefox will open Okular when a PDF link is clicked. This ties in with file dialogs and application selection screens, mimetype and protocol handling (such as mailto), and proxy settings. It also uses KDE icons and widgets and can add RSS feeds to Akregator. Several other distributions have switched their KDE default browser to Firefox recently as well, but openSUSE is the only one to try and integrate it so completely. More features are being planned in this area for future releases. As in many distributions, "check for updates" is disabled, but this is the only time I can really see a good reason why.

Some changes can be seen in the installer as well. KDE has been returned to the default desktop choice of the install DVD. GNOME is listed first, but KDE is ticked. Apparently this was done to lessen the number of choices a new user might have to make during the install process. Ext4 is the new default filesystem and Btrfs is available for particularly brave souls. Full disk encryption is now available for the security minded. And for netbook users, the live CDs can be copied to and booted from USB memory sticks.

At the desktop, KDE's Strigi and Nepomuk are disabled by default to lessen system requirements and improve performance. In contrast to KDE, the new GNOME theme is receiving quite a bit of praise. Pidgin remains the included instant messenger for GNOME instead of migrating to Empathy like some others, and new microblogging clients were added for the two major desktops, Gwibber and Choqok. The primary desktops are KDE 4.3.1, GNOME 2.28, Xfce 4.6.1, and Enlightenment 1.0. OpenOffice.org has been upgraded to 3.1.1, Firefox to 3.5.4, and GIMP 2.6.7. Under the hood is Linux kernel 2.6.31.5, Xorg X Server 1.6.5, and GCC 4.4.1.

Issues Encountered

openSUSE has traditionally been a very polished and professional system; rock solid underneath with pretty GUIs on top. However, 11.2 has slipped some. After installation, the fonts were very ugly and distracting. I've seen complaints in the past about openSUSE fonts, but I've never personally been affected. But with 11.2, my desktop was almost unusable until I tweaked the fonts. However, try as I might, I still could not bring openSUSE 11.2 fonts up to par with my other systems.

It's not uncommon for sound to only emit from my two rear speakers in some Linux distributions. I don't consider this a problem, really, since the front is usually a mirror of the rear with my card, but when sound only comes from one of the rear speakers, then I've got to say that something is wrong somewhere. My sound card is detected with similar output as in other distributions, the same ALSA modules are used, and the mixer channels were thoroughly reviewed. So, at this point, it's a mystery why this old and usually well supported card went oblong in this release.

I've also had real issues with Akregator in KDE. Admittedly, it's always been unstable in KDE 4, but I've been experiencing more frequent crashes in openSUSE. It seems to crash, taking the rest of Kontact with it, five or six times a day. Sometimes it loses all the articles pulled in previously making it quite a chore to continue using it.

KDE settings, in general, have been acting strangely too. For example, losing settings between openings, settings that never take or change the behavior, and settings that won't change - they appear to change, but don't take effect and when I check back, the original settings are depicted in the input box as if I'd never touched them.

As far as performance, which seems to be a hot topic this year, 11.2 does seem to boot faster, but I'm not seeing anything impressive in its KDE. Many issues found might have been lessened if KDE had been updated to one of the newer releases, because 4.3.1 (even with some 4.3.2 backports) still has many performance and functionality issues.

Conclusions

Overall this version of openSUSE acts more like a point-0 release or even a release candidate. Everything feels rough around the edges and as though lots more work is needed. There's no dispute that openSUSE developers are the most aggressive between minor version releases, but this is the most dramatic effect I've witnessed from them. Polish and excellence have always been trademarks of openSUSE, so much so that I've come to expect only that. So, it's shocking to have seen an openSUSE released in such rough condition.

Having said that, I still look forward to 11.3 and have confidence that it will be up to openSUSE's usual standards. In Linux, developers are always fighting "the damned if you do and damned if you don't" paradox. If they don't release when users expect, then they risk losing lots of momentum, much like PCLinuxOS experienced in 2008 and early 2009. Or on the other hand, if they release on time, even though they know there are issues, they risk the bad press and decreased user confidence like that seen with the latest release (or two) of Ubuntu.

I would like to give openSUSE the benefit of the doubt but my best recommendation is for folks to wait for the next release, especially if they are KDE users. GTK/GNOME users might have better luck. However, overall, 11.2 isn't the best example of its work and we should wait for the next release so that it can sand down the rough edges.

Comments (13 posted)

New Releases

Fedora 12 released

Full Story (comments: 20)

Fedora 12 LXDE Spin withdrawn (for now)

Full Story (comments: 3)

Knoppix 6.2 arrives (The H Open)

Comments (none posted)

openSUSE 11.2 is out

Full Story (comments: 6)

Ubuntu Studio 9.10 Karmic Koala

Full Story (comments: none)

Vector Linux 6.0 Kde Classic

Comments (none posted)

XtreemOS 2.0 released

Comments (3 posted)

Announcing Fedora Electronic Lab 12

Full Story (comments: none)

Fedora-ARM 12

Full Story (comments: none)

Distribution News

Debian GNU/Linux

Bits from the FTPMaster meeting

Full Story (comments: none)

Fedora

RPM Fusion repositories for Fedora 12 now available

Full Story (comments: none)

Mandriva Linux

Mandriva 2010 Spring development has begun

Comments (none posted)

SUSE Linux and openSUSE

openSUSE board meetings to be public

Comments (none posted)

What does the openSUSE Board do?

Comments (none posted)

Ubuntu family

Minutes from the Ubuntu Technical Board meeting

Full Story (comments: none)

Developer Membership Board public meeting: 2009-11-24 15:00 UTC

Full Story (comments: none)

Distribution Newsletters

DistroWatch Weekly, Issue 329

Comments (none posted)

Fedora Weekly News 202

Full Story (comments: none)

OpenSUSE Weekly News/97

Comments (none posted)

Ubuntu Weekly Newsletter #168

Full Story (comments: none)

Distribution reviews

Fedora 12 Linux Tackles Virtualization (InternetNews.com)

Comments (none posted)

New Ubuntu OS Features Create Good Karma (LinuxInsider)

Comments (none posted)

Fedora 11 is almost awesome (Rubenerd Blog)

Comments (none posted)

Page editor: Rebecca Sobol

Development

Officeshots: making ODF truly interoperable

Complex file formats, such as those used for office documents, inevitably lead to differences in interpretation by application developers. If a user sends a document to someone else who views it in a different application or version, chances are that the output shows some subtle differences or, by bad luck, that the formatting is completely munged. For people that give presentations regularly, this is a constant nightmare: they have to hope that the office application on the conference laptop is able to show the presentation without mangling the slides. These problems are not tied to proprietary file formats: open standards such as ODF (Open Document Format) also have interoperability issues.

A web service, Officeshots, was recently launched to remedy this problem. The project is in public beta and users can register for free to upload their ODF documents. The web site then generates the output of the document using various office applications, which enables the user to check for interoperability issues. The launch of the public beta took place during the second ODF plugfest in Orvieto, Italy on November 2nd and 3rd. A lot of vendors and developers using ODF in their software gathered in Orvieto, such as IBM, Google, OpenOffice.org, Novell, KOffice, AbiWord, and Microsoft.

Officeshots is a project by NOiV (Netherlands in Open Connection), a Dutch government program to promote the use of open standards and open source, in collaboration with the OpenDoc Society and NLnet Foundation, a Dutch non-profit organization that financially supports contributors to an open information society. LWN talked to Sander Marechal, who developed the bulk of the Officeshots code and is the project leader. He owns Lone Wolves, a small non-profit open source development company based in The Netherlands.

In June 2008, Sander was invited by Michiel Leenaars (of OpenDoc Society and NLnet) to give a talk at Sun Microsystems in Hamburg about another Lone Wolves project, ODF-XSLT. Sander drove to Hamburg with Michiel and the two talked about their mutual interests. That car drive started the ball rolling:

As the director strategy for NLnet and member of the OpenDoc Society, Michiel Leenaars had a lot of contacts with office software vendors, both open source and proprietary, including Sun, Novell, and Google. He got them interested in the Officeshots project and talked with other developers. During the recent plugfest, the project even got some Microsoft Office licenses as a gift.

Document factories

The Officeshots web site has a very simple user interface: the user submits a document, and the site delivers a PDF export, a screenshot, or a round-trip ODF file produced by the applications the user selects. A round-trip ODF means that an application opens the ODF document and then saves it again. So if the user chooses round-trip ODF as the output format, he gets an ODF document back. What's the point of this? Sander explains the importance:

Currently supported applications are different versions of AbiWord, Gnumeric, EuroOffice, Go-oo, Corel WordPerfect, KOffice, OpenOffice.org, StarOffice, TextMaker, and PlanMaker, in Linux/BSD as well as in Windows. Supported document formats are Open Document texts, spreadsheets, and presentations. The user can also create a public gallery to show conversion errors to others. A simple test using some ODF files in the example content that comes with Ubuntu definitely shows interoperability issues.

Under the hood, the user's uploaded file gets distributed to rendering servers hosted by vendors and the community. The Officeshots project calls each server that is producing output a factory. Most of the factories are run by the Officeshots project, which has a couple of virtual machines running on the Xen hypervisor to guarantee that the service is always able to produce some output.

Other factories are run by people from AbiWord, Gnumeric, and other projects, and a couple are run by volunteers. Sander highlights the first two projects:

The Officeshots project has a list of factories (currently 14) and a list of active factories (at the moment of writing 5). At this moment, the project is waiting for a new server that will host virtual machines with various Linux distributions, as well as Windows with Microsoft Office.

Contribute to Officeshots

The Officeshots project not only provides the free online web service, but also provides the code for the underlying framework (Affero GPLv3-licensed). While Sander admits that there haven't been that much external code contributions yet, he points out that there are a lot of other means by which one can contribute to the project: people can run a factory, translate Officeshots to their language, or donate hardware or software licenses.

People who want to run their own factory should contact Officeshots and consult the manual. The code can be downloaded from the Officeshots Subversion repository. The manual also explains how to implement a backend for a not-yet-supported application. The simplest way is if the application offers command-line conversion functionality. This led at least one team to implement this feature into their office application, Sander remarks:

But actually, one doesn't have to go that far to give a helping hand to the project's mission: if a user detects interoperability issues thanks to Officeshots and reports the problem to the relevant office applications, then the project has succeeded.

New functionality

The Officeshots developers have a couple of ideas to implement in the future. Of course they will add new backends. For example, Sander has already written a backend for an older version of Microsoft Word using the Sun ODF plugin, so when the Windows virtual machines are ready, a new Microsoft Office backend will be one of the possibilities. They will also add backends for the office viewer of Symbian S60 smartphones.

But other than new backends, the project has some additional new features in the pipeline. One notable feature is an ODF diff tool. "We are looking at a commercial tool by DeltaXML.com, which is very useful because normal XML diffs generate too much noise," Sander explains. "Using it shows clearly that Microsoft Office replaces formulas and charts when saving." Another feature in the pipeline is a service running the ODF Validator against an uploaded document. "But we are also looking into ODF validators that can generate messages a normal human being can understand, instead of throwing cryptic XML exceptions like most XML validators do." Another plan is to integrate the complete ODF 1.0 test suite into Officeshots. A factory could then be periodically offered a set of hundreds of documents to automate parts of the test suite.

Privacy

The project is also seeking some ways to protect the user's privacy. If users upload documents with sensitive information, they should know that Officeshots and the factories can read this information. At the moment, the project asks their users to have trust in the Officeshots project and third-party factories. Sander adds:

Because the anonymizer will run on the Officeshots server, the factories receive the modified text, so that users don't have to trust the third-party factories. But it still asks users to trust the people of the Officeshots server which runs the code that anonymizes the uploaded document. Concerned people can install itools locally (it is packaged in a couple of Linux distributions) and use the iodf-greek.py script (added in itools 0.60.3) to anonymize their documents before uploading them. For very sensitive documents, it is possible to run a local copy of the Officeshots web service and backends, but that takes time to install and configure.

Conclusion

The Officeshots web site is a handy service for users that are evaluating which office application to migrate to. Thanks to the project, they don't have to install each application locally to check for interoperability issues. With the web service, they can easily check if each application does what it says. Also consider template designers and people creating documents for public release. With Officeshots, they can easily check if their documents work everywhere. Last but not least, it is also a helpful tool for the office software vendors who can spot errors in their ODF support. In these ways, the Officeshots project should accelerate interoperability in the office software market.

Comments (1 posted)

System Applications

Audio Projects

JACK 0.118.0 + D-Bus released

Full Story (comments: none)

PulseAudio 0.9.20 released

Comments (none posted)

Database Software

Elixir 0.7.1 released

Full Story (comments: none)

MySQL Community Server 5.1.41 has been released

Full Story (comments: none)

PostgreSQL Weekly News

Full Story (comments: none)

Device Drivers

libshcodecs 0.9.8 released

Full Story (comments: none)

Mail Software

Exim 4.70 released

Full Story (comments: none)

Networking Tools

gevent 0.11.1 released

Full Story (comments: none)

Web Site Development

Midgard2 9.09 "Mjolnir" released

Full Story (comments: none)

nginx 0.7.64 announced

Comments (none posted)

Desktop Applications

Audio Applications

Ardour 2.8.4 released

Comments (none posted)

Desktop Environments

Erlandsen: What We Talk About When We Talk About Zeitgeist

Comments (3 posted)

GNOME Software Announcements

• atk 1.29.2 (bug fix)
• at-spi 1.29.2 (bug fixes and translation work)
• AT-SPI2 0.1.2 (new features, bug fixes and code cleanup)
• Brasero 2.29.2 (new features, bug fixes and translation work)
• easygconf 0.02 (alpha release)
• Empathy 2.29.2 (new features, bug fixes and translation work)
• EOG Plugins 2.29.2 (translation work)
• Evince 2.29.2 (new features, bug fixes and translation work)
• Eye of GNOME 2.29.2 (new features, bug fixes and translation work)
• File Roller 2.29.1 (bug fixes and translation work)
• GCalctool 5.29.2 (new features and translation work)
• GDM2 2.29.0 (new features, bug fixes and documentation work)
• gedit 2.29.2 (unspecified)
• GNOME games 2.29.2 (bug fixes and translation work)
• GOK 2.29.2 (bug fixes and translation work)
• goobox 2.1.1 (new features and documentation work)
• gtkaml 0.2.9.1 (bug fixes)
• librep 0.90.3 (new features and documentation work)
• LibUnique 1.1.2 (rc) (new features, bug fixes and documentation work)
• LibUnique 1.1.6 (rc) (build fix)
• mousetweaks 2.29.2 (bug fixes and translation work)
• Orca 2.29.2 (bug fixes and translation work)
• PDF Mod 0.8 (new features, bug fixes and translation work)
• Sawfish 1.5.3 (bug fixes)
• Tomboy 1.0.1 (bug fixes and translation work)
• Tomboy 1.1.0 (bug fixes and translation work)
• tracker 0.7.7 (new features)

Comments (none posted)

KDE Software Announcements

• dropbox-servicemenu-kde 0.1 (initial release)
• KDE4 Twitter Notifications 1.0 (initial release)
• Konversation 1.2.1 for KDE 4 (new features and bug fixes)

Comments (none posted)

Xorg Software Announcements

• xinit 1.2.0 (new features, bug fixes and code cleanup)

Comments (none posted)

Electronics

XCircuit 3.6.163 released

Comments (none posted)

Interoperability

Wine 1.1.33 announced

Comments (5 posted)

Mail Clients

The notmuch mail client

Comments (53 posted)

Sylpheed 3.0beta2 (development) released

Comments (none posted)

Multimedia

Amarok 2.2.1 released

Full Story (comments: none)

Music Applications

BigBand - compose music for small ensembles

Full Story (comments: none)

FluidSynth 1.1.0 released

Full Story (comments: none)

Office Applications

YaMA 1.6.4 released

Full Story (comments: none)

Video Applications

De Icaza: The future of Moonlight

Comments (49 posted)

Web Browsers

Firefox 3.6 Beta 3 is available

Full Story (comments: none)

Languages and Tools

Caml

Caml Weekly News

Full Story (comments: none)

Java

IcedTea7 1.12 (Milestone 5) released

Full Story (comments: none)

Perl

Parrot 1.8.0 released

Full Story (comments: none)

www.perl.org redesigned (use Perl)

Comments (1 posted)

Python

ftputil 2.4.2 released

Full Story (comments: none)

Urwid 0.9.9 released

Full Story (comments: none)

Python-URL! - weekly Python news and links

Full Story (comments: none)

Tcl/Tk

Tcl-URL! - weekly Tcl news and links

Full Story (comments: none)

IDEs

Pydev 1.5.1 released

Full Story (comments: none)

Version Control

GIT 1.6.5.3 released

Full Story (comments: none)

Mercurial 1.4 released

Full Story (comments: none)

Miscellaneous

GNU patch 2.6 released

Full Story (comments: none)

Page editor: Forrest Cook

Announcements

Commercial announcements

Android Dev Phone 2 available

Comments (16 posted)

Astaro offers free firewall for SMBs

Full Story (comments: none)

Chumby Guts!--New Product from The Maker Shed Store

Full Story (comments: none)

MontaVista's Montebello Stack Incorporated in Dell Latitude ON

Full Story (comments: none)

PyIMSL Studio 1.5 now available at no charge

Full Story (comments: none)

Rev 4.0 Released

Full Story (comments: none)

Legal Announcements

EFF obtains behind-the-scenes FISA records

Full Story (comments: none)

The Open Web Foundation Agreement

Comments (1 posted)

Articles of interest

Microsoft Patents Sudo?!! (Groklaw)

Comments (21 posted)

Lenovo returns to the Linux desktop (ComputerWorld)

Comments (4 posted)

New Linux-Based PCs Encourage Seniors To Learn The Internet (redOrbit)

Comments (21 posted)

Nicaragua Builds An Innovative Agricultural Information System Using Open Source Software (Linux Journal)

Comments (1 posted)

The Future of Linux is Google (PCWorld)

Comments (60 posted)

New Books

Hello, Android, Second Edition--New from Pragmatic Bookshelf

Full Story (comments: none)

Resources

FSFE Newsletter

Full Story (comments: none)

Linux Foundation Newsletter

Full Story (comments: none)

Surveys

Poll on PyPI rating system

Full Story (comments: none)

Education and Certification

LPI Affiliate promotes Linux education in Middle East

Full Story (comments: none)

UKUUG and O'Reilly announce Training Days

Full Story (comments: none)

Upcoming Events

PyCon talks announced

Full Story (comments: none)

Events: November 26, 2009 to January 25, 2010

If your event does not appear here, please tell us about it.

Web sites

EFF: International Activists Launch New Website to Gather and Share Copyright Knowledge

Full Story (comments: 3)

planet LAD launched

Full Story (comments: none)

Page editor: Forrest Cook