LWN.net Weekly Edition for January 17, 2008

Making code reviews easier with Review Board

Reviewing code is a thankless, but very important, task for any software project. For free software projects, the "many eyes make all bugs shallow" aphorism only works if the eyes actually focus on the code in question. Review Board is a web-based application that helps reviewers examine the code, while making it easier for a developer to track those reviews.

Borne out of frustration with the process of code reviews at VMware, Review Board has made a great deal of progress since being released last May. The idea behind it is to centralize all of the pieces that need to come together for a review: code diffs, screenshots of UI functionality, comments by other developers, etc. On many projects, reviews are handled by email, but that can be difficult to use; various pieces of the puzzle are spread around in multiple messages and locations.

Often a reviewer needs to see more context than a simple email diff provides or wants to comment on a related section of code that is not contained in the diff; each requires a reviewer to do more work. In a complicated set of changes, ensuring that the developer and any other reviewers can follow what code the comments pertain to can also be difficult. It is these kinds of problems that Review Board is meant to solve.

Review Board presents a side-by-side diff view, shown at right, with lots of extras, many of which will be familiar to users of other graphical diff tools. Changed lines are highlighted in different colors based on whether they are additions, deletions, or changes. Changes on a particular line are highlighted in a slightly darker color so that they can be distinguished more easily as well. The numbered tabs along the left edge provide a link to a reviewer's comments about that section of the code. This is where Review Board shows that it is much more than just a diff viewer.

Using AJAX techniques, Review Board allows a reviewer to interact very naturally with the code. They can highlight a certain section, which will pop up a text widget that records comments associated with that section of code. When other reviewers or the developer read those comments, the code snippet is included, with a link back to the code in the diff view. Each of these comments can then be commented upon which allows for a conversation about the code to develop.

It is not just code that can be annotated; screenshots of application functionality or bugs can be attached to reviews, as well. Sections of the screenshot can be highlighted and commented upon, as shown at left. This feature is an excellent example of where a web-based tool can shine; doing the same task in text-based email would be painful. Not all projects need it, but those that do will find it quite useful as anyone who has spent time trying to describe a UI problem in email will attest.

Inter-diffs is another useful feature that Review Board provides. Often in the code review process, several revisions of the original patch are made. It can be tedious to wade through a large diff, most of which has been uncontroversial (or resolved earlier) to get to the changes in the area of interest. Review Board has the ability to see changes between any two revisions of the patch, which should reduce much of the hassle.

Another thing that Review Board does is to assist in managing code reviews. When a developer posts something for review, various reviewers can be notified via email. Review Board keeps track of that information, presenting users with a "dashboard" view of their pending reviews, both those they submitted and those that others have asked them to do. This high-level overview is the first screen the user sees when they log on to the system, shown at right. This makes keeping track of work that needs to be done – or who to prod to get a review moving again – much easier.

Currently, Review Board best supports the Subversion and Perforce version control systems (VCS), but support for others, including distributed VCS Mercurial and git, are being actively developed and are usable in their current states. Released under an MIT license, Review Board is written in Python, using the Django web framework. Development is hosted at Google Code; the developers, unsurprisingly, uses the software for internal code reviews.

Other systems to assist in the code review process do exist. Codestriker is a Perl based web application that has similar aspirations to Review Board. Also of interest is Python founder Guido van Rossum's first project at Google: a code review system he calls "Mondrian". It is closely tied to Google proprietary code, though, so it seems unlikely to be released as free software – though it might make an appearance as a tool for Google Code projects to use.

Code reviews are very powerful, but generally painful to perform; any tool that claims that "Code reviews are fun again! ...almost.", as Review Board does, will be welcomed by many. It will be interesting to see whether a code review tracker becomes a standard part of newer free software projects. Over the last few years, we have seen the rise of distributed VCS, bug trackers, and wikis to assist in distributed development. Will Review Board – or something like it – be the next tool to be added?

Comments (9 posted)

SAMP?

Sun is making the right noises about how this deal will work. There is no talk of taking MySQL proprietary or changing its license. MySQL will continue to be supported on all platforms, and not just Solaris. A series of grants will be made to help university researchers advance the state of the art in database management systems. There is a lot of talk about continuing to support "the community," though details are (perhaps necessarily) scarce. CEO Jonathan Schwartz says that Sun will be working to improve "the rest of the LAMP" stack, though he says nothing about the "L" (for Linux) part.

Chances are that this deal will be a good thing for MySQL users. Sun is clearly making MySQL an important part of its overall strategy (in these days, one does not toss $1 billion toward unimportant objectives) and can be expected to continue - or accelerate - development of the system. Sun's free software orientation is strong enough that the chances of parts or all of MySQL going proprietary seem small. Indeed, nothing in Sun's releases says anything about MySQL's commercial licensing business; the emphasis appears to be strongly on support and services. So MySQL might just become even more open than it is now.

Sun appears to be positioning itself to compete strongly with Oracle. Both companies are working hard to be able to offer the entire software stack to their customers. So Oracle's push into the Linux distribution business and Sun's database venture are both aimed at having the same story for their sales staff to tell: we, in some way, own and control all of the software you are looking to run. No problems with incompatibilities, finger-pointing, etc. As an added bonus, Sun will happily sell you the hardware you need too. Do expect an increase in efforts aimed at moving MySQL users away from the (Oracle-owned) InnoDB engine, though.

For Sun to sell that story, though, it will to have continue to push Solaris hard as an alternative to Linux. Either that, or the company will eventually find itself shopping for a Linux distributor of its own. Either way, it seems likely that competitive pressures for operating systems (and higher layers) sales and support are set to increase, especially in the high-performance web server area. Red Hat, whose PostgreSQL-based database offering appears to have fallen below the radar, may find itself scrambling for a response.

Sun makes a big point of being able to sell the entire package, and there is some truth to that. Processors, storage, systems software, database software, programming languages, office suites, and more can all be had from one company. What remains to be seen is whether this is really what customers want. There is a lot of value in being able to integrate components from multiple sources and not being dependent on a single vendor. Your editor, who managed a transition from being an all-DEC shop to an all-Sun shop some twenty years ago, is not convinced that those days are worth going back to.

Comments (12 posted)

Ten-year timeline, part 2: the bubble days

Just how one might go about looking for such a deal is not immediately obvious - especially if you're a bunch of technical people who have no clue about how corporate acquisitions are done. Somehow, hanging an "Acquire Us!" sign on the front page did not quite seem like the right way to go. After some thought, we decided that the best approach might be to just quietly slip the word to a few people that we might be open to offers, then sit back and see what happened. As it turned out, that was all we needed to do. Much of the following story has never been told - but all of the non-disclosure agreements have run out by now, so this seems like the right time.

Meanwhile, things were happening at a furious pace in the Linux community.

• August 26, 1999: Red Hat and Caldera get around to year-2000 compliance. The 2.3.15 patch is "huge", touching all of 600 files (2.6.24 currently has changes to over 10,000 files). The first Ottawa Linux Symposium concludes.

• September 2, 1999: Sun buys StarDivision, but uses its "community source license" for the code. Red Hat shuts down "Red Hat Linux" vendors on Amazon.

• September 9, 1999: SCO (old SCO, mind you, not the current company) trashes Linux in Europe. Bruce Perens worries that Sun may be trying to grab control of the Linux desktop through its acquisition of StarDivision. Disruptive changes in the "stable" 2.2 kernel upset users.

• September 16, 1999: the 2.3 kernel goes into "feature freeze," with Linus predicting a release by the end of the year. He neglected to specify which year, though. Cobalt networks files to go public. LinuxOne - a company nobody had ever heard of - files to go public. Andover.net (the company which had bought Slashdot) files to go public. The first ext3 filesystem patches are released.

The 2.3 feature freeze is instructive - 2.4.0 was not released until January, 2001 - 16 months after this "freeze" went into effect. Over the next months we'll see plenty of reasons for the delay in the 2.4.0 release; Linus was famously not a great release manager. But releases which failed to arrive were the norm back in those days. Free software was much like proprietary software in that regard. One has to look back to realize just how much better we have gotten at getting software releases out in a reasonable period of time.

The IPO filings were beginning to pile up - much to your editor's chagrin. Actually reading those things is a painful chore, and we felt that we needed to examine all of them. The relative newcomers out there may be wondering who that LinuxOne company is. So were we, at the time. LinuxOne materialized out of thin air, slapped its name onto a copy of Red Hat Linux, and called itself a Linux company. They clearly hoped to get in on the general mania and make a bunch of money before people caught on - they nearly achieved it, too.

• September 30, 1999: Caldera spinoff Lineo gets going - remember Embedix and Embrowser? Red Hat drops LWN news from its web site.

Lineo got spun out of Caldera for a couple of apparent reasons: (1) to isolate the DR-DOS lawsuit which was being pursued against Microsoft, and (2) to try to double the number of public offerings. The first objective was achieved, and the suit was ultimately successful. In the end, though, Lineo still failed to get off the ground.

• October 7, 1999: Sun announces that it will be releasing the Solaris source code. The OpenBSD project grabs the last freely-licensed version of ssh and starts the OpenSSH project.

• October 14, 1999: TurboLinux gets a big chunk of venture money. SCO (old SCO) buys a chunk of the Linux Mall. Crypto export rules in the U.S. begin to soften. The devfs discussion continues. SGI, VA Linux, and O'Reilly launch a commercialized version of the Debian distribution. VA Linux files for its IPO.

Old-timers will remember the Linux Mall - that was the place, once upon a time, where we bought our Linux CDs (and stuffed penguins too). Yes, we actually bought Linux on CD and waited for it to show up via mail, though it may seem a little strange now. The Linux Mall, and its founder Mark Bolzern, were fixtures in the early days of Linux. As Linux grew and bandwidth increased, though, the Linux Mall was having a bit of a hard time of it. The name was famous, though, and the site got a lot of traffic, so companies interested in getting into the Linux hype were interested in it.

It may be getting a bit ahead of the story, but this is as good a place as any to let it be known that one of the things that the Linux Mall wanted to do with its new-found wealth was to acquire a media outlet like LWN. It was part of the bigger plan of creating a full-featured e-commerce "mall" centered around Linux. We considered the offer long and hard, but, in the end, declined it. Just as well: the Linux Mall missed the IPO boat and got folded into EBIZ, which, in turn, eventually went bankrupt. Had we taken that path, there would be no LWN now.

• October 21, 1999: LinuxToday is acquired by Internet.com; co-founder Dave Whitinger leaves the building. ATI announces that it will be releasing 3D programming information for its video adapters - the good news here is that it's finally getting around to doing that.

• November 4, 1999: DVD encryption is cracked and DeCSS is released. The Y2K-related "windowing" patent threatens the kernel. Burn all GIFs day. The kernel gets past the longstanding 1GB limit on installed memory. Slackware 7 (the successor to Slackware 4) is released. The non-profit Red Hat Center for Open Source launches - and is never heard from again.

• November 11, 1999: Cobalt network goes public, shares begin trading at $130.

• November 18, 1999: The Linux Business Expo is held as part of the once-famous COMDEX event. Red Hat acquires Cygnus. BitKeeper is said to be getting closer to release. Mozilla hits milestone 11 and is said to be getting closer to release. Advogato.org launches.

LWN has only rarely operated booths at conferences, but we did have one at the Comdex Linux Business Expo. For the curious, here's a picture from the event featuring LWN editor Rebecca Sobol. That week's LWN edition was produced from that booth after the floor closed, under the watchful eye of security guards who didn't think we should be there. Your editor remembers it as one of the coldest experiences of his life. During the show, we subjected to constant, highly-amplified screaming obnoxiousness from the large booth being run by LinuxToday - the acquisition, it seemed, had put that site onto a rather less dignified path.

The other thing LWN was doing at this event was talking with potential suitors. One of those was a company called Atipa, which was operating a large booth of its own. Atipa was a VA-style Linux box vendor with a grand plan for a Linux portal site which would, eventually, be the place people went for Linux information. They thought that LWN would make a good addition to that portal, and were pushing hard to make a deal.

We met a few times with Atipa's CEO, a charismatic man who told a good story. The company, he said, was going to outdo even the coming VA Linux IPO, which was already clearly going to be big. Along the way he was going to pick up companies like Applix and open-source the ApplixWare office suite - something which would have been nice at the time. He stated flat out that he was soon to be a billionaire, and that we could share in that bonanza. It was quite the tale, but we tended to walk out of these meetings believing every word of it.

With some distance, though, the glow always faded. We wondered why our visit to the company's headquarters revealed a building almost devoid of people. The magic "profit happens here" step in their plans seemed less inevitable when looked at later. In the end, we did not take this deal. Thereafter, we received (unverifiable) word that Atipa's investors started asking some harder questions and found that, perhaps, they, too, had allowed themselves to be charmed more than they should have. Atipa rather abruptly found a new CEO, the IPO never happened, and investors, presumably, lost their money.

Also at the Linux Business Expo, we met with some representatives from O'Reilly. They were getting the O'Reilly network off the ground, and thought that LWN might make a good addition to it. They eventually offered us a deal (which looked more like a traditional angel investment than an acquisition) and a network affiliation which would have given us a portion of the revenue from the ads they sold. Your editor, who has a lot of respect for the people at O'Reilly, has always had a bit of regret at turning down this offer. It was an opportunity to get business advice from some very smart people. But it would almost certainly have been fatal to LWN once the advertising market fell apart.

Meanwhile, the acquisition of Cygnus by Red Hat led to a fair amount of online worrying about whether Red Hat was set to take over Linux by virtue of employing a number of GCC developers. Such fears look a little silly now, but they seemed real then.

• December 9, 1999: Andover.net goes public. The kernel gets NUMA support (during a feature freeze, remember). Sun announces a Linux Java release, rolling over the "Blackdown" team which had been working on this release for years.

• December 12, 1999: VA Linux goes public, setting the record for the largest first-day gain in NASDAQ history. Eric Raymond gets rich and lets us all know about it. The non-free BitKeeper license is revealed. LinuxCare acquires the Puffin Group and gets another $32 million. The Linux Capital Group launches; it starts by funding Progeny Linux. Companies send out "we use Linux" press releases in an attempt to make their stock price go up.

The VA IPO was not just the peak of the Linux bubble - it could well be the peak of the dotcom bubble as a whole. It was not possible to watch that stock rise to well over $300 a share on the first day and not be overwhelmed by a sense of unreality. Still, it seemed like no more than what Linux deserved, and people somehow expected it to continue.

• January 6, 2000: Linux survives Y2K. Red Hat buys Hell's Kitchen Software, does nothing with it. VA Linux launches the SourceForge site.

• January 13, 2000: Caldera Systems (later to become SCO) files for its IPO. The kernel gets a new block driver API and 32-bit UIDs - still during the feature freeze.

• January 20, 2000: LinuxCare files for its IPO. Linus Torvalds shuts down the sale of a number of Linux-related domain names. Secure Computing Corporation announces that it will be developing (what becomes) SELinux. Enoch becomes Gentoo Linux. TurboLinux completes another funding round.

Once upon a time, Caldera Systems was supposed to be among the biggest winners in the distribution sector - they had the business connections and the distribution channels. "Linux for business" got the company far enough to do an IPO, but not much beyond that. This is, of course, the company which eventually became the SCO Group.

Caldera was well overshadowed by LinuxCare, though. The distribution business always looked like a hard one to maintain over the long term - that is why Red Hat was trying to be a web portal company. Services were going to be the real gold mine, and LinuxCare was going to be at the top of the Linux support industry. The company got money from left and right (a funding round produced offers of ten times the target amount) and hired a long list of well-known Linux hackers.

Need we say that LWN's editors paid a visit to LinuxCare during this time? It was a hard time for LinuxCare to discuss acquisitions, since the IPO process was already underway, but discuss they did. So we went to the famous San Francisco headquarters. Your editor's memories from that day are strong. LinuxCare was filled with hundreds of people who all believed they were on the way toward an IPO that would exceed even VA Linux; suffice to say they were happy about the prospect. Meanwhile, though, a couple hundred of them were all working in a single not-very-large room called "the barn"; it resembled, more than anything else, a school lunchroom filled with long tables. Everybody worked on a laptop because there was no room in their tiny piece of table space for anything else. They all complained about having colds. It looked awful.

LinuxCare's negotiator was an ex-fighter jet pilot who retained the "top gun" attitude. When valuations were discussed, we were told that offering LinuxCare's pre-IPO shares at $50-60 each was being generous to us. Issues like editorial control were not really even on the table. In the end, we turned this deal down, but with a feeling like we were throwing a winning lottery ticket in the trash. Of course, subsequent events showed that we need not have worried about this particular missed opportunity.

• February 10, 2000: Real-time Linux turns out to be patented. VA Linux acquires Andover.Net. The KDE project moves to SourceForge. Atipa acquires Enhanced Software Technologies. The Linux Fund announces that it will be filing for an IPO.

The Andover.Net acquisition was announced at LinuxWorld in New York - LWN was there, of course. The initial deal included a massive pile of cash to be handed to Andover.Net's shareholders, but people questioned that handout to the extent that it eventually went away. Andover.Net's owners had to content themselves mostly with VA Linux shares, which, already, were worth considerably less than they had been on IPO day. In the end, Andover.Net turned out to be a good buy for VA Linux, once it became clear that the Linux-installed computer business was harder than it had looked.

We were approached by a VA executive at LinuxWorld to see if we were interested in maybe being acquired sometime. By then, though, we had so many offers that we couldn't really give them all serious consideration. So we did not pursue that opportunity.

But, at this event, we did talk with some representatives from ZDNet, who were also looking for a Linux site to buy. The offer they made was, by far, the most generous of any. By some reckoning, we should have taken it. Certainly it would have come out better than most of the other options we had. But ZDNet would have exercised more editorial control than we would have liked, and, being already a public company, it didn't offer that IPO "pop" that we somehow thought was our due. So we ended up not taking that path.

• February 17, 2000: devfs is merged into the mainline kernel. Also merged is the "softnet" core networking rework. Remember, the kernel is in a feature freeze.

• February 24, 2000: Eazel is founded with the goal of improving Linux usability.

To your editor, Eazel never made sense from the beginning. There was, truly, no revenue model. Indeed, it seemed like a scam designed to draw venture money for the purpose of writing Nautilus. To that extent it succeeded, but the investors cannot have been happy in the end.

• March 2, 2000: Atipa announces $30 million in investments.

• March 23, 2000: Caldera Systems goes public; its share price merely doubles. The planned date for LinuxCare's IPO passes with no offering.

• April 4, 2000: Linuxcare's IPO is pushed back to April 24 - or so they say. EBIZ acquires longtime Linux CD distributor InfoMagic. Atipa Linux Solutions acquires DCG Computer Corp. Sendmail Inc. gets $35 million in funding.

This was the point where LWN announced that it had been acquired by a company called Tucows. We had, in fact, been talking with them for some months, and had made the decision in February. It took some time, though, for the lawyers to hammer out the final agreement. In the end, we were probably exceedingly lucky: market conditions were going downhill in a hurry by this point and, had the negotiations stretched out much longer, Tucows might have started looking for reasons to back out of the deal.

Or maybe not. We went with Tucows for a number of reasons, but at the top of the list was that they were clearly smart and decent people who, while arguably being carried away by the bubble like the rest of us, clearly had a functioning business underneath it all. Their acquisition of LWN never yielded the benefits they were looking for, but the people at Tucows always treated us well and we still count them as friends. Perhaps we were smart, or perhaps we were just very lucky, but, in retrospect, we came out of a complex, high-stakes process having made what was probably the best possible decision.

The Tucows acquisition made it possible for LWN editors Rebecca Sobol and Forrest Cook to join as regular staff members. It also positioned us within a safe harbor for the dotcom crash, which was already in progress. But the story of those years will be the subject of next week's installment.

Comments (29 posted)

Page editor: Jonathan Corbet

Security

A kernel security hole

Security holes can sneak into code in surprising ways, even in highly scrutinized codebases. Perhaps even more surprising is how long they can persist in something as popular as the Linux kernel before someone notices. The release of stable kernels 2.6.22.16 and 2.6.23.14 this week are instructive for both of those reasons.

The bug that led to the releases is fixed by a two line patch, but might be exploitable to cause filesystem corruption. If it were a bug in a driver for an obscure piece of hardware, with relatively few users, it might have been less eye opening, but it was in the Virtual File System (VFS) layer of the kernel. VFS is the abstraction that allows all kernel filesystems to be used identically regardless of their underlying implementation. The open() system call is used to open any file on any type of filesystem; VFS is what makes that work.

In fact it is the open() path that is affected by the bug. Due to a faulty test, the bug allows directories to be opened for writing, which is generally a recipe for disaster. It could also allow a file on a read-only filesystem to be opened for writing – depending on the underlying filesystem implementation, that could lead to corruption. In both cases, they are only locally exploitable.

The bug was introduced in a change to support NFS in October of 2005 – more than two years ago; all kernels since 2.6.15 are affected. The change was aimed at making NFSv4 open calls be atomic (because an open is really a lookup followed by an open), but also did some code reorganization that changed the semantics of a flag variable. That variable was being used to determine the access mode for directories and read-only filesystems, so that change subtly broke the tests.

Part of the problem is that the tests are in a function called may_open(), which takes two flag parameters:

    int may_open(struct nameidata *nd, int acc_mode, int flag)

There may be no easy solution to this kind of problem – other than more scrutiny. Using different types, rather than plain int, for each flag might have helped, but since the tests were using the right kind of bit values for flag, that is a somewhat hard sell.

Something unpleasant to consider in all of this is that this may not be the first time this problem has been noticed. It may just have been the first time it was noticed by someone who reported it. Folks with a malicious intent are much less inclined to report bugs. This particular bug is not one that would be particularly useful to attackers, but we would do well to remember that fixing a two year old hole means that systems were vulnerable for all that time. It is not only the good guys who can read code.

Comments (26 posted)

New vulnerabilities

apache: several vulnerabilities

Comments (1 posted)

claws-mail: insecure temp file

Comments (none posted)

drupal: multiple vulnerabilities

Update to 5.6, security fixes:

DRUPAL-SA-2008-005
DRUPAL-SA-2008-006
DRUPAL-SA-2008-007

Comments (none posted)

fail2ban: denial of service

Comments (none posted)

gforge: SQL injection

Comments (none posted)

httpd: cross-site scripting, denial of service

Comments (1 posted)

kernel: denial of service vulnerabilities

Comments (none posted)

libxml2: denial of service

Comments (none posted)

moodle: cross-site scripting

Comments (none posted)

openafs: denial of service

Comments (none posted)

paramiko: insecure random pool usage

Comments (none posted)

R: buffer overflows

Comments (none posted)

xfce4: multiple vulnerabilities

Comments (none posted)

xine-lib: buffer overflow

Comments (none posted)

Updated vulnerabilities

apache2: information disclosure

Comments (2 posted)

apache: multiple vulnerabilities

Comments (1 posted)

apache: cross-site scripting

Comments (none posted)

apache2: denial of service

Comments (1 posted)

httpd: denial of service, cross-site scripting

Comments (none posted)

Asterisk: denial of service

Comments (none posted)

asterisk: possible SQL injection

Comments (none posted)

autofs: privilege escalation

Comments (1 posted)

autofs: insecure default configuration

Comments (none posted)

bind: insecure permissions

Comments (1 posted)

cacti: SQL injection vulnerability

Comments (none posted)

cacti: denial of service

Comments (none posted)

cairo: integer overflow

Comments (none posted)

clamav: denial of service

Comments (none posted)

clamav: multiple vulnerabilities

Comments (none posted)

clamav: mystery vulnerability

Comments (1 posted)

clamav: integer overflow and off-by-one

Comments (none posted)

cups: denial of service

Comments (none posted)

cups: buffer overflow

Comments (1 posted)

cups: multiple vulnerabilities

Comments (none posted)

debian-goodies: privilege escalation

Comments (none posted)

Django: denial of service

Comments (none posted)

dovecot: multiple vulnerabilities

Comments (none posted)

dovecot: privilege escalation

Comments (none posted)

dovecot: directory traversal

Comments (none posted)

e2fsprogs: integer overflows

Comments (none posted)

emacs: buffer overflow

Comments (none posted)

emacs: command execution via local variables

Comments (1 posted)

evolution: format string error