User: Password:
Subscribe / Log in / New account

Leading items

Striking gold in binutils

By Jake Edge
March 26, 2008

A new linker is not generally something that arouses much interest outside of the hardcore development community—or even inside it—unless it provides something especially eye-opening. A newly released linker, called gold has just that kind of feature, though, because it runs up to five times as fast as its competition. For developers who do a lot of compile-link-test cycles, that kind of performance increase can significantly increase their efficiency.

Linking is an integral part of code development, but it can be invisible, as it is often invoked by the compiler. The sidebar accompanying this article is meant for non-developers or those in need of a refresher about linker operation. For those who want to know even more, the author of gold, Ian Lance Taylor, has a twenty-part series about linker internals on his weblog, starting with this entry.

For Linux systems, the GNU Compiler Collection (GCC) has been the workhorse by providing a complete toolchain to build programs in a number of different languages. It uses the ld linker from the binutils collection. With the announcement that gold has been added to binutils, there are now two choices for linking GCC-compiled programs.

A linker overview

For non-developers, a quick overview of the process that turns source code into executable programs may be helpful. Compilers are programs that turn C—or other high-level languages—into object code. Linkers then collect up object code and produce an executable. Usually the linker will not only operate on object code created from a project's source, but will also reference libraries of object code—the C runtime library libc for example. From those objects, the linker creates an executable program that a user can invoke from the command line. The linker allows program code in one file to refer to a code or data object in another file or library. It arranges that those references are usable at run time by substituting an address for the reference to an object. This "links" the two properly in the executable. Things get more complicated when considering shared libraries, where the library code is shared by multiple concurrent executables, but this gives a rough outline of the basics of linker operation.

The intent is for gold to be a complete drop-in replacement for ld—though it is not quite there yet. It is currently lacking support for some command-line options and Linux kernels that are linked with it do not boot, but those things will come. It also currently only supports x86 and x86_64 targets, but for many linker jobs, gold seems to be working well. The speed seems to be very enticing to some developers, with Bryan O'Sullivan saying:

When I switched to using gold as the linker, I was at first a little surprised to find that it actually works at all. This isn't especially common for a complicated program that's just been committed to a source tree. Better yet, it's as fast as Ian claims: my app now links in 2.6 seconds, almost 5.4 times faster than with the old binutils linker!

Performance was definitely the goal that Taylor set for gold development. It supports ELF (Executable and Linking Format) objects and runs on UNIX-like operating systems only. Only supporting one object/executable format, along with a fresh start and an explicit performance goal are some of the reasons that gold outperforms ld.

Tom Tromey likes the looks of the code:

I looked through the gold sources a bit. I wish everything in the GNU toolchain were written this way. It is very clean code, nicely commented, and easy to follow. It shows pretty clearly, I think, the ways in which C++ can be better than C when it is used well.

Because the implementation is geared for speed, Taylor used techniques that may confuse some. He has some concerns about the maintainability of his implementation:

While I think this is a reasonable approach, I do not yet know how maintainable it will be over time. State machine implementations can be difficult for people to understand, and the high-level locking is vulnerable to low-level errors. I know that one of my characteristic programming errors is a tendency toward code that is overly complex, which requires global information to understand in detail. I've tried to avoid it here, but I won't know whether I succeeded for some time.

Overall, it seems to be getting a nice reception by the community, with O'Sullivan commenting that he is "looking forward to the point where gold entirely supplants the existing binutils linker. I expect that won't take too long, once Mozilla and KDE developers find out about the performance boost." Once gold gets to that point, Taylor is already thinking about concurrent linking—running compiler and linker at the same time—as the next big step.

There are two other ongoing projects that are working with the greater GCC ecosystem in interesting ways: quagmire and ggx. Quagmire is an effort to replace the GNU configure and build system—consisting of autoconf, automake, and libtool—with something that depends solely on GNU make. Currently, that system uses various combinations of the shell, m4, and portable makefiles to make the building and installation of programs easy—the famous "./configure; make" command line. The tools were written that way to try and ensure that users did not need to install additional packages to configure and build GNU tools. Quagmire, which has roots in a posting by Taylor recognizes that GNU make is ubiquitous, so basing a system around that makes a great deal of sense.

The ggx project is Anthony Green's step-by-step procedure to create an entire toolchain that can build programs for a processor architecture that he is creating as a thought experiment. The basic idea is to design the instruction set based on the needs of the compiler, in this case GCC, rather than the needs of the hardware designers. He is using GCC's ability to be retargeted for new architectures, along with its simulation capabilities to create a CPU that he can write programs for. As of this writing, he has a "hello world" program working, along with large chunks of the GCC test suite passing. Well worth a look.

Comments (146 posted)

Toward a free metaverse

By Jonathan Corbet
March 26, 2008
Last month, an article about another attempt to free the proprietary Ryzom game expressed frustration with the implied idea that the free software community could not, on its own, create a game experience comparable to Ryzom. One of the resulting comments took issue with (what was seen as) a dismissive attitude toward the Second Life client and pointed out some of the work which is being done based on that client. So your editor decided to take another look. The bottom line is this: the work being done in this area is still in an early and unstable state, but it does have the potential to open a new frontier for free software in the area of virtual environments.

The Second Life client for Linux is now in a beta release. "Beta," in this case, means that all of the features have, in some way, been implemented; now it's just a matter of making it all actually work. Your editor found the client to be slow, unwieldy, crash-prone, and very fussy about its graphics environment. Your editor's well-supported (in X) Intel-based desktop was not adequate for this client, for example; the associated documentation recommends a long list of cards which (for now) are only supported with proprietary drivers. Still, on the right system, the client is able to render three-dimensional worlds with the same quality that, well, Second Life has on any platform.

An alternative is OpenViewer, a C#/Mono-based, BSD-licensed viewer project. Your editor had little luck getting this client going, but the screenshots are nice. The developers appear to have made significant progress toward the creation of a functional, three-dimensional client; this is a project to watch. Less far along is the Aether project, which is working on a OpenViewer-based client meant to run within Firefox; thus far, it has a nice design diagram but not much else.

There is also RealXtend, a project based on the Second Life client which is emphasizing performance and visual quality. Unfortunately, it also seems to be emphasizing Windows support, so your editor did not give it a try.

Free software clients are certainly an important tool to have; we will not be able to access this kind of virtual environment without them. But it would be a real shame if these clients simply facilitated a world where we use free clients to access locked-down, proprietary virtual worlds on somebody else's server. What would be much better would be the ability to create our own virtual worlds - using free software, of course - and to link those worlds into a larger virtual universe. That is the formula which made the World Wide Web (and many other Internet services) work, and it should certainly be applicable in this context as well.

The good news is that people are working in this area. One project, OpenSim, has the look of something which is about to achieve much wider awareness as its features mature. In short, OpenSim is a virtual world server which can be deployed to create environments much like what one would find in Second Life. It works with the Second Life client and with OpenViewer as well, and it presents a very similar experience - at least, in the virtual worlds which have been deployed so far. Since it's free software, it can be customized toward the creation of different kinds of environments, including role-playing games and such.

It is written with C# and Mono - seemingly a common choice for this kind of software. The Mono environment, for all its faults and potential pitfalls, may well make it easier to create a cross-platform application with the requisite features.

What makes OpenSim really interesting, though, is its ability to connect servers together in a "grid" mode. Once this is done, a virtual world is not limited to a single entity's server (or imagination). Servers across the net can be interconnected into a single, larger world. This is the feature which has the potential to take OpenSim from another interesting project into something which transforms the net.

There are a number of people organizing grids with OpenSim now; there is a list of public grids on the OpenSim site. Some of them appear to be relatively proprietary operations offering the opportunity to buy virtual land - though subprime loans are unavailable. Others allow anybody connect their server into the grid and become part of the whole. These grids appear, in general, to be in a sort of early adopter state at the moment, but much of the fundamental functionality is there. How hard could it be to make it all work properly at this point?

The answer to that question, of course, is "quite hard." But the fact remains that people are working on this very interesting problem, and they are making significant progress toward solving it. These projects bear watching; they may well be planting the seeds of the systems we will all be using in the coming years.

Comments (14 posted)

Bruce Perens and the OSI board

By Jake Edge
March 24, 2008
The Open Source Initiative (OSI) was formed almost ten years ago to safeguard the "Open Source" name. Over the years it has approved licenses and attempted some other activities while, generally, having little relevance to the wider community. It has often been seen as a relatively closed and non-democratic organization. Now one of OSI's founders is trying to get back into the organization and change its direction; the outcome of the resulting discussion may (or may not) change the direction of the OSI.

Bruce Perens has launched a bid to be elected to the OSI board of directors, but this bid has not been particularly well received by the current board. His on-line petition to collect community support specifies a number of reasons that he wants to be on the board—those reasons are ruffling some feathers. Outgoing board member Matt Asay has taken Perens to task for some of his statements as has OSI president Michael Tiemann.

Perens's reasons for wanting to be on the board are threefold: reducing the over-representation of vendors, trying to ensure Microsoft does not get a seat on the board, and reducing license proliferation. The idea of a Microsoft seat on an open source organization's board is sure to rile a segment of the community, which is undoubtedly part of what Perens is hoping for. The likelihood of that happening is pretty small, though. Tiemann makes it clear that the board doesn't elect companies at all:

The OSI nominates people to the board despite their corporate affiliations, not because of them. The idea that the OSI would elect a "Microsoft" board member is as absurd as the idea that we'd elect a "Google" board member or an "IBM" board member. We elect people based on their own merits, not the merits (or demerits) of the companies or organizations they are affiliated with.

Microsoft and its employees do not currently contribute to open source in any substantial way, so there is little that would lead the board to nominate them. If that ever changes, it would be pretty disingenuous to deny someone a seat because of their employer's past—or even at that time, current—misbehavior. In addition, it is hard to see how one board member—Perens or someone "controlled" by Microsoft—is going to make such a crucial difference in what the board does anyway. In many ways, the Microsoft connection is a red herring—one sure to rally the troops, though.

Reducing license proliferation is a noble goal, one that the OSI tried to tackle a few years back without much in the way of tangible success. Perens states that he would like to see OSI do more reduce the number of licenses, but his claims about the number of licenses needed have raised eyebrows:

Another problem is the failure to reduce the number of different licenses in general use. My own work in this area shows that only four licenses, all compatible with each other, can satisfy all common business and non-business purposes of Open Source development. Three of these licenses have essentially the same text, and the fourth is very short. Life would be easier if more projects used them. While it would be difficult to shut down approval of new licenses, I think OSI could be more proactive at reducing license proliferation.

Part of the reason that Tiemann and others are skeptical is due to some obvious bad blood between the board and Perens over the license proliferation committee. LWN covered some of that "debate" in August 2005. Perens clearly believes he should have been a member just as strongly as others on the board seem to feel he should not have been. When the board was formed without him as a member, Perens refused to participate in the process in any way. It seems to stick in the craw of some for Perens to now claim that he has the solution. Russ Nelson, former OSI president and current board member—as well as a member of the committee—sums up the frustration in a comment on Tiemann's post:

I don't see how Bruce can claim to have a short list of four licenses. I start with BSD, GPLv2, GPLv3, LGPLv2 and LGPLv3 and that's five. If he thinks that people should simply agree with him that all GPLv2 should be relicensed GPLv3, I invite him to spend some time with Linus Torvalds, who notoriously and politely disagrees.

Having a solution is not the same as convincing people to adopt it.

It is rather interesting to see Perens trying to get back on the board that he famously resigned from in 1999 after having founded the organization with Eric Raymond in 1998. This is not the first time Perens has lost interest and/or resigned from some form of community leadership position; Debian and UserLinux spring to mind. Though none of the expressed concerns about his candidacy have mentioned it, some must be wondering how long it would be before ideology or a shifting focus caused Perens to move on from a board position if he were elected.

Perens has been an excellent advocate for free software and/or open source over the years, but his tendency towards self-promotion grates on some. It may not be an ego thing, as he claims, but it certainly rubs some people the wrong way. The ego issue is one of the reasons that board observer Andrew Oliver does not support Perens for the board:

A return to a very Amerocentric hacker culture voice with big egos is not the answer to OSI's problems. I think OSI is on the path to real fundamental change. I'd like to hear Bruce explain what he'd do differently in collaboration with others who may not always agree with him.

Asay certainly doesn't see Perens as having the right credentials either:

The OSI needs a vibrant membership of those currently shaping the open source landscape. It's possible that its current make-up doesn't reflect this. Point well taken. But it's equally possible - indeed, I'd say probable - that Bruce's directorship wouldn't change this. I like Bruce but aside from the occasional picketing he does, I can't point to anything substantive he has done for open source in the past half-decade or so.

The petition drive came about because Tiemann encouraged Perens to show that there was strong community support for him to be a part of the board. As of this writing, the petition has garnered more than 1700 "signatures", which Perens believes is enough:

Regarding my candidacy, OSI's board, through its president, asked me to show an uprising of strong community support if the board was to to elect me. I have. Now that I have done what you asked, are you going to hide behind complaints about my campaign, which is really quite mild in its criticism and is in no way the "scorched earth" that Matt refers to, or are you going to do what you said? If you OSI can't handle a political opponent on my laid-back scale, you'd only looking for yes-men.

The OSI board is "self-replacing" with current board members nominating and electing candidates for empty slots. Each director serves for a three-year term, with roughly one-third coming up for election each year—though this year there are five slots to be filled. Three directors are standing for re-election, leaving two slots open. Unfortunately, it's not clear when the actual election will be held, nor is there likely to be any advance notice of who has been nominated. Transparency, it seems, is not one of the attributes of OSI.

Self-replacement and overlapping terms of office tend to give a certain stability to a board, but it also creates a kind of inbreeding. It is unlikely that a board will nominate people who think substantially differently from themselves. This is one thing that Perens is trying to circumvent with his very public candidacy. Whatever else can be said about Perens's candidacy, it is clear that he would bring a different voice into the OSI boardroom.

But, what is OSI really? Is it an organization that is somehow supposed to represent all of the diverse voices in the community? At the moment it appears to exist for the purpose of approving licenses and "protecting the Open Source Definition". Perens thinks it could be more than that. OSI itself seems to agree as they have been moving towards more relevance in the community. Oliver describes that effort:

OSI is trying to solve its problems, by becoming more grassroots and less bottom up. Meanwhile, it is trying to grow the movement by expanding its international representation. Corporations do influence OSI, in that not all of the board has a free hand to say what is on their mind publicly. However, the solution is to make the OSI board what it should be: a governance board.

OSI and its board are currently in a state of flux, trying to define a role for themselves that is broader than just a license approval body. There doesn't seem to be a lot of discontent within the board that might lead to Perens or another controversial figure being added. Whether this leads to continued stagnation or a more vibrant OSI remains to be seen. A more interesting question might be: will anyone care?

If OSI starts to do visible things for the community, it will finally acquire some relevance. Given the attitude towards his candidacy, it seems unlikely that Perens will be able to lead the board in that direction. Which leaves it up to the current board and the two new members—neither of which are likely to be Perens—to find a way to make the community care.

Comments (55 posted)

Page editor: Jonathan Corbet
Next page: Security>>

Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds