The up-and-coming LLVM compiler has been an irritation to some GCC developers for some time; LLVM apparently comes off as an upstart trying to muscle into territory which GCC has owned for a long time. So it's not surprising that occasionally the relationship between the two projects gets a little frosty.
Consider the case of DragonEgg, a GCC plugin which replaces the bulk of GCC's optimization and code-generation system with the LLVM implementation. DragonEgg is clearly a useful tool for LLVM developers, who can focus on improving the backend code while making use of GCC's well-developed front ends. Jack Howarth recently proposed the addition of DragonEgg as an official part of the GCC code base. Some developers welcomed the idea; Basile Starynkevitch, for example, thought it would make a good plugin example. But from others came complaints like this:
It's not clear that this is a majority opinion; some GCC developers see DragonEgg as an easy way to try out LLVM code and compare it against their own. If LLVM comes out on top, GCC developers can then figure out why or, possibly, just adopt the relevant LLVM code. Those developers see only benefit in some cooperative competition between the projects.
Others, though, see the situation as more of a zero-sum game; when viewed through that lens, cooperation with LLVM would appear to make little sense. But free software is not a zero-sum game; the more we can learn from each other, the better off we all are. GCC need not worry about being displaced by LLVM (or anything else) any time in the near future. Barring technical issues with the merging of DragonEgg (and none have been mentioned), accepting the code seems like it should be ultimately beneficial to the project.
In a side discussion, GCC developers wondered why LLVM seems to be more successful in attracting developers and mindshare in general. One suggestion was that LLVM has a clear leader who is able to set the direction of the project, while GCC is more scattered. Others have a different view; in this context, Ian Lance Taylor's notes are worth a look:
There is also the matter of the old code base, the lack of a clean separation between passes, and, most important, weak internal documentation.
Some of these issues are being fixed; others will take longer. It seems clear that attending to these problems is important for the long-term future of the project.
Lest things look too grim, though, it's worth perusing this posting from Taras Glek on his success with the GCC "profile-guided optimization" (PGO) feature. PGO works by instrumenting the binary, then rebuilding the program with optimization driven by the profile information. With Firefox, Taras was able to cut the startup time by one third and to reduce initial memory use considerably as well. Taras says:
There's no shortage of interesting, development-oriented tools being integrated into GCC, and the addition of the plugin architecture can only result in an acceleration of this process. Things have reached a point where more projects should probably be looking into the use of these tools to improve the experience for their users.
Meanwhile, on the LLVM side, the developers have recently unveiled the LLVM MC project. "MC" stands for "machine code" in this context; in short, the LLVM developers are trying to integrate the assembler directly into the compiler. There are a number of reasons for doing this, including performance (formatting text for a separate assembler and running that assembler are expensive operations), portability (not all target systems have an assembler out of the box), and the ability to easily add support for new processor instructions. Much of this functionality is required anyway for LLVM's just-in-time compiler features, so it makes sense to just finish the job.
This work appears to be fairly well advanced, with much of the basic functionality in place. Chris Lattner says:
In summary: there is currently a lot going on in the area of development toolchains. Given that all of us - including those who do no development - depend on those toolchains, this can only be a good thing. Computers can do a lot to make the task of programming them easier and more robust; despite the occasional glitch, developers for both GCC and LLVM appear to be working hard to realize that potential.
Greg Kroah-Hartman delivered some "tough love" to Android in his keynote at this year's Embedded Linux Conference (ELC). He is very clearly excited about Android and what it can do—uses it daily as his regular phone—but is unhappy with Google's lack of community engagement. There is hope that things will change, he said; there has been a fair amount of "introspection" at Google that he hopes will lead it in a more community-oriented direction.
CE Linux Forum architecture group chair and ELC organizer Tim Bird introduced Kroah-Hartman by noting how many kernel subsystems he maintains: eight. The "amount of work that Greg does in the Linux kernel is really hard to comprehend", Bird said. In addition, he has a knack for "involving people in the community". Kroah-Hartman is "omnipresent" in kernel circles and "when he sees a problem, he just comes up with a solution for it". It would seem that the keynote was targeted at solving the "Android problem".
As part of his disclaimer, Kroah-Hartman noted that he comes at the problem as an experienced kernel developer who also has a background in the embedded space. While he works on MeeGo for Novell as his day job, "they don't even realize I'm here giving this talk". He also noted that he is only looking at the Android kernel, as the user space is "weird" and won't be part of his talk.
Five years ago, phone manufacturers approached the kernel developers because they wanted to use Linux on phones. They needed a stable kernel and X server, along with a free Java. The latter was the sticking point because, at that time, there was no free Java—and it isn't something that the kernel hackers could provide. But Android changed that, providing the third piece that the phone makers need.
He pointed out that all of the complaints he was about to make about Android could be fixed "tomorrow" and that the Android user-space applications wouldn't have to change at all. But it's not something that the kernel community can do alone, Google needs to be involved to change the libraries that make up the platform.
There were several things that Google did right, Kroah-Hartman said, starting with its choice of Linux. In an aside, he noted that all phone manufacturers bring up their phones using Linux, including Apple with the iPhone; "a little-known fact". He also lauded Google for following the kernel license, which is something that Palm didn't initially do with WebOS, he said. He pointed to android.git.kernel.org as a "wonderful site" that contains all of the Android code in easily accessible Git repositories. But "that's all the good".
The list of things that Google got "wrong" starts with android.git.kernel.org because it is so disorganized and chaotic. There are six different kernel trees available there with 33 separate branches. There are three branches of 2.6.34, which is "not even released yet". The trees go back to 2.6.25 and some of the older trees are still being updated. Some of the branches are for different hardware, but some aren't.
There are also things like an "old stale Linus tree" in the collection, along with two standalone drivers in a separate repository. One of those drivers has 13 branches, "not all for just one kernel version". There is also an empty repository—with four branches. "It's a mess", he said, and Google is "burying the code in public". That's a common thing for companies to do; "they are doing it well, and it's crap".
He looked at a branch based on 2.6.34-rc2, noting that there were 283 files changed, with 47,715 lines added and 363 lines removed. Half of that was driver code, 30% for filesystems, 15% for architecture-specific code, and the "really scary one" was the 5% that were changes to core kernel code. That "changes the kernel in ways that make it not Linux", he said.
Kroah-Hartman then went through a long list of drivers that were in Google's changes, but were not upstream. For most of them, there is "no reason this code needs to be out of the kernel". Some of the changes like the pmem driver, which the Google developers have publicly called "voodoo", and a logging infrastructure that should be done in user space, are not good candidates for the upstream kernel, but the vast majority is.
The big problem with all of this code is that it implements features and fixes bugs that others in the community need. Google rewrote the USB gadget code because the kernel version couldn't easily support multi-function gadgets in 2007, but never got it into the mainline. So, Samsung had to fix the mainline gadget code. The Android developers also fixed a "bunch of bugs" in the Bluetooth code, which is a "nasty protocol to get right", but they never pushed it upstream.
There are lots of small changes in various locations that make it much harder for those who want to port Android to new hardware, he said. Missing a three-line change in the inotify code will cause some applications to fail. There is also a special Android-specific /proc file type that needs to be ported into any kernel bound for Android. And on and on.
One of the big problems with much of the Android code is wakelocks, which are an Android-specific power-management feature. There are wakelocks in "every Android driver", but wakelocks are not in the mainline. Kernel developers have tried tearing the wakelocks out of the drivers, but then the code diverges quickly. The right solution is to get wakelocks upstream, which has gotten close to happening, but hasn't.
Wakelocks are a good example of how Google is ignoring the community, Kroah-Hartman said. The wakelock code would be submitted for inclusion, there would be some discussion and then the Android developers would "disappear for six months". When they returned, the process would start over. The solution is to "submit and be persistent", to answer the questions, and participate in the discussion. Wakelocks were "one patch submission away from being accepted", he said, but the developers disappeared again.
Google is allowed to ignore the community, but "it's sad". He took the Android drivers into the staging tree and Google said it would help get them into mainline shape, but they didn't. So Kroah-Hartman had to remove them, which means that no one else gets the benefit of those drivers. When the drivers were in the staging tree, Fujitsu was working on fixing bugs in the Android low memory killer to use on their "big supercomputer stuff", but now that work has been dropped.
The community wants the code, but Google thinks that it's unique and doesn't need to work with the community. "Don't think you are unique, you aren't", he said. He was not picking on Google, because "this applies to everyone". Ironically, the Google server folks would like to use some of the changes that are in the Android tree, but they aren't upstream.
Google's response has been that "it's just a fork, no big deal". Every embedded shop has done a fork and it is abiding by the license, "so who cares?". But Kroah-Hartman wants to see the Android platform succeed and Google is making its partners' jobs harder by making them depend on extra code that needs to be ported to their hardware. In order for the platform to succeed, Google needs to work with the community. If it wants to do it alone, "that's fine", but "they don't and we don't want them to". He noted that Qualcomm, a company not noted for community engagement, had approached the kernel developers complaining about the Android situation, which is a glaring indicator of a problem.
That was the end of the "fire and brimstone" portion of Kroah-Hartman's talk. Looking to the future, he said there is "hope" (accompanied by a slide of the Android logo in the style of Shephard Fairey's iconic Barack Obama "Hope" poster). There will be a meeting very soon where the kernel hackers and Android developers are going to "lock themselves in a room and hash this all out", he said.
For vendors using Android, Kroah-Hartman recommended that they "push on Google to make these changes", and that he will be "pushing as hard as I can on the kernel side". He pointed out that it takes less time to get the code upstream than will be spent maintaining it moving forward. Android is "not an end-of-life project", with 50 new handsets announced at the last trade show. "It's not that much work, truly", he said, and there are plenty of consultants that would be willing to help if the Open Handset Alliance wanted to fund this work.
It was noted that no one from Google attended the talk, and an audience member asked how that could be. Kroah-Hartman seemed a bit disappointed that there were no Google representatives, and did say that they had been invited. Perhaps it was the "long drive from the South Bay" that kept them away, he noted wryly. It may also be that memories of his Canonical bashing from the 2008 Linux Plumbers Conference linger; Google folks may not have wanted to sit through something like that.
The Android user space is "divergent from everything you ever thought about Linux user space", which is "great", he said. The idea that you can run something entirely different on top of the kernel makes it very interesting. Google could replace Linux with BSD or something else and all of the applications would still run. But Kroah-Hartman would like to see "Linux succeed for this market" and Google has followed the kernel license as they should; "I want to reward them for that". He has offered to help before and reiterated that offer. It's clear that he is a big fan of the platform, and is just trying to help fix the problems that he sees.
Few people in the open source community have touched as many projects as Leslie Hawthorn, the now-former open source program manager for Google. As one of less than ten employees in Google's open source programs office, Hawthorn was at the center of the Google Summer of Code — a project that has worked with hundreds of projects and thousands of college students since its inception in 2005. When Hawthorn announced at the end of March that she was leaving Google, we decided to catch up with her and find out what she's learned from her time at Google and what she has planned next.
LWN: A big part of your job has been the Google Summer of Code program. What lessons have you taken away from the program, and what would you do differently if you could reboot the program?
I think the most important lesson I learned was to just get out there and make things happen, no matter how many mistakes you'll make along the way and no matter how many pairs of eyes are watching you make those mistakes. When I started with the team, I had no FOSS experience whatsoever and was quite worried about doing the wrong thing, and very publicly no less. Over time, I realized that it was far better to make as much progress as possible and make the inevitable course corrections required rather than try to get everything right from the start.
I've also learned that a small group of people who are dedicated to making the world a better place can be incredibly successful in doing just that. It may sound like a cliche, but I never imagined when I started working on Summer of Code that I would have the chance to help so many individuals and projects make things better, from finding new contributors to improving their community structure and relations. Granted, having the power of Google as a company and brand behind the program contributed a great deal, but the real power in Summer of Code is the individuals who contribute to making it a successful way for students to or contribute to FOSS.
As for what I would do differently if I could reboot the program, I think it's working quite well the way it is. It would be excellent to scale pay for the student participants according to their respective geographies or vary the program time line a bit more to take into account various school schedules, but that's just not possible with 2-3 people on the Google side managing logistics. I like to imagine how epic Summer of Code could be with a team of ten people working on it, but that's not particularly logical when you consider the overall size of the Open Source Team.
There are times I think that removing the monetary component would be a good thing - just have folks do the program for the recognition and the t-shirt. It would certainly make administration easier and allow more students to participate in the program. However, I also realize that one of the original goals of the program was to provide students with gainful employment in addition to programming experience, and t-shirts, while completely awesome, do not pay the rent or put food on the table.
LWN: In your experience, how effective has GSoC been in drawing new contributors to open source projects?
I think it has been incredibly effective. When I meet up with GSoC students, most of the individuals I speak with had only been using FOSS, but had not contributed patches or fixed more than a few minor bugs. The best aspect of the program, though, is simply spreading the idea of FOSS' importance to the next generation of programmers. The participants in the program become wonderful evangelists for open source, whether or not they remain core contributors to the projects they worked on for GSoC. I think many of the students who have participated in the program will be instrumental in seeing that FOSS is used in the companies that employ them once they enter the workforce.
LWN: How effective has GSoC been for bringing in new code that becomes part of projects?
I think about half the code produced during GSoC ends up merged into project source trees, but that's just an educated guess. Several projects ask their students to work on research implementations of particular features and, as with all software development, some of what is produced ends up not being the most practical solution. On the other hand, I sometimes hear from program mentors more than a year after a particular GSoC that their student's code just entered the mainline. I think the education that the students receive in FOSS and software development in general is more important than the project work they produce, but certainly a great deal of useful code has been written due to GSoC.
LWN: GSoC was probably the most visible part of the open source programs office, as it touched so many projects. What other projects have you worked on that you're happy with?
I'm particularly proud of the Google Highly Open Participation Contest, which was the world's first global initiative to introduce pre-university students to open source. The contest has only occurred once, though I believe Google has plans to hold it again in the future. The effort was incredibly successful - more than 350 students worldwide participated and completed more than 1,000 bite sized tasks for 10 open source projects. The best thing about GHOP was that it focused on all aspects of FOSS, including documentation, user experience research, marketing and advocacy, not just code. That holistic approach to software development made FOSS more accessible to a wider range of young contributors. I'm actually in Vermont right now as I'll be accepting an award for GHOP from the National Center for Open Source in Education on Friday [April 9, 2010].
I'm also proud of launching and managing the Google Open Source Blog, which now has over 17,000 subscribed readers. The blog gave the team the opportunity to talk about all the great work Google does for the open source community. It also gave Google a great venue to showcase all the wonderful things the community was able to do with company's help, from news about GSoC to FOSS development sponsored by Google.
LWN: A big part of GSoC is mentoring. You've obviously observed quite a few projects mentoring students — what tips and advice would you give readers who are participating in open source projects about mentoring?
First and foremost, make it very clear if you are willing to mentor folks on your project. A "help wanted" note on on your project home page is a good start. Don't be afraid to clearly spell out exactly the kind of contributions you are looking for - if your project will only benefit from folks who have advanced experience in machine learning, that's OK. Folks with a less developed skill set in that area will find another good home. It's also great to have someone specifically assigned dealing with newcomers and helping them get up to speed on the basics quickly before handing them over to someone who can help guide them in a particular area in depth.
Share your mistakes with your mentees. It is incredibly easy for someone less experienced to feel like you are some kind of deity and that your hard won knowledge is somehow an innate ability. By showing off some particularly ugly code or talking about your own faux pas on a development mailing list, you give those with less experience more confidence in their ability to go from newbie to seasoned contributor.
Be available as much as you possibly can be, especially when first starting the mentoring relationship. People do well when they feel like their contributions matter and their voice is heard, and they'll feel neither if they don't hear from you early and often. Once the relationship has been established, you can take a bit more time to answer questions, etc., but during the first few weeks being there as much as you can be is incredibly important. It's also worthwhile to be there often so you can steer your mentee to asking the community for help so that he is not solely reliant on your guidance or opinions. As long as your mentee feels like you're there for a particularly complex or troubling issue, she will be more likely to ask others for help on the easier matters and become more integrated into the wider community as a result.
LWN: On the flip side, any suggestions or advice to readers looking to be mentored with projects — especially those who don't have avenues like GSoC?
Don't be nervous about your lack of experience, just dive right in. Check out a project's mailing list for an idea of what's most important right now and what has been important in the past six months or so. Then join the project IRC channel and lurk for awhile. Don't worry if no one acknowledges you or says anything at first. Eventually, when you have an idea of what you'd like to do for the project, say implement a cool feature or host a user group in your home town, then speak up in the channel. Don't ask to ask a question, just ask it. Be prepared for people to not think you'll carry through on your intended efforts - many people get very excited by the idea of a project but don't actually get stuff done, so the folks you talk to will naturally wonder if you fall into that category. This is OK and completely natural - once you've sent in your first patches, updated the documentation or created that project presentation and asked for feedback, you'll see more engagement with your ideas and get more help from the community to further expand on your work.
LWN: You recently gave a talk at LibrePlanet's Women's Caucus about mentoring women in open source. Can you talk a bit about that, and give specific advice on mentoring women for those who are interested in attracting a more diverse group of contributors to projects?
Thanks to Deb Nicholson of the Free Software Foundation, an entire day's track at Libre Planet was devoted to getting more women involved in Free Software. My talk revolved around some basic points for mentoring, a few of which are enumerated above. The points aren't gender specific, but I hear from many, many projects that they are specifically interested in mentoring folks who come from groups underrepresented in their project, particularly women. I think mentoring is particularly valuable to women and other underrepresented groups since the social group that they are approaching doesn't have as many people in it who look like them, literally, and it can be difficult to feel comfortable and welcome in such an environment.
For groups looking to attract a more diverse group of contributors, start with the basics. Go to the members of your community who are underrepresented and ask them what you think can be done to improve the situation. (Note that no woman can speak for all women in FOSS, just like no man could speak for all men in FOSS, so this may be a difficult conversation at first.) Ask them why they were excited about the project in the first place, whether they would recommend working on the project to their colleagues and what the project can do to be most welcoming to newcomers in general. When you hear great ideas, implement them.
Make sure to include women speakers on the agenda of your project's conferences. Women in particular tend not to attend technical conferences if they feel like they are the only woman present. The closer you can come to gender parity on your conference program, the more women you'll attract to your conference. And I think we all know the importance of connecting in person to making FOSS happen.
Another way to help increase diversity in a project is to publish a diversity statement or a mission statement that explicitly states that the project is open to all and encourages participation from as many different kinds of people as possible. It may be obvious to all the project members that you are open to anyone's participation, but actually reading that a project is looking for people from all walks of life makes people feel welcomed and valued. Specifically mentioning that you are looking for newbies is a great way to attract a more diverse community if most of your contributors eat, sleep and breathe your project. More perspectives help a project grow and thrive.
LWN: Women in open source is a topic that's received a lot of attention the past four or five years. Where do you think we're at in terms of interesting women in participating in open source and retaining new contributors?
Please note that I am going to make a whole bunch of generalizations here, and these are just my opinions.
I think things are getting a whole lot better. The conversation is taking a more positive tone - we're not just focusing on the problems anymore, we're giving people concrete advice on how to make things better. A lot of my colleagues have been focusing on providing positive role models for other women to participate and talking about all the reasons why we enjoy contributing to FOSS, all of which helps more women get involved. I've also seen much more discussion about the role that FOSS plays in making the world a better place, which definitely appeals to women; women tend to want to know that their work makes a wider impact rather than just wanting to scratch their own itch.
I think the growth of women in FOSS will really take off in the next few years since more women are stepping up and being vocal about their contributions to the community and their passion for their work. Sharing one's enthusiasm and joy make FOSS that much more attractive to all new contributors, not just women.
LWN: What are the biggest obstacles here? Is it entirely a cultural problem, or are there outside factors at play as well?
I think there are many factors involved. Women tend to be socialized away from technical endeavors, women tend to get their first computer much later than men do, women tend to have fewer technical role models than men do, etc. I think there's definitely a cultural aspect, too; studies have shown that women are less likely to get involved in competitive scenarios and to underrate their skills when in competitive scenarios. The very nature of FOSS is pretty competitive - one solution vies with another for primacy, one often has to be more vocal to get one's opinions heard - and women may be less likely to be comfortable in those situations. I'm not suggesting FOSS development practices should change overall, but easing all new contributors into your community processes will yield better long term results since more people will feel invested in the project if they have a positive initial experience. Save the harshest critiques of patches or the RTFM responses for those who are a bit more advanced in FOSS and I think we'll see many more people stick around and continue contributing, especially women.
LWN: You've been with Google for quite a while, long enough to see it grow from a company where you knew all the engineers to the enormous company it is today. After six years, you've decided to take on a new challenge. Surely you've been offered other positions — anything in particular that made you decide it's time to leave Google now?
Honestly, I just felt like it was time for a change. I loved my role at Google and I love working with the FOSS community, but I wanted to do something different. Consulting allows me to get a wider range of experience in business and I'm looking forward to expanding my skill set.
LWN: You mentioned you'll be working with hackers in Costa Rica. What, specifically, will you be doing? What's the organization you'll be working with?
Heh. I've been idly talking about creating an intentional community for years now and hackers are just my kind of people. I love Costa Rica and the company that I'll be working with in the near term has offices in the Valley and Costa Rica. I'm looking forward to investigating the viability of a "Costa Rican hacker colony" while consulting. Specifically I'll be working on marketing and business development, and consulting gives me plenty of time to continue pursuing my passion for FOSS.
LWN: Beyond Google - how has the open source community evolved in your view in the last decade or so, and is it improving? Any disappointments?
I'm delighted to say that I am seeing the community taking a more expansive role of the ideas of contribution and contributor. Once upon a time, if you couldn't write code you simply weren't needed. I'm now seeing that attitude change and more individuals and projects explicitly state that any and all contributions are to be valued, be they coding, documentation, running a booth at a conference or organizing user group meetings. All of these require different talents and all of these efforts make a project better. The more inclusive we all are at recognizing contributions, the more contributors we'll find donating their time and expertise to FOSS.
Thanks, Leslie, for taking the time to answer our questions.
A media kit is a PDF that describes the site to advertisers (and advertising agencies) in terms that they understand. Essentially, it gives an overview of the site, its content, authors, editors, and, most crucially, its readers. Advertisers are trying to reach decision makers and those who advise them, so they want to understand the demographics of a site's readership in those terms. We have engaged Don Marti to help us create a media kit as the first step towards trying to bring in more advertising revenue—hopefully with fewer ads. Once it is complete, we will be trying to sell advertising to companies in the Linux and free software industries.
We and Don have come up with a survey that we would like our readers to fill out. It combines questions that will be aggregated into the media kit with some others about things like LWN content and your usage of it. We will not distribute any personally identifiable information from your answers, only aggregated statistics. We will also summarize the results of the survey once it has closed, which will be in two weeks. We would really appreciate it if you could take a few minutes and fill out the survey. Thanks in advance from all of us here at LWN.
Page editor: Jonathan Corbet
Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds