LWN.net Weekly Edition for May 31, 2013

ALS: The open source talent war

Linux Foundation executive director Jim Zemlin kicked off the Spring (at least in the northern hemisphere) edition of the Automotive Linux Summit with some advice for the automotive industry executives and managers in the audience. There is battle going on for developers and other open-source-knowledgeable employees, and it is an important battle to win. There is so much open source development going on that talented people are being snapped up by the competition—not just competitors in the automotive world, but mobile, web, cloud, and other areas as well.

He started off by noting that a typical car now has more than 100 million lines of code in it. Customer expectations are changing all aspects of computing and automotive is just part of that trend. What people see in their mobile phones, tablets, and other consumer electronic devices is creating a demand that leads to a "huge increase in the amount of software in vehicles". That's why the industry is using open source: because it is an essential tool to help meet those expectations.

But just using open source is not enough. In addition, the automotive industry needs more software developers. More software means more developers, he said. In Silicon Valley right now, the demand for developers in the consumer electronics, web, mobile, and cloud areas is huge; there is a war for talent going on. In fact, top developers now have agents, much like Hollywood actors.

Zemlin has learned some lessons about hiring the best programmers in the world. The reason he knows something about it is because he is Linus Torvalds's boss. Torvalds is an incredible programmer who shares some traits with someone else that Zemlin is the boss of: his daughter. Both are adorable, geniuses, and, most importantly, neither of them "listen to anything I have to say", he said to a round of laughter.

More seriously, in working with Torvalds he has learned some things about how to work with the best programmers. There is a big difference between the best programmer and an average one; Zemlin likened it to the difference between a house painter and Picasso. The best programmers can do the work of 100 average programmers, he said.

Five lessons

There are five lessons he has learned about hiring great programmers. The first is to hire developers who play at work; programmers "who goof off". That may sound crazy, but if you look back at Torvalds's original email about Linux it says (paraphrased) "not doing anything big, just something for fun". Torvalds created Linux because it was fun and he still does it today because it continues to be fun. All of the best programmers do their work because they find it fun.

Zemlin mentioned a study from a book called Drive (by Daniel Pink) that looked at what motivates creative people. Since software is a creative profession, it gives insights into our industry, he said. In the study, people were divided into three groups and paid at three different levels (a lot, an average amount, and very little) for doing various tasks. For manual tasks, like factory work, those who were paid the best did the best work. But for creative tasks it was reversed, those who were paid the most did the worst. The study was originally run with students at MIT, but in case the students were atypical, they ran the study again in India: same result.

The "best creative people are not motivated by money alone", Zemlin said. Money is important, but it's not the only thing. What motivates great programmers is to be in an environment where they have the opportunity to "master their craft". Employers should be looking for people who are driven to master the skill of software development, he said.

Hire people who want to give their software away was lesson number two. If you really love what you do, you don't mind giving it away, he said. That leads to an obvious question: how do you make money if you give your software away? There are companies who have figured out how to make money in new ways, which is different from the old way of "keeping everything inside and selling it". He put up a comparative stock chart from 2008 to 2012 for three companies: Red Hat, IBM, and Microsoft. In that time, Red Hat has doubled, IBM is up 85% and Microsoft is flat, so the one who barely gives away any of its software is the one that has seen no gain in its share price. Automotive companies are a lot like IBM, Zemlin said, they don't need to make money on the software, they can make it on products, services, and so on.

Lesson three is to hire developers who don't stick to a plan. He asked: is planning important to the automotive industry? It is, of course, so he is not saying to have no overall plan, but companies shouldn't try to centrally plan everything, he said. Torvalds controls what goes into Linux, but he doesn't have a plan for what comes next. Without a plan, though, Linux seems to do well enough, with 1.3 million Linux-based phones activated daily, 92% of the high performance computing market running Linux, 700,000 Linux-based televisions sold daily, nearly all of the major stock exchanges and commodities markets running on Linux, and on and on.

In software development, it is good to let organic ideas grow, rather than to try to plan everything out in advance. For example, an organic community formed that cared about Linux battery life. That community worked on fixing the power performance of Linux, which turned out to help the high performance computing (HPC) community because most of the cost of HPC is power. So, without any kind of central planning, a problem was fixed that helped more than just those originally interested in it.

"Hire jerks" is Zemlin's fourth lesson. Linux developers "can be kind of difficult", he said, they will engage in flame wars over the code that is submitted to the linux-kernel mailing list. That public criticism was a problem for Japanese and other Asian developers when they first started getting involved. But public criticism actually helps create better ideas, he said.

A 2003 study done at the University of California, Berkeley looked at how people create the best ideas. The participants were split into two groups, and one was told to brainstorm about ideas. That meant that all ideas were considered good ideas and that criticism was not allowed because it might stop the flow of ideas. The other group was told to be critical, to debate and argue about the ideas as they were raised. The group that used criticism was eight times better than the other group; it created more ideas, better ideas, and was far more successful than the brainstorming group. That means "it's OK to be a jerk", Zemlin said, but don't go overboard. The conclusion is that it is important to be critical of others' ideas.

Zemlin's last lesson is that automotive companies should hire a person to manage their external research and development. They should borrow an idea from the consumer electronics companies, Intel, IBM, Red Hat, and others to have someone that helps determine their open source strategy. That person would help decide which projects to participate in, what efforts to fund, and so on. It would be a person who is familiar with open source licenses, who knows how to hire open source developers, and is knowledgeable about how open source works.

"Talent is what is going to make the difference", Zemlin said. Open source is going to "help you compete", but automotive companies have to hire the best software developers. The good news for the automotive industry is that "everyone loves cars". The industry has a reputation for being "sexy" and "exotic", so auto companies can "leverage that position to hire cool developers". He concluded with an admonishment: software development is a talent war, hire the best and you will succeed, but if you don't, your competition certainly will.

[ I would like to thank the Linux Foundation for travel assistance so that I could attend the Automotive Linux Summit Spring and LinuxCon Japan. ]

Comments (16 posted)

The Linus and Dirk show

Linus Torvalds and Dirk Hohndel sat down at LinuxCon Japan 2013 for a "fireside chat" (sans fire), ostensibly to discuss where Linux is going. While they touched on that subject, the conversation was wide-ranging over both Linux and non-Linux topics, from privacy to diversity and from educational systems to how operating systems will look in 20-30 years. Some rather interesting questions—seemingly different from those that might be asked at a US or European conference—were asked along the way.

Hohndel is the CTO of the Intel Open Source Technology Center, and Torvalds "needs no introduction" as Linux Foundation executive director Jim Zemlin said at the outset. Given Zemlin's comment, though, Hohndel asked how Torvalds introduces himself, does he mention that he is a "benevolent dictator", for example. But Torvalds said that in normal life, he doesn't try to pull Linux into things, he just introduces himself as "Linus" (interestingly, he pronounced it as lie-nus) and leaves it at that. He has, he says, a regular life and doesn't get recognized in the streets—something he seemed quite happy about.

Releases and merging

The 3.10-rc3 release was made just before they had left Portland for Tokyo, so Hohndel asked about where things are heading and what can be expected in coming releases. Torvalds said that the kernel release cycle has been "very stable" over the last few years and that there are never any plans for new features and when they will be released. Instead, he releases what is ready at the time the cycle starts. People know that if they miss a particular release with their new feature, ten weeks down the line there will be another merge window for it to get added.

Most of the changes that go in these days are for new hardware, as the core has stabilized, Torvalds said. The changes for new hardware come both in drivers and in support for new CPUs, particularly in the ARM world. Over the last few years, there has been a lot of work to clean up the ARM tree, which was a mess but has gotten "much better". These days, ARM and x86 are the two architectures that get the most attention, but Linux supports twenty or so architectures.

Noting that Torvalds had seemed a little more unhappy than usual recently, Hohndel asked if it was caused by the number of patches he was merging. Hohndel said that the 3.10 merge window was the largest ever, and that the -rc3 announcement showed some displeasure with how things were going. Torvalds said that the size of the merge window was not really a problem, unless the code merged is "untested and flaky". It is a problem when there are a lot of fixes to the newly merged features that come in during the -rc releases. "I want code to be ready" when its merge is requested. Given the ten week release schedule, there are only six or seven weeks to get everything to work, so he is unhappy when people ask him to merge code that makes it harder to converge on the final release. When that happens, it results in "a lot of cursing", he said.

If he gets "too annoyed" at some subsystem or area of the kernel, Torvalds sometimes resorts to refusing to pull code from the developer(s) in question. It is "the only thing I can do", when things get to that point. It is an indication that "you need to clean up your process because I don't want the pain you are causing". Normally that happens in private with a rejection of a pull request in a "try again next time" message, but sometimes he does it publicly. His job is to integrate changes, so he wants to say "yes", which makes it painful for both sides when he gets too frustrated and has to say "no".

Diversity

Hohndel noted that Kernel Summit pictures tend to contain only white males, but he thinks we are making some progress on making the kernel community more representative of the world we live in; "is it improving?", he asked. Torvalds said that he thinks it is improving, but that the Kernel Summit is the "worst possible example" because it mostly represents those who have been involved for 10-15 years. In the early days, Linux was mostly developed in western Europe and the US, which makes the diversity at the summit rather low.

Beyond geographic diversity, the number of women in the open source software world is low, though Torvalds is not clear on why that is. It is getting better through efforts by companies like Intel and organizations like the Linux Foundation to help women get more involved so that the community won't be so "one-sided". He noted that there were few Japanese kernel developers when the first Japan Linux Symposiums started, but that has now changed. Japan (and Asia in general) are much better represented these days.

The first-time contributors to the kernel are more diverse than they were a few years ago, Hohndel said, which is a step in the right direction. There is a problem using that as a measure, though, Torvalds said, because it is fairly easy to do one patch or a few trivial patches. Going from one patch to ten patches is a big jump. There are a lot of people who do something small, once, but that taking the next step is hard. Something like half of all the kernel contributors have only ever contributed one patch. That is probably healthy overall, but looking at first-time contributors may not be an indicator of the makeup of the actual development community.

An audience member pointed out that in addition to the low numbers of women at the conference, there was also a lack of college and high school students. Torvalds said that he didn't find that too surprising, as even those using or developing open source at school probably wouldn't attend a conference like LinuxCon. There is definitely a need for more women participating, though, so hopefully the outreach programs will help there. Hohndel mentioned the Outreach Program for Women, which has kernel internships funded by the Linux Foundation. Sarah Sharp of Intel has been overseeing the program, which has been "extremely successful" in getting applicants. Torvalds said that it had brought in over a hundred patches to the kernel from the applicants.

Education

Another audience member mentioned a university in Switzerland that uses open source software as part of its curriculum, guiding the students to the culture of open source, IRC channels, and the like. Torvalds said that there are other universities with similar programs, which is good. He pointed out that it is "not just about the kernel", which is a hard project to enter, but that other open source projects are good steppingstones to kernel development. Often times, a project needs help from the kernel, so that's how its participants start to get involved with the kernel.

In answer to a question about the differences in educational systems and whether there are specific advantages in learning technology, Torvalds noted that he had first-hand experience with Finland's system and second-hand with the US system through his kids. Finland makes an excellent example, he said, because there is a lot of technology that comes out of a fairly small country "in the middle of nowhere". It has a population of five million and there are cities in Japan that are bigger. In Finland, education is free, so that students don't have to worry about how to pay for it. That means that they can "play around" some rather than just focusing on school work.

Torvalds spent eight and a half years at his university in Finland, and only came away with a Masters degree. In some sense, that's not very efficient, he said, but he worked on Linux during that time. Finland's system gives people the freedom to take risks while they are attending school, which is important. Some will take that freedom to just drink and party, but some will do something risky and worthwhile. In the US, he can't imagine someone going to a good university for that long, because they can't afford to do so. Finland is not necessarily so strict about getting people to graduate in four years, which gives it a "huge advantage" because of that openness and freedom.

Contributing

A non-developer in the audience asked about how he could make a contribution to the kernel. Torvalds was emphatic that you don't need to be a developer to contribute. Part of the beauty of open source is that people can do what it is they are interested in. When he started working on Linux, he had no interest in doing the things needed to turn it into a product. There is documentation, marketing, support, and so on that need to be done to make a product, but he only wanted to write code. He left it to others to do the work needed for making it a product.

Depending on one's interests there are countless things that can be done to contribute. Translating documentation or modifying a distribution so that it works better for Japanese people are two possibilities that he mentioned. Beyond that, Hohndel said testing is a huge part of the process. Running kernels during the development cycle and reporting bugs is a crucial part of making Linux better. But it is not just about the kernel, he said. When Torvalds is on stage the conversation naturally drifts in that direction, but Hohndel said that there are tens of thousands of open source projects out there. People can document, translate, and test those programs. There are a "ton of opportunities" to contribute.

For example, Torvalds and Hohndel work on Subsurface, which is a graphical dive log tool. Torvalds said that it involves lots of graphical user interface (GUI) work that he has "no interest in at all". A GUI designer, even one who can't write code, would be welcome. Creating mockups of the interface that someone else could write the code for would be very useful. Of course, "real men write kernel code", he said. Hohndel chided him by noting that statements like that might be part of the reason why there aren't more women involved. A somewhat chagrined Torvalds agreed: "real men and women write kernel code".

The future

Another question concerned non-volatile main memory and how Torvalds thought that would change computers. First off, there has been talk about non-volatile main memory for a long time and it's always just a few years off, Torvalds said, so he is skeptical that we will see it any time soon. But if it is finally truly coming, he thinks its biggest impact will be on filesystems. For many years, we have been making filesystems based on the block layout of disks, but non-volatile memory would mean that we get byte addressability for storage. That would allow us to get away from block-based organization for filesystems.

For main memory, even if it is all non-volatile, he thinks working memory will still be treated as RAM is today. Non-volatile memory will make things like suspending much easier, but there won't be a lot of externally visible changes. There will still be a notion of long-term storage in files and databases. But the internals of the kernel will change. It will take a long time before we see any changes, even if the hardware is available soon, Torvalds said. It takes a good bit of time before any new technology becomes widespread and common. There is "huge inertia" with existing hardware; people tend to think technology moves faster than it actually does.

What did he think operating systems would look like in 20-30 years was another audience question. "Not much different" was Torvalds's answer. Today's operating systems are fairly similar to those of 40 years ago, at least conceptually. They are much bigger and more complicated, but the basic concepts are the same. If you looked at an operating systems book from the 1970s, today's operating systems would be recognizable in it. The outward appearance has changed, GUIs are much different than the interfaces on those older systems, but much of the rest is the same. "Files are still files", we can just store more of them. There won't be that much change because "most of what we do, we do for good reasons", so throwing that away makes little sense.

Another recent trend is wearable computing, as typified by Google Glass, Hohndel said. What did Torvalds think of that? The idea of a small screen that is always there is attractive to him, Torvalds said, but the problem is not on the output side, it is the input part that is difficult. He hates writing email on his cell phone, so he would love to see some kind of "Google Glassy thing" for input. He hates voice recognition; perhaps it would work for writing a letter or something, but you can't edit source code that way. "Up up up up up" just doesn't work. Maybe someone in the audience will figure out a better way, he said.

The privacy implications of Google Glass don't really bother Torvalds. He said that others (like Hohndel) are much more protective of their privacy than he is. "My life is not that interesting", Torvalds said, and doesn't need to be that private. "All of the interesting things I do, I want to put out there", so there are just a few things (like his bank password) that he cares about protecting. While people are unhappy with Google Glass because it can record anything the person sees, it is something people could already do without Glass, so it's "not a big issue" to him.

Young and stupid

"I was young, I was stupid, and I did not know how much work it would be". That's how Torvalds started his answer to a question about his inspiration to write Linux. He wanted an operating system, and was even willing to buy one, but there was nothing affordable available that he wanted to run. Some are religious about open source, but he is not, and would have bought something if it was available. So he started writing Linux and made it open source because he wanted to "show off" and didn't want to sell it.

Torvalds knew that he was a good programmer, but he also knew he was not a good businessman. If Linux hadn't attracted other people right away, the project probably would have died within six months. But the involvement of others was a "huge motivating factor" for him to continue, because it "made it much more fun". Some people ask him if he regrets making Linux open source, because he could be as rich as Bill Gates today if he hadn't. It's a kind of nonsensical question, because Linux wouldn't be where it is today if he hadn't opened it up, so he wouldn't have reached Gates-level wealth even if he had kept it closed. But there is "no question in my mind" that making Linux open source was the right thing to do.

He has never had a plan for Linux, it already did what he wanted it to do in 1991. But many others in the Linux community did have plans for Linux and those plans were quite different. Some were interested in small devices and cell phones, others in putting it into gas pumps, some wanted to use it in space, still others for rendering computer graphics. All of those different ideas helped shape Linux into what it is today. It is a better way to develop a stable operating system, Torvalds said. When a single company has a plan for their operating system, which changes with some regularity, it destabilizes the system. Linux on the other hand has many companies who know where they want it to go, which has a tendency to keep Linux stable.

[I would like to thank the Linux Foundation for travel assistance to attend the Automotive Linux Summit Spring and LinuxCon Japan.]

Comments (100 posted)

The Open Font License and Reserved Font Names

The SIL Open Font License (OFL) is the dominant software license in the open font community, for a variety of reasons—including the fact that it was written specifically to meet the needs of type designers, as opposed to being an adaptation of another license. But one of its most controversial clauses has long been the Reserved Font Name (RFN) clause, an option that allows the licensor to require any derivatives of the licensed font be renamed. To some, RFNs are an essential tool necessary to cope with the peculiarities of digital fonts, but to others, they are a non-free relic that causes multiple practical problems. Recently, the OFL's authors asked for comments on an update to the official license FAQ, which spawned the latest debate on whether RFNs are ultimately beneficial or harmful in the context of free software, and, if they are harmful, how best to resolve the problem.

The OFL was written by Nicolas Spalinger and Victor Gaultney at SIL International, and specifically attempts to adhere to the Free Software Foundation (FSF)'s definition of free software, the Open Source Definition (OSD), and the Debian Free Software Guidelines. In general, it grants the licensee the right to use, copy, modify, and redistribute the licensed font, with the expected requirements on preserving the copyright notice and not changing the license terms when redistributing.

But it does contain some clauses that are atypical among free software licenses. For one, it requires that the font not be sold as a standalone offering. The FSF highlights this as "unusual," but harmless, since even the inclusion of a simple "Hello world" script satisfies the requirement. The RFN requirement is more strict, requiring a new name to be assigned for any modification, not just for formal or ongoing forks:

Any RFNs claimed by the designer must be specified as such in the copyright statement placed at the beginning of the license. SIL's FAQ goes into additional detail about RFNs, noting that transforming the font into a different format normally does constitute creating a modified version, and that giving a modified version a name that incorporates the RFN (such as Foo Sans or Foo Remade, in reference to an RFN Foo) is not permitted. It even notes that rebuilding the font from source counts as creating a modified version (thus triggering the need for a rename) if the rebuild produces a final version that is not identical to the font as released by the designer.

Nevertheless, a type designer is not required to specify any RFNs when releasing a font under the OFL. SIL encourages type designers to use RFNs, however, citing four reasons in a paper entitled "Web Fonts and Reserved Font Names." First, the paper says, RFNs help to avoid confusing name collisions when a user has both the original and a modified version of a font installed (particularly in application "Font" menus, which offer limited space for displaying font names). Second, they help "protect" the designer from dilution concerns—i.e., broken or inferior derivatives of the font being presented to users with the same name. The third reason is a corollary; sparing the designer of the original font from responding to support requests that actually stem from bugs in a modified version. Finally, an RFN requirement "encourages derivatives" by forcing modifiers to consciously choose a new, distinct name for their derivative fonts.

This paper is a draft, currently posted for public comment. Gaultney emailed the Open Font Library mailing list in May to ask for feedback on the paper, and on a new revision to the OFL FAQ, which adds new questions about deploying OFL fonts in mobile or embedded devices and about bundling OFL fonts in web templates.

Reservations

The historical justification for the RFN clause is that the font name is one of type designers' few ways to distinguish their work from that of the competition (after all, however much metadata may exist in the actual file, an application's "Font" menu still presents the name alone). Requiring derivative works to use a different name is not unheard of in free software (the OSD permits it), but it is rare; as Khaled Hosny pointed out, the LaTeX license also requires renaming. But some in the open font world say that the RFN clause poses an unreasonable burden in practice, especially when seen in light of HTTP-delivered web fonts.

After all, the clause requires explicit written permission from the original designer in order to release a modification that reuses the RFN, and the designer may be difficult or impossible to contact (e.g., a designer who has subsequently passed away). Moreover, the standard for what constitutes a modified version of a font is extremely low; as SIL's draft paper explains in detail, even common practices like subsetting (stripping out unused characters) or optimizing the font for delivery are considered modifications. This standard is certainly lower than the clauses found in most other free software licenses; it is hard to imagine any license requirement being triggered by rebuilding the software from source.

In fact, since optimizing a font for delivery in a web page is considered creating a modified version, then every web server that hosts an OFL font with RFNs must rename its copy of the font or else get prior written permission from the original designer to serve it as-is. Neither choice is particularly appealing; as Dave Crossland said in the thread "it's unreasonable to expect every person publishing a blog who makes their own subset to contact every copyright holder every time they want to use a new OFL-RFN web font." For a popular font, that could quickly add up to tens of thousands of requests. Alternatively, those tens of thousands of servers will deliver the font under a score of other names, and that might not hurt the designer's reputation, but it does little to improve it.

Plan B

To some open font designers, specifying an RFN simply is not worth it, and they attach no RFNs to their fonts. Barry Schwartz argued that it is too much trouble, and the benefits too small; the majority of the free software world managed to cope with the potential of name collisions and misdirected support requests just fine:

But other designers are interested in maintaining the "brand" established by their OFL fonts. Without assigning an RFN, the question remains whether such brand protection is possible. In the email cited above, Crossland advocated using trademarks to protect font names. Trademarks offer similar protections of the font name, and it is a common practice in free software to publish trademark usage guidelines that spell out acceptable uses without requiring prior permission. He pointed to the Subversion project's guidelines, although there are plenty of examples.

Of course, a trademark approach has its problems. How trademarks work varies from jurisdiction to jurisdiction (and may even involve registration fees). Is it also common for a trademark infringement claim to be weakened or denied if the trademark holder demonstrably allows other infringement. Vernon Adams proposed an alternative solution; writing a "preemptive permission" statement to accompany the OFL, which grants permission to modify an RFN font in specific ways—in particular, an "engineering exception" listing the common modifications made when deploying a web font.

Crossland replied that SIL is unlikely to compose such a boilerplate exception, since it would dilute the OFL. Gaultney concurred, adding that writing such an exception would also involve "the basic conceptual difficulty of defining and evaluating what changes would be allowed." Adams, however, disagreed about the notion of diluting the OFL:

It could be a while before there is any resolution to the debate over RFNs. SIL has certainly not expressed an interest in revising the license, which it sees as meeting the desired goals. Both an RFN "engineering exception" permission grant and a trademark usage policy would require careful thinking and writing before deployment. Expecting each type designer to write their own policy is unlikely to bear fruit, but as this debate illustrates, the open font community clearly has many more issues to discuss before it could produce any general consensus on a suite of options. In the meantime, the rest of the free software community might find the discussion informative; as we saw in XBMC's case, trademarks and name collisions can affect software of any stripe.

Comments (12 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: Pondering the X client vulnerabilities; New vulnerabilities in chromium, kvm, moodle, owncloud, ...
• Kernel: IPC and kdbus; Atomic I/O operations; Kernel skiplists.
• Distributions: TDC: A runnable Linux IVI image; Boot to Qt, Fedora, Ubuntu, ...
• Development: IVI audio routing in Tizen; Elpy 1.0; OpenRelativity; GSoc 2013 projects announced; ...
• Announcements: Linux Foundation New Members, events.