A gathering of kernel developers

The now-traditional kernel panel, with yet another new set of participants, was held at LinuxCon North America (LCNA) in New Orleans, Louisiana on September 18. The questions ranged from the kernel development process to more personal queries about the kernel hackers on stage. It is a popular session (Linux Foundation Executive Director Jim Zemlin called it his favorite in the introduction) that helps give the audience a glimpse into the personalities of some of those who create and maintain the kernel that underlies their businesses and, perhaps, other parts of their lives as well.

Red Hat's Ric Wheeler moderated the panel, asking his own questions as well as some from the audience. The panel consisted of Sarah Sharp from Intel, who works on USB 3; Tejun Heo from Red Hat, who is mostly working on control groups and resource management these days, he said, but has also done work on workqueues and per-CPU data structures; and Greg Kroah-Hartman of the Linux Foundation who maintains several subsystems and is the stable kernel maintainer. Linus Torvalds rounded out the panel, noting that he no longer did any real work as he had "turned to the dark side": management. He just merges other people's patches these days, he said with a grin.

With much of the focus on embedded devices today, has the kernel moved too far in that direction and away from servers, Wheeler asked. Torvalds said that there was a good balance in the last merge window; while there were lots of commits for "wild wacky devices", there was also a lot of scalability work that was added. Heo mostly agreed, saying that the companies doing the kernel work are allocating their resources to generally maintain the balance. No one seemed to think there was a real problem or concern that servers would be ignored in favor of the latest smartphone.

Getting involved

Wheeler turned to the question of diversity within the community, asking if it had gotten easier for new developers to get involved. He note that Sharp has been leading an Outreach Program for Women (OPW) effort for the kernel and turned to her first. Sharp said that the OPW kernel effort came out of an interest in helping women find a bigger project, with a mentor to assist them in getting up to speed. Seven internships were awarded, and several of the participants were presenting at the conference, she said.

Kroah-Hartman said that he was an OPW mentor this year and was pleased to see some 60 patches from participants in the last merge window, including some to the TTY drivers. He also noted that there is a professor in the Czech Republic that is making "get a patch into the kernel" an assignment for his students. While it was difficult for some, a handful (3-5) said it was "easy" and would continue working on the kernel.

Sharp also pointed out that the documentation for getting started has improved, partly because of the tutorial that she wrote for OPW on creating a first patch and interacting with the community to get it merged. The participants in the program "are doing real work", she said, including speeding up the x86 boot process by parallelizing it.

While the kernel can be difficult to get involved with because of its complexity, it can also be easier to do so because of all of the different kinds of contributions that can be made, Torvalds said. People can contribute drivers, bug fixes, documentation, and so on, which gives more opportunities to contribute than some other open source projects. "Just look at the numbers", he said, since there are patches from more than a thousand developers every release, "it can't be that hard" to contribute.

It really just takes two hours to go into a file and look at it, Heo said, you will likely find things that are "stupid" in the code. That provides plenty of opportunities for new developers. He also noted the rise in the numbers of Chinese developers, which has been noticeable recently and is good to see. Wheeler said that there are few places in the world that don't submit patches to the kernel these days. Kroah-Hartman chimed in that Antarctica should be included in the list as patches had come from there in the past.

The first audience question was a bit whimsical. "Apparently there is an opening for a CEO at Microsoft", Wheeler said to laughter from both audience and panel, was there any interest in the position? The silence (or chuckles) from the panel members made it pretty clear what they thought of the idea.

The airplane seat-back entertainment system seems to be a popular choice for the "most embarrassing place you have seen Linux used". Normally, you only know that it runs Linux because it has crashed, as Sharp pointed out. Torvalds said that he hates seeing the kernel version numbers (like 2.2.18) that sometimes pop up when embedded Linux crashes.

What do you see for Linux beyond your lifetime was the next question up. Kroah-Hartman said that he wanted to see it continue to succeed, which means that it needs to continue to change. "If we stop our rate of change, we will die", he said, because hardware keeps on changing and Linux needs to keep adapting to it. "I can't argue with that", Torvalds said; he hopes that hardware innovation continues its pace.

On the other hand, Heo said that he has no long-term view because he "can't predict reality". He can't see much beyond getting the user-space interface to control groups into a usable form, "after that, who knows?". Sharp said she "would like to see a community that is welcoming to all" to applause.

Kernel and user space

Linux is an ecosystem, Wheeler said, so are there important user-space issues that concern the panel? "That's why we have the Plumbers conference", Kroah-Hartman said. That conference and LCNA overlapped on the day of the panel and Plumbers is an opportunity to put the kernel and user-space developers together to resolve any issues. Torvalds would like to see more kernel developers work in user space. He wouldn't necessarily like to lose them, but it would be worth it to see some of the kernel culture spread to user space. In particular, he complained about APIs breaking with every release; "we know how to do these things" correctly, he said.

But Heo said that the kernel developers could learn from user space as well. There is a need for more communication "between those using our features and us", he said. It would be "beneficial if we talked more". Sharp pointed to power management as one place where collaboration could be better. Linux could have the best power management of any operating system, but it will take cooperation between the kernel and user space.

Another audience question was mostly targeted at Torvalds. It is known that he is a diver and likes conferences that are near good diving locations, so what would be good locations for upcoming conferences? He suggested that "more conferences in the Caribbean" would be desirable. Heo suggested Hawaii as a destination as well. Both were unsurprisingly met with widespread applause.

The only one who responded directly (most just laughed) to the question of "have any of you been approached by the US for a backdoor?" was Torvalds. After a bit of a pause, he bobbed his head up and down in assent, while saying "NO!", which brought another laugh from the packed house.

How the panel members ended up involved in kernel programming was the next question on the agenda. Kroah-Hartman's girlfriend (now his wife) told him about sitting in on a talk with a "strange bearded guy" (Richard Stallman) about free software, which was all new to him. Some years went by and he eventually learned about Linux and got bored when his wife and daughter went away on a trip—so he wrote a driver. In that same vein, Sharp's boyfriend (now husband) was involved with open source rocketry, so she got involved with Linux to work on those rockets.

Torvalds said that lack of money is what drove him to kernel development—creation really. He couldn't afford to buy Unix, nor to buy games (he had to type them in), so he turned to kernel development out of necessity. Heo said that he was unsure why but always wanted to do operating system programming. He is from Korea and universities there did not offer operating system development without pursuing a Masters degree. That led him to Linux, where no one cares what degree you have: "if you can do it, you can do it".

The difficulties of being a maintainer was next up. Kroah-Hartman said that he had a whole talk on what makes his life hard, but it could be summed up as "read the documentation". One of Torvalds's major sources of stress is last minute pull requests—"and I am looking at you James [Bottomley]". Those make him hurry, which he hates to do. If the code isn't ready, just wait until the next release rather than making him hurry, he said.

How to interact with Torvalds (and others, like Kroah-Hartman) is not really documented, Heo said, and it takes six months or a year to come up to speed on that. Beyond that, maintaining a piece of the kernel is "not that hard", he said. For Sharp, getting patches without a justification is one of the hardest problems she deals with. Submitters need to convince her that the patch is actually needed; "why should I care?" If it fixes a bug or adds a feature, the patch should say that. Torvalds emphatically agreed with Sharp; submitters should not just send patches in a "drive-by" fashion, but be ready to answer questions and justify their patch.

Combining two questions, Wheeler asked about what the panel members did in their non-software time and what, if anything, might draw them away from Linux development eventually. Kroah-Hartman joked that Linux was once his hobby, but then he got a Linux job and lost his hobby. Lately he has been building a kayak. Sharp listed several things she likes to do outside of kernel hacking including bicycling, gardening, and fantasy gaming (e.g. Magic: The Gathering and Dungeons & Dragons).

Diving and having a "regular life" occupy Torvalds. He doesn't see anything "coming along that would be more interesting than Linux". Nothing else would "fill that void in my life". Heo is also happy in his job and "could not imagine" doing anything else for work. He has recently moved to New York City, so he spends time "hanging out in the city" and "trying to have a life", he said with a grin.

Working with the community

The final question came out of what is sometimes called "The Greg and Jim show", which is when Kroah-Hartman and Zemlin travel to different companies to talk with them about how to get better at working with the kernel community. Zemlin retook the stage to ask the panel if they had advice for those companies' engineers. Getting involved with the community early in the hardware design phase is important, Kroah-Hartman said. Some companies understand that, to the point where code for an Intel chip that never shipped had to be ripped out of the kernel a few cycles ago, he said.

Torvalds said it goes beyond just looking at the problems that come from your company's new hardware, you need to look at the bigger picture. The "perfect solution" for a specific problem may not be useful for others with similar problems. Features need to be added "in a way that makes sense for other people".

Heo said that it is important for companies to budget extra time and resources to work with upstream. Some think they can "send it and forget about it", but it doesn't work that way. Sharp said that companies need to design their changes more openly; don't design a new API behind closed doors. Instead, working with the kernel developers on the design will help build up the trust, which eases work with the community.

As a closing note, Zemlin not only thanked the panel and moderator, but also reported on a conversation that he recently had. He has spoken with the airline entertainment system company and reassured everyone that it would be upgrading its kernel "soon". With luck, perhaps, that means it will crash less often too.

[ I would like to thank LWN subscribers for travel assistance to New Orleans for LinuxCon North America. ]