LinuxCon Brazil: Q&A with Linus and Andrew

Jim started things off by asking: did either Linus or Andrew ever expect Linux to get so big? Linus did not; he originally wrote the kernel as a stopgap project which he expected to throw away when something better came along. Between the GNU Project and various efforts in the BSD camp, he thought that somebody would surely make a more capable and professional kernel. Meanwhile, Linux was a small thing for his own use. But, in the end, nothing better ever did come along. Andrew added that, as a kernel newbie (he has "only" been hacking on it for ten years), he has less of a long-term perspective on things. But, to him, the growth of Linux has truly been surprising.

How, Jim asked, do they handle the growth of the kernel? Andrew responded that, as the kernel has grown, the number of developers has expanded as well. Responsibility has been distributed over time, so that he and Linus are handling a smaller proportion of the total work. Distributors have helped a lot with the quality assurance side of things. At this point, Andrew says, responsibilities have shifted to where the kernel community provides the technology, but others take it from there and turn it into an actual product.

Linus noted that he has often been surprised at how others have used Linux for things which do not interest him personally at all. For example, he always found the server market to be a boring place, but others went for it and made Linux successful in that area. That, he says, is one of the key strengths of Linux: no one company is interested in all of the possible uses of the system. That means that nobody bears the sole responsibility of maintaining the kernel for all uses. And Linus, in particular, really only needs to concern himself with making sure that all of the pieces come together well. The application of a single kernel to a wide range of use cases is something which has never worked well in more controlled environments.

From there, Jim asked about the threat of fragmentation and whether it continues to make sense to have a single kernel which is applicable to such a wide range of tasks. Might there come a point where different versions of the kernel need to go their separate ways?

According to Linus, we are doing very well with a single kernel; he would hate to see it fragment. There are just too many problems which are applicable in all domains. So, for example, people putting Linux into phones care a lot about power management, but it turns out that server users care a lot too. In general, people in different areas of use tend to care about the same things, they just don't always care at the same time. Symmetric multiprocessing was once only of interest to high-end server applications; now it is difficult to buy a desktop which does not need SMP support, and multicore processors are moving into phones as well. Therein lies the beauty of the single kernel approach: when phone users need SMP support, Linux is there waiting for them.

Andrew claimed that its wide range of use is the most extraordinary technical attribute of the kernel. And it has been really easy to make it all work. It is true, though, that this process has been helped by the way that "small" devices have gotten bigger over time. Unfortunately, people who care about small systems are still not well represented in the kernel community. But the community as a whole cares about such systems, so we have managed to serve the embedded community well anyway.

Next question: where do kernel developers come from, and how can Brazilian developers in particular get more involved? Linus responded that it's still true that most kernel developers come from North America, Europe, and Australia. Cultural and language issues have a lot to do with that imbalance. When you run a global project, you need to settle on a common language, and, much to Linus's chagrin, that language wasn't Finnish. It can be hard to find people in many parts of the world who are simultaneously good developers and comfortable interacting in English. What often works is to set up local communities with a small number of people who are willing to act as gateways between the group and the wider development community.

Andrew pointed out that participation from Japan has grown significantly in recent years; he credited the work done by the Linux Foundation with helping to make that happen. He also noted that working via email can be helpful for non-native speakers; they can take as much time as needed at each step in a conversation. As for where to start, his advice was to Just Start: pick an interesting challenge and work on it.

Open source software, Linus said, is a great way to learn real-world programming. Unlike classroom projects, working with an active project requires interacting with people and addressing big problems. Companies frequently look at who is active in open source projects when they want to find good technical people, so working on such projects is a great way to get introduced to the world. In the end, good programmers are hard to find; they will get paid well, often for working on open source software. Andrew agreed that having committed changes makes a developer widely visible. At Google, he is often passed resumes by internal recruiters; his first action is always to run git log to see what the person has done.

Linus advised that the kernel might not be the best place for an aspiring developer to start, though. The kernel has lots of developers, and it can be kind of scary to approach sometimes. Smaller projects, instead, tend to be desperate for new developers and may well be a more welcoming environment for people who are just getting started.

At this point, a member of the audience asked about microkernel architectures. Linus responded that this question has long since been answered by reality: microkernels don't work. That architecture was seen as an easy way to compartmentalize problems; Linus, too, originally thought that it was a better way to go. But a monolithic kernel was easier to implement back at the beginning, so that's what he did. Since then, the flaw in microkernel architectures has become clear: the various pieces have to communicate, and getting the communication right is a very hard problem. A better way, he says, is to put everything you really need into a single kernel, but to push everything possible into user space.

What about tivoization - the process of locking down Linux-based systems so that the owner cannot run custom kernels? Linus admitted to having strong opinions on this subject. He likes the fundamental bargain behind version 2 of the GPL, which he characterizes as requiring an exchange of source code but otherwise allowing people to do whatever they want with the software. He does not like locked-down hardware at all, but, he says, it's not his hardware. He did not develop it, and, he says, he does not feel that he has any moral right to require that it be able to run any kernel. The GPLv2 model, he feels, is the right one - at least, for him. Licensing is a personal decision, and he has no problem with other projects making different choices.

Another member of the audience questioned the single-kernel idea, noting that Android systems are shipping kernels which differ significantly from the mainline. Jim responded that people who created forked versions of the kernel always come back - it's just too expensive to maintain a separate kernel. Andrew said that the Android developers are "motivated and anxious" to get their work upstream, both because it's the right thing to do and because the kernel changes too quickly. Nobody, he says, has the resources to maintain a fork indefinitely.

Linus cautioned that, while forks are generally seen as a bad thing, the ability to fork is one of the most important parts of the open source development model. They can be a way to demonstrate the validity of an idea when the mainline project is not ready to try it out. At times, forks have demonstrated that an approach is right, to the point that the kernel developers have put in significant work to merge the forked code back into the mainline. In the end, he says, the best code wins, and a fork can be a good way to show that specific code is the best. Rather than being scary, forks are a good way to let time show who is right.

Another audience member asked Linus if he would continue to work on the kernel forever. Are there any other projects calling to him? Linus said that "forever is a long time." That said, he'd originally thought that the kernel was a two-month project; he is still doing it because it stays interesting. There are always new problems to solve and new hardware to support; it has been an exciting project for 19 years and he is planning to continue doing it for a long time. He may have an occasional dalliance elsewhere, like he did when writing Git, but he always comes back to the kernel because that's where the interesting problems are.

Jim described Linus and Andrew as a couple of the most influential people in technology. They are, he said, at the same level as people like Bill Gates, Steve Jobs, and Larry Ellison. Those people are some of the richest in the world. His questions to Linus and Andrew were: "are you crazy?" and "what motivates you?"

Andrew replied that his work is helping people getting what they want done. It is cool that this work affects millions of people; that is enough for him. In typical fashion, Linus answered that he is just the opposite: "I don't care about all you people." He is, he says, in this for selfish reasons. He was worried about finding a way to have fun in front of a computer; the choice of the GPL for Linux made life more fun by getting people involved. He has been gratified that people appreciate the result - that, he says, gives meaning to life in a way that money does not. It is, he says, "taking by giving."

The session ended with a short speech from Jim on the good things that Linus and Andrew have done, followed by a standing ovation from the audience.