An interview with Linus Torvalds [LWN.net]

LWN has been reporting on the Linux and free software community for well over five years now, but, during that time, we've never gotten around to interviewing Linus Torvalds, the creator and maintainer of the Linux kernel. That oversight has now been rectified. In the following interview, Linus talks about 2.5, 2.6, and 2.7, SCO, and how the kernel development process works.

Kernel releases

What are, in your opinion, the most significant accomplishments from the 2.5 development series?

There's a number of them, and the ones I think are most important others may not find as interesting. I personally tend to find "infrastructure" things more important than adding particular drivers support, for example, even though to most _users_ the actual drivers are often more important.

During 2.5.x, the things I thought were most noticeable are a nicer and better VM subsystem, a better block IO layer, and the improved threading support. All of them do help performance in various circumstances, but more importantly (to me) they were all fairly central cleanups and help keep the code maintainable.

Any regrets or things you wish had come out differently in 2.5?

Hey, I always wish we'd do stable releases more often, and I always end up ignoring my own wish and go for "more changes". It makes the release process a lot harder (more things have changed results in more verification and debugging effort), but it does make development more satisfying.

Looking forward to 2.7, do you have any particular goals in mind for that development series?

No, I seldom do. In 2.5.x, we fixed the things I was worried about and that I wanted fixed.

But inevitably, new needs and uses will come up, and I'm not worried about running out of stuff to do. I just don't plan much ahead, I much prefer to take a reactionary stance and see what people actually complain and care about, rather than having a "5-year plan".

Do you have any particular expectations or hopes for the upcoming kernel summit in Ottawa?

To me, the biggest thing is to hear what people are working on or interested in, and get together to just discuss stuff under reasonably organized circumstances. For example, I bet I'll have more of a notion of what people are holding back for 2.7.x..

It took the better part of a year - after 2.4.0 - for the 2.4 series to stabilize sufficiently for the 2.5 fork to happen. Do you foresee doing anything differently to stabilize 2.6 more quickly?

It's always hard to tell. One big problem for the 2.4.x series was the VM, and I think we're _already_ in better shape in 2.5.x than we ever were in 2.4.x. So I'm pretty optimistic, but it's always hard to anticipate everything that pops up when a lot of new people start moving over from 2.4.x to 2.6.x.

Development process

Over the course of 2.5, a number of developers, some of whom have contributed useful stuff, bowed out of the kernel project after facing too much criticism that was too harsh. Do you think this is a "if you can't stand the heat, get out of the kitchen" sort of situation, or could the process perhaps change to be a little more friendly?

I do believe that the kernel mailing list has become too acerbic at times. It can be amusing to read the flame wars if you take them the right way, but not everybody is willing or able to stand back and enjoy the fireworks. It's something I worry about - it tends to be always easier to criticise than to actually fix things.

I'll see what I can do about it, if anything.

There have been complaints that recent development has been strongly oriented toward large-system scalability at the expense of the rest of us with "normal" systems. Over the longer term, however, a high priority has been placed on not allowing support for high-end systems to compromise performance for everybody else. How do you feel about the balance between the kernel's support for large and small systems? Does anything need to be done to ensure scalability to the low end?

I think the drive toward "sexy" systems (lots of power, lots of CPU's, tons of memory, etc) is a fairly natural one, and it's something that gets attention, and I think that's also why people see that development more.

And yes, scalability has improved a lot, but at the same time you should realize that 99% of all Linux development is still done on basic desktop machines. So most developers still care mostly about that kind of hardware, and so while the "big iron" thing gets most attention and is most visible, it's not where most of the action _really_ is.

I personally, for example, always just work with a "high end desktop" system, expecting that what is high end today will be pretty much regular in another year or two.

In many ways, the kernel development process appears to be working better than it ever has. The flow of patches into the mainline is astounding, and most of the major developers seem to be relatively happy. Things appeared rather rougher at the beginning of 2.5; to what do you attribute the improvement? Is it all due to BitKeeper, or are there other things going on?

I definitely think BitKeeper helped, but on the whole people are always happier in development kernels than they are when you have to be careful. So expect some grumbling during 2.6.x when developer frustrations mount, when they can't just go wild.

The lawsuit

SCO has finally fingered some specific contributions to the kernel as, they say, infringing on their rights. Do you think there's a chance that things like RCU and JFS will have to come out before 2.6 can be released? How do you think you might respond if SCO demands their removal?

I don't think it's likely, especially since everything that SCO has fingered as being stuff they object to, they don't actually seem to have any IP rights claim over. They're all stuff written by IBM (or Sequent, which was bought by IBM), and everything looks very much above board as far as IBM goes.

I'd find it very unlikely that IBM had given exclusive licenses to SCO for the thing, especially as IBM apparently used some of the same technology for other projects earlier (ie OS/2). So from what I can tell, SCO really doesn't have a case - at least on the IP side of things.

Whether SCO has a case on the contract side, I just don't know. I'd be surprised. But I don't even have to care, since any contractual issues are clearly between IBM and SCO, and have nothing to do with me or the kernel (and contract law is a whole different area from IP rights, so SCO's blathering about Linux not respecting IP rights seems to be just a rabid rat frothing at the mouth, as far as I can see).

Do you foresee any changes to the kernel development process in the future to avoid the possibility of proprietary code being incorporated?

Hey, I claim that open source is a lot safer than proprietary code bases: people have full visibility in what goes in, and we can go back through archives etc to see who did what. In other words, we already _have_ the process in place to make sure that people don't try to misuse IP rights.

Miscellaneous topics

You've just announced a move over to OSDL, to work full-time on the kernel. Do you have any great plans for your extra time?

I've never had problems with "extra time". As far as I know, "extra time" as a concept is right up there with Santa Claus and the Tooth Fairy.

Recently you have been peppering the kernel with __user annotations which can be used by the "sparse" tool to find improper use of user-space pointers. I've always wondered why the kernel doesn't simply define a "userptr" type which would allow mistakes to be caught by the compiler?

The problem with a "userptr" type is that there is not just _one_ type of user pointer, there are hundreds. User pointer to _what_? You need a bit outside/perpendicular the regular type system, to say it's a "user pointer to a 'struct stat'".

I mentioned that to some gcc people, and nothing ever appeared, so I decided to do it myself.

Would it not make sense to make a similar distinction between physical and kernel virtual addresses?

It could certainly be done, with the tool I already wrote. We've never really had that as a major problem, though. Usually we use "unsigned long" for physical addresses (or things like "page frame numbers", ie they are an index to physical pages, not a whole address). Those have never been directly dereferencable, so we've not had the same kind of "buggy code works by mistake" situation that we've had with user pointers.

Thank you, Linus, for taking the time to answer these questions.

(Log in to post comments)

little systems

Posted Jul 2, 2003 19:40 UTC (Wed) by jimi (guest, #6655) [Link]

Linus has stated previously that scalability goes both ways: large systems and small systems. Software that only scales up isn't very scalable, it must also scale down. So I think he and others are aware of trying to make things work well in very small systems (in fact, newer kernels often have increased performance on the same old hardware).

As for SMP, the locking improves constantly. But the best solution probably isn't to have a kernel running on each CPU. Interrupts from the same device aren't always received on the same CPU. Imagine trying to coordinate interrupt processing between seperate kernels. Fouther problems arise in memory management, particularly on non-NUMA systems (which kernel owns which memory? How do you keep one kernel from stomping on the memory of another? How do you allocate large chunks of memory for an app? Let the kernels fight it out? Does that mean that an app would be tied to one kernel?)

definition of scalability

Posted Jul 3, 2003 23:09 UTC (Thu) by giraffedata (subscriber, #1954) [Link]

>Software that only scales up isn't very scalable

The wording here actually weakens a very good point. There is no such thing as "only scales up." That's like saying a property line only runs East. Scalable means the size can be big or small. If you make Linux work great on a large system and not on a small system, it doesn't scale up. It's always up.

little systems / BIG systems

Posted Jul 2, 2003 22:05 UTC (Wed) by StevenCole (guest, #3068) [Link]

having a kernel run on each proc or quad rather than 1 kernel is a much better way I think

This is probably harder to do than it seems, and it seems pretty difficult. For some light reading, see A Practical Approach to Linux Clusters on SMP Hardware by Karim Yaghmour or Scaling Linux with (partially) CC Clusters by Larry McVoy.

This talk, Linux Scalability for Large NUMA Systems, looks like it will be interesting, along with many others.

Since Jon will be there, perhaps he can provide a nice review.

little systems

Posted Jul 3, 2003 22:55 UTC (Thu) by iabervon (subscriber, #722) [Link]

A lot of the big system things are for reducing overhead, which is important when you've got a ton of processes (e.g.) and limited low memory, but also important when you've got some processes and very little total memory.

Clustering, for now, seems to work fine without changing the mainline kernel. It's possible that a solution will turn up which is more generally applicable, but, presently, building the hardware for a cluster is enough of an effort that the software aspects aren't that big a deal.

A lot of SMP things are now moving to per-processor variables, which gives you a lot of the benefits of per-processor kernels without a lot of the difficulty. Of course, some locks disappear when you compile for uniprocessor, and other locks are actually helpful for handling pre-emption, which is significant for embedded devices and desktops. It's just necessary to put thought into designing locking schemes, and not just throw locks everywhere.

An interview with Linus Torvalds

Posted Jul 3, 2003 0:32 UTC (Thu) by mbp (subscriber, #2737) [Link]

Well, to some extent this can be done by setting different options or merging non-Linus patches. Distributors are probably the right people to do this, not Linus. For example, RedHat give you a different kernel in the desktop-oriented install rather than the server-oriented install, and Gentoo by default uses a 2.4 with patches for better interactive/desktop performance. End users should not need to recompile.

We don't want too much divergence though. Eventually it ought to be possible to have a single codebase that builds for both systems, but as development proceeds different patches can favor one over the other.

An interview with Linus Torvalds

Posted Jul 3, 2003 1:07 UTC (Thu) by StevenCole (guest, #3068) [Link]

My experience with recent 2.5.x kernels for desktop use has been different. After several hours of heavy use, the 2.5 systems remain responsive, while the default kernels of RedHat 9 and Mandrake 9.1 become less responsive and make more use of swap. The mouse with 2.5 is much slower and I have to increase the pointer acceleration in KDE to make 2.5 useable. Your mileage does vary, and it's important that your issues get resolved.

Linus's dual AMD box from two years ago was assembled about 3 blocks from where I lived at the time. Nice machine then, nice machine now. Recently he's been using a 4-way desktop (according to a post on lkml), so he may not notice some regressions which are evident to others.

As Linus pointed out above:

...it's always hard to anticipate everything that pops up when a lot of new people start moving over from 2.4.x to 2.6.x.

Desktop useability will always be one of the most important things. Maybe Linus will announce 2.6.0-test1 at OLS next month, which will likely increase the number of testers substantially. I hope those testers having difficulties will complain loudly and intelligently enough to be heard.

As mbp noted, distributers can and do provide differing kernels for differing purposes, but this can and should only be carried so far. If the core code diverges too far from what is optimal for desktop use, then maintaining those different kernels will become an increasing burden.

An interview with Linus Torvalds

Posted Jul 3, 2003 14:38 UTC (Thu) by proski (subscriber, #104) [Link]

The switch exists already. It's called CONFIG_PREEMPT. If you care about jerky mouse, enable it. If you care about average speed, disable it.

An interview with Linus Torvalds

Posted Jul 4, 2003 1:51 UTC (Fri) by daniel (subscriber, #3181) [Link]

"2.5 is horrible on the desktop - several users (myself included) so far have complained about sound skipping and jerky mouse and window movement"

Con Kolivas' interactivity patch fixes things up a great deal. It's now in 2.5.74-mm (Andrew Morton's kernel):

http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.74/2.5.74-mm1

Window dragging doesn't make sound skip any more. You can still make sound skip by scrolling in Mozilla, but Con's busy tweaking the patch. Before this, ogg playing on 2.5 was definitely painful, now it's tolerable. Expect more improvement in the not too distant future, especially if you keep testing and complaining :-)

An interview with Linus Torvalds

Posted Jul 12, 2003 18:48 UTC (Sat) by Ekdikeo (guest, #12867) [Link]

Try 2.5.75. I just went from 2.5.73 to 2.5.75 and was -totally- amazed by the difference. I had a ton of broken modules before, all of them compiled flawlessly. I couldn't insert any modules before (and yes, I had the correct mod tools) because dependencies couldn't be resolved (i presume because half my modules wouldn't compile) .. they all compile, and insert. it's beautiful. it's like butta.

Also, my memory usage on startup went from like 65MB to 20MB, and after loading X (with fvwm2) I still have about 70MB free on a 128MB system.

The response time is INCREDIBLE compared to 2.4 and previous 2.5 series.

Also, if you're running Debian, make sure that your X package is NOT set to nice the Xserver by -10. That improved response time in 2.4 and previous, but is actually a bad thing for the improved task schedulers in 2.5.