An interview with Linus Torvalds

[Posted July 2, 2003 by corbet]

LWN has been reporting on the Linux and free software community for well over five years now, but, during that time, we've never gotten around to interviewing Linus Torvalds, the creator and maintainer of the Linux kernel. That oversight has now been rectified. In the following interview, Linus talks about 2.5, 2.6, and 2.7, SCO, and how the kernel development process works.

Kernel releases

What are, in your opinion, the most significant accomplishments from the 2.5 development series?

There's a number of them, and the ones I think are most important others may not find as interesting. I personally tend to find "infrastructure" things more important than adding particular drivers support, for example, even though to most _users_ the actual drivers are often more important.

During 2.5.x, the things I thought were most noticeable are a nicer and better VM subsystem, a better block IO layer, and the improved threading support. All of them do help performance in various circumstances, but more importantly (to me) they were all fairly central cleanups and help keep the code maintainable.

Any regrets or things you wish had come out differently in 2.5?

Hey, I always wish we'd do stable releases more often, and I always end up ignoring my own wish and go for "more changes". It makes the release process a lot harder (more things have changed results in more verification and debugging effort), but it does make development more satisfying.

Looking forward to 2.7, do you have any particular goals in mind for that development series?

No, I seldom do. In 2.5.x, we fixed the things I was worried about and that I wanted fixed.

But inevitably, new needs and uses will come up, and I'm not worried about running out of stuff to do. I just don't plan much ahead, I much prefer to take a reactionary stance and see what people actually complain and care about, rather than having a "5-year plan".

Do you have any particular expectations or hopes for the upcoming kernel summit in Ottawa?

To me, the biggest thing is to hear what people are working on or interested in, and get together to just discuss stuff under reasonably organized circumstances. For example, I bet I'll have more of a notion of what people are holding back for 2.7.x..

It took the better part of a year - after 2.4.0 - for the 2.4 series to stabilize sufficiently for the 2.5 fork to happen. Do you foresee doing anything differently to stabilize 2.6 more quickly?

It's always hard to tell. One big problem for the 2.4.x series was the VM, and I think we're _already_ in better shape in 2.5.x than we ever were in 2.4.x. So I'm pretty optimistic, but it's always hard to anticipate everything that pops up when a lot of new people start moving over from 2.4.x to 2.6.x.

Development process

Over the course of 2.5, a number of developers, some of whom have contributed useful stuff, bowed out of the kernel project after facing too much criticism that was too harsh. Do you think this is a "if you can't stand the heat, get out of the kitchen" sort of situation, or could the process perhaps change to be a little more friendly?

I do believe that the kernel mailing list has become too acerbic at times. It can be amusing to read the flame wars if you take them the right way, but not everybody is willing or able to stand back and enjoy the fireworks. It's something I worry about - it tends to be always easier to criticise than to actually fix things.

I'll see what I can do about it, if anything.

There have been complaints that recent development has been strongly oriented toward large-system scalability at the expense of the rest of us with "normal" systems. Over the longer term, however, a high priority has been placed on not allowing support for high-end systems to compromise performance for everybody else. How do you feel about the balance between the kernel's support for large and small systems? Does anything need to be done to ensure scalability to the low end?

I think the drive toward "sexy" systems (lots of power, lots of CPU's, tons of memory, etc) is a fairly natural one, and it's something that gets attention, and I think that's also why people see that development more.

And yes, scalability has improved a lot, but at the same time you should realize that 99% of all Linux development is still done on basic desktop machines. So most developers still care mostly about that kind of hardware, and so while the "big iron" thing gets most attention and is most visible, it's not where most of the action _really_ is.

I personally, for example, always just work with a "high end desktop" system, expecting that what is high end today will be pretty much regular in another year or two.

In many ways, the kernel development process appears to be working better than it ever has. The flow of patches into the mainline is astounding, and most of the major developers seem to be relatively happy. Things appeared rather rougher at the beginning of 2.5; to what do you attribute the improvement? Is it all due to BitKeeper, or are there other things going on?

I definitely think BitKeeper helped, but on the whole people are always happier in development kernels than they are when you have to be careful. So expect some grumbling during 2.6.x when developer frustrations mount, when they can't just go wild.

The lawsuit

SCO has finally fingered some specific contributions to the kernel as, they say, infringing on their rights. Do you think there's a chance that things like RCU and JFS will have to come out before 2.6 can be released? How do you think you might respond if SCO demands their removal?

I don't think it's likely, especially since everything that SCO has fingered as being stuff they object to, they don't actually seem to have any IP rights claim over. They're all stuff written by IBM (or Sequent, which was bought by IBM), and everything looks very much above board as far as IBM goes.

I'd find it very unlikely that IBM had given exclusive licenses to SCO for the thing, especially as IBM apparently used some of the same technology for other projects earlier (ie OS/2). So from what I can tell, SCO really doesn't have a case - at least on the IP side of things.

Whether SCO has a case on the contract side, I just don't know. I'd be surprised. But I don't even have to care, since any contractual issues are clearly between IBM and SCO, and have nothing to do with me or the kernel (and contract law is a whole different area from IP rights, so SCO's blathering about Linux not respecting IP rights seems to be just a rabid rat frothing at the mouth, as far as I can see).

Do you foresee any changes to the kernel development process in the future to avoid the possibility of proprietary code being incorporated?

Hey, I claim that open source is a lot safer than proprietary code bases: people have full visibility in what goes in, and we can go back through archives etc to see who did what. In other words, we already _have_ the process in place to make sure that people don't try to misuse IP rights.

Miscellaneous topics

You've just announced a move over to OSDL, to work full-time on the kernel. Do you have any great plans for your extra time?

I've never had problems with "extra time". As far as I know, "extra time" as a concept is right up there with Santa Claus and the Tooth Fairy.

Recently you have been peppering the kernel with __user annotations which can be used by the "sparse" tool to find improper use of user-space pointers. I've always wondered why the kernel doesn't simply define a "userptr" type which would allow mistakes to be caught by the compiler?

The problem with a "userptr" type is that there is not just _one_ type of user pointer, there are hundreds. User pointer to _what_? You need a bit outside/perpendicular the regular type system, to say it's a "user pointer to a 'struct stat'".

I mentioned that to some gcc people, and nothing ever appeared, so I decided to do it myself.

Would it not make sense to make a similar distinction between physical and kernel virtual addresses?

It could certainly be done, with the tool I already wrote. We've never really had that as a major problem, though. Usually we use "unsigned long" for physical addresses (or things like "page frame numbers", ie they are an index to physical pages, not a whole address). Those have never been directly dereferencable, so we've not had the same kind of "buggy code works by mistake" situation that we've had with user pointers.

Thank you, Linus, for taking the time to answer these questions.

(Log in to post comments)

SCO is a rabid rat frothing at the mouth

Posted Jul 2, 2003 5:33 UTC (Wed) by rjamestaylor (guest, #339) [Link]

Worth the price of admission.

An interview with Linus Torvalds

Posted Jul 2, 2003 6:40 UTC (Wed) by jarek (guest, #4105) [Link]

I much prefer to take a reactionary stance and see what people actually complain and care about, rather than having a "5-year plan".
I honestly think this state of mind is what has made Linux the success story it is. It speaks volumes about "satifiying customer needs" as the business world would put it. As far as I see, Linus never wanted to write a Unix clone, he just wanted to HAVE a Unix-like OS to run on his 386.

From my amateur viewpoint...

Posted Jul 2, 2003 8:34 UTC (Wed) by flewellyn (subscriber, #5047) [Link]

...it seems to me that the best developers of software are those who use it themselves. Witness Linux, the GNU system, XFree86, KDE, GNOME, GIMP, etc., etc., etc. The people working on those projects care about what they're doing, because they will use the tools themselves.

On the other hand, I hear tell from spies inside Microsoft that many Windows developers actually use *nix on their home machines. That should tell us something. :-)

From my amateur viewpoint...

Posted Jul 3, 2003 16:07 UTC (Thu) by danielf (guest, #11075) [Link]

I know many Windows admins at the senior level that use Linux at home :)

Windows developers use non-Windows: no scandal

Posted Jul 3, 2003 23:00 UTC (Thu) by giraffedata (subscriber, #1954) [Link]

I don't think it says much that Windows engineers and admins use something other than Windows at home. It says Windows isn't oriented towards computer geeks. That's OK -- there's a huge market for Windows among non-geeks.

Male gynecologists probably develop a lot of things they don't use at home.

little systems

Posted Jul 2, 2003 16:25 UTC (Wed) by johnjones (subscriber, #5462) [Link]

how about 8MB of RAM and 8MB of ROM do people worry about this ?
they need too its what phones run
(e.g. I dont see people doing things like XIP as much as people worry about no. of threads )

and can someone please sort out clustering...
2 best solutions I can see are
being able to move process's over the wire( network or within the local system for mem
locality and resource util)
doing the message thing ( MPI etc )

as more and more SMP systems come about the locking is going to turn into a solaris
nightmare unless people do somthing soon. having a kernel run on each proc or quad rather
than 1 kernel is a much better way I think... Having a solution in mainline kernel would help
things and I belive it's better to have both of these solutions.

regards

John Jones

little systems

Posted Jul 2, 2003 19:40 UTC (Wed) by jimi (subscriber, #6655) [Link]

Linus has stated previously that scalability goes both ways: large systems and small systems. Software that only scales up isn't very scalable, it must also scale down. So I think he and others are aware of trying to make things work well in very small systems (in fact, newer kernels often have increased performance on the same old hardware).

As for SMP, the locking improves constantly. But the best solution probably isn't to have a kernel running on each CPU. Interrupts from the same device aren't always received on the same CPU. Imagine trying to coordinate interrupt processing between seperate kernels. Fouther problems arise in memory management, particularly on non-NUMA systems (which kernel owns which memory? How do you keep one kernel from stomping on the memory of another? How do you allocate large chunks of memory for an app? Let the kernels fight it out? Does that mean that an app would be tied to one kernel?)

definition of scalability

Posted Jul 3, 2003 23:09 UTC (Thu) by giraffedata (subscriber, #1954) [Link]

>Software that only scales up isn't very scalable

The wording here actually weakens a very good point. There is no such thing as "only scales up." That's like saying a property line only runs East. Scalable means the size can be big or small. If you make Linux work great on a large system and not on a small system, it doesn't scale up. It's always up.

little systems / BIG systems

Posted Jul 2, 2003 22:05 UTC (Wed) by StevenCole (guest, #3068) [Link]

having a kernel run on each proc or quad rather than 1 kernel is a much better way I think

This is probably harder to do than it seems, and it seems pretty difficult. For some light reading, see A Practical Approach to Linux Clusters on SMP Hardware by Karim Yaghmour or Scaling Linux with (partially) CC Clusters by Larry McVoy.

This talk, Linux Scalability for Large NUMA Systems, looks like it will be interesting, along with many others.

Since Jon will be there, perhaps he can provide a nice review.

little systems

Posted Jul 3, 2003 22:55 UTC (Thu) by iabervon (subscriber, #722) [Link]

A lot of the big system things are for reducing overhead, which is important when you've got a ton of processes (e.g.) and limited low memory, but also important when you've got some processes and very little total memory.

Clustering, for now, seems to work fine without changing the mainline kernel. It's possible that a solution will turn up which is more generally applicable, but, presently, building the hardware for a cluster is enough of an effort that the software aspects aren't that big a deal.

A lot of SMP things are now moving to per-processor variables, which gives you a lot of the benefits of per-processor kernels without a lot of the difficulty. Of course, some locks disappear when you compile for uniprocessor, and other locks are actually helpful for handling pre-emption, which is significant for embedded devices and desktops. It's just necessary to put thought into designing locking schemes, and not just throw locks everywhere.

An interview with Linus Torvalds

Posted Jul 2, 2003 18:04 UTC (Wed) by torsten (guest, #4137) [Link]

Amazing. You have interviewed the top programmer in my [Linux} world, and you fail to ask the most important question.

Vi or Emacs?

An interview with Linus Torvalds

Posted Jul 2, 2003 18:24 UTC (Wed) by xach (guest, #2349) [Link]

This is in the unofficial FAQ.

An interview with Linus Torvalds

Posted Jul 2, 2003 20:31 UTC (Wed) by johnny (guest, #10110) [Link]

Linus loathes Emacs. I think he uses a modified version of vi.

Emacs vs VI

Posted Jul 8, 2003 5:13 UTC (Tue) by lovelace (guest, #278) [Link]

Actually, according to question 10 of the FAQ which someone else posted, it says he uses
Microemacs. Granted, ME isn't emacs or vi, but it seems to be a lot closer to emacs than vi
which would tend to suggest that he doesn't quite loathe emacs...

An interview with Linus Torvalds

Posted Jul 2, 2003 23:38 UTC (Wed) by Lovechild (guest, #3592) [Link]

2.5 is horrible on the desktop - several users (myself included) so far have complained about sound skipping and jerky mouse and window movement.

And to my horror it seems that the hackers are still aiming for the sky, and everytime someone points out a possible improvement towards desktop users, somebody yells database, server, etc. and the idea is never tested.

Maybe it's time to make a toplevel switch to desktop/server - so we could get the best of both worlds in the source and let the user decide which to use at compile time. This would mean that if DESKTOP is set then CFQ replaces AS as the default IO scheduler, Different process scheduler setting is used, and other tweaking. I would hate for Linus to decide that the desktop was to be left to vendors to handle in their own trees.

An interview with Linus Torvalds

Posted Jul 3, 2003 0:32 UTC (Thu) by mbp (guest, #2737) [Link]

Well, to some extent this can be done by setting different options or merging non-Linus patches. Distributors are probably the right people to do this, not Linus. For example, RedHat give you a different kernel in the desktop-oriented install rather than the server-oriented install, and Gentoo by default uses a 2.4 with patches for better interactive/desktop performance. End users should not need to recompile.

We don't want too much divergence though. Eventually it ought to be possible to have a single codebase that builds for both systems, but as development proceeds different patches can favor one over the other.

An interview with Linus Torvalds

Posted Jul 3, 2003 1:07 UTC (Thu) by StevenCole (guest, #3068) [Link]

My experience with recent 2.5.x kernels for desktop use has been different. After several hours of heavy use, the 2.5 systems remain responsive, while the default kernels of RedHat 9 and Mandrake 9.1 become less responsive and make more use of swap. The mouse with 2.5 is much slower and I have to increase the pointer acceleration in KDE to make 2.5 useable. Your mileage does vary, and it's important that your issues get resolved.

Linus's dual AMD box from two years ago was assembled about 3 blocks from where I lived at the time. Nice machine then, nice machine now. Recently he's been using a 4-way desktop (according to a post on lkml), so he may not notice some regressions which are evident to others.

As Linus pointed out above:

...it's always hard to anticipate everything that pops up when a lot of new people start moving over from 2.4.x to 2.6.x.

Desktop useability will always be one of the most important things. Maybe Linus will announce 2.6.0-test1 at OLS next month, which will likely increase the number of testers substantially. I hope those testers having difficulties will complain loudly and intelligently enough to be heard.

As mbp noted, distributers can and do provide differing kernels for differing purposes, but this can and should only be carried so far. If the core code diverges too far from what is optimal for desktop use, then maintaining those different kernels will become an increasing burden.

An interview with Linus Torvalds

Posted Jul 3, 2003 14:38 UTC (Thu) by proski (subscriber, #104) [Link]

The switch exists already. It's called CONFIG_PREEMPT. If you care about jerky mouse, enable it. If you care about average speed, disable it.

An interview with Linus Torvalds

Posted Jul 4, 2003 1:51 UTC (Fri) by daniel (subscriber, #3181) [Link]

"2.5 is horrible on the desktop - several users (myself included) so far have complained about sound skipping and jerky mouse and window movement"

Con Kolivas' interactivity patch fixes things up a great deal. It's now in 2.5.74-mm (Andrew Morton's kernel):

http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.74/2.5.74-mm1

Window dragging doesn't make sound skip any more. You can still make sound skip by scrolling in Mozilla, but Con's busy tweaking the patch. Before this, ogg playing on 2.5 was definitely painful, now it's tolerable. Expect more improvement in the not too distant future, especially if you keep testing and complaining :-)

An interview with Linus Torvalds

Posted Jul 12, 2003 18:48 UTC (Sat) by Ekdikeo (guest, #12867) [Link]

Try 2.5.75. I just went from 2.5.73 to 2.5.75 and was -totally- amazed by the difference. I had a ton of broken modules before, all of them compiled flawlessly. I couldn't insert any modules before (and yes, I had the correct mod tools) because dependencies couldn't be resolved (i presume because half my modules wouldn't compile) .. they all compile, and insert. it's beautiful. it's like butta.

Also, my memory usage on startup went from like 65MB to 20MB, and after loading X (with fvwm2) I still have about 70MB free on a 128MB system.

The response time is INCREDIBLE compared to 2.4 and previous 2.5 series.

Also, if you're running Debian, make sure that your X package is NOT set to nice the Xserver by -10. That improved response time in 2.4 and previous, but is actually a bad thing for the improved task schedulers in 2.5.

Does he follow LWN?

Posted Jul 8, 2003 17:24 UTC (Tue) by avik (guest, #704) [Link]

One question I missed, do you read lwn?

Can't imagine how he keeps up if he doesn't :)