LCA: Andrew Tanenbaum on creating reliable systems

The problem with that sentiment, and the whole article, is that it focuses solely on the kernel.

I don't think I've had more than 3 or 4 Linux failures in my life, and most of those were when using very new drivers (or NVIDIA).

I have had X crash or lock, various GNOME and KDE components crash or lock, various regular applications crash and lock more times than I can possibly count. Definitely into the triple digits, if not quadruple by now.

If you take Tanenbaum's suggestion to heart, the 5-10% "penalty" of the micro-kernel design is irrelevant, because you won't just be swapping in a micro-kernel underneath the bloated, unreliable layers we've built on top of Linux. You'll be building an entire new system, bottom to top, with less bloat and more reliability. Will that total system have a 5-10% penalty over my current system? I doubt it. You can't even *begin* to speculate, because there are just far, far too many variables to really judge that.

Minor problems in sight here...

• To get decent performance out of current hardware the system has to be able to shove large amounts of data in one go. Bye, bye "Use IO to do data movement"
• The whole "serialize request, send it over, unserialize, check, act, serialize results, send them back, unserialize, check" business is costly on current hardware. The relative cost of context switches is going up, so this isn't getting cheaper.
• I just fail to see how microkernels (which require each separate part of the system to be able to handle multiple requests simultaneously (if you don't want to make even a single user system unbearably slow) can win over handling the whole synchronization problem once
• Once everybody has to keep enough state to know how to restart requests that failed because the server crashed, we are in a whole new universe of pain. The tendency is exactly in the opposite direction: TCP is so nice because it handles all sorts of problems in the underlying net transparently. Few people are able to write software that is able to handle random failures, that is the reason why highly-reliable software is so expensive to produce.

A nice pipe dream.

Yes, I know that way back when people resisted compilers for the fear of loosing complete control over the machine, and getting slower programs. I do know that with today's plummeting hardware costs and balloning capabilities, and the ever better compilers and subtly changing hardware underneath, that it is madness to write complete programs in assembler. Maybe awt's time will come, but not in the near future.

If you send the wrong command you will deadlock the part and the computer will need to be reset.

Computer ? Probably not. GPU ? Absolutely. It can be done without any hardware redesign. ATI drivers for Windows are doing it (not so sure about Linux ones).

What we really need is a better MMIO controller such that devices can have multiple privilege rings (or capability tokens); with that in place, it could be made *impossible* for devices to DMA into memory other than that the CPU wants it to DMA into.

But as far as I know nobody has written such a controller, let alone put it in any sort of affordable hardware. I'd be overjoyed to be corrected.

High performance software relies on tricks which are, for the most part, quite unsafe. Your 3D game only works because the GPU is allowed unchecked access to main memory. If you were to start being careful about that access, performance will suffer.

There's always a performance/reliability tradeoff. But if ask the users of *most* systems -- outside of the limited scope of 3D gaming -- you'll find that reliability is more important, as it should be. And we're not just talking about desktop PCs: it's also about the embedded devices that we're all putting more trust into these days.

It's hard to focus on performance first and reliability later -- I've tried this in my own software, and what usually happens is the bug fixes and redesigns to address instability blow away all of the performance gains you started with. It's much easier as a developer to get it right first, and worry about the performance second. Sure, it's a close second, but it's still second.

Tanenbaum's points should be first discussed and debated on their own merits, regardless of performance implications. Do these principles really guarantee higher reliability? Are microkernels the best way to implement these principles? Trying to address performance concerns at the same time only complicates things.

Your 3D game only works because the GPU is allowed unchecked access to main memory.

As far as I can tell, this statement alone is incorrect. All graphics cards produced since AGP become widespread, are now using an IOMMU called GART, which ought to stop the GPU from making memory requests to bad addresses in the main memory.

So running microkernel for reliability and then for pine is yeah nice joke :)

BSDL has proven to facilitate intra-projects which rather are in their own loop and not easily doing I/O in terms of development, and a stance like "if you were my student" doesn't help to create real following.

I guess it's sort of "QNX going opensource" might change things, but not for the mass market (where we see de-appliancization of mobile phones, which can hang or crash these days -- and drain battery in a day).

Maybe it's continuation of "CIO's problem with Linux", namely "nobody to blame". If we use flakey software like Windows (as Tannenbum does), we have lots of code by others to blame; if it's inherently reliable (and running on proper hardware, as one can see it's yet another problem) then it's clearly us to blame, and we don't like that very much deep inside.

---

One might start with actually enjoying the day job, particularly its high-quality, useful results which should not require arcane marketing to put down someone else's throat; and not resorting to the entertainment industry out of frustration... to sort of feel the scale of the problem.

And you get quite a performance hit over the "unixy" way of using it. But it could be a lot more secure.

The idea of the 432 was to close the semantic gap, by directly supporting high-level language constructs in hardware. The VAX architecture had similar ideas, though the VAX didn't take things as far. The problem is that you pay through the nose for this stuff, and there's no way to avoid the penalty, even in cases where the compiler can easily prove that the condition the hardware is protecting against can't happen. Complex microcode is needed to handle all the possible corner-cases.

The end result is that even though the hardware has specialized stuff to handle the complex language constructs, much simpler hardware beat the pants off of heavyweight monstrosities, with vastly less silicon area and power.

It's not that hard to sell reliability. Mobile phone manufacturers for one are getting very worried about the implications of the huge amount of code on their phones. Which is why they are funding things like the formal proof of the L4 microkernel.

Most ISPs are deeply interested in the stability and uptime of their routers and switches. One of the great disappointments of Cisco's IOS XR is that although it runs on the QNX microkernel it doesn't exploit that and runs huge processes that need to be restarted to have bug fixes applied.

The talk was timely because it refocusses the discussion on to building the best computer possible, rather than merely building an operating system better than Microsoft's.

Linus in 30 years...

Andrew Tanenbaum and Linus buried the hatchet long ago

I don't think they were ever close to bearing hatchets in each other's company, but I was sitting in the row behind Linus in the talk, and I noticed that his applause was not the most heartfelt at the end of Andrew's talk. I don't expect a microkernel from Linus in the next year.

That's obviously relative to microkernel advocates, they don't go away no matter how much people criticise them. :)

>but if you think about what is the main reason to use microkernels you have to agree with Tanenbaum. Security, robustness, ..., yes, Linux is very stable and secure but the cost to get it is too high and just minor changes in the kernel can crash the system. OK, microkernels have other drawbacks, but my opinion is operating systems will follow the approach sooner or later. Look at virtualization technology: it is not just to save money, it is for security too. Some security protocols does not allow to run critical applications with others normal ones, but virtualization enables to do that in the same machine, but not in the same OS. Microkernels have the same goal, but here the execution domains are not full isolated, but they are full protected.

I didn't understand how Tanenbaum's design and robustness features were supposed to help so much.

What was shown was that he was able to kill a really simple IDE driver and have it recover, in his test environment (as opposed to hitting a real bug).

Minix apparently can't address issues where the hardware isn't being programmed in exactly the right way. I think this is where a lot of Linux driver bugs come from. These bugs are far worse than a simple driver crash, because your data can get trashed or lost.

It also doesn't address the issue of the wrong bit of data being written, or data being written in the wrong place. Andrew claimed these are usually caught very early in alpha testing. I'm sure this is true for a simple block device driver. I can't say the same would be true for something with one or two orders of magnitude more complexity, like an advanced filesystem.

From what I see, most bugs in Linux are not caused by one kernel component stepping on another, so microkernel protection won't help much; nor by simple mistakes that are easy to detect and/or recover from. So what do minix's tricks do? Hide the most mundane 10% of the bugs encountered, maybe. I can't see how it would reduce the amount of bugs by even one order of magnitude. In fairness it is a work in progress, so let's see what happens with it.

Actually I'm sure that if such a design could be built that meets all the promises then it would be widely used, and Linux would probably adopt ideas from it. It is also great to have such smart people like Tanenbaum researching all these different ideas and making these interesting kernels.

But is minix any less an unproven research toy now than it was during the Torvalds vs Tanenbaum debate?

So all talk about a stable kernel are great, but the real problem lies elsewhere.

I hope that in this time where PCs and handheld consumer devices are converging in functionality, the software developers will recognize how bloated and slow their software is, and do something about it.

I recently created a DVD with some simple menus using dvdauthor. It worked fine with Oggle. It worked fine on my $25 DVD player. It worked fine on my sister's DVD player in France. It worked fine on my mother's neighbour DVD player. It does not work on my mother's Philips DVD player. Only God knows...

Go to any web site dedicaced to the very subject of DVD players and you will find that this model plays this but not that, etc...

http://preview.tinyurl.com/qhuhg

>if the file system crashes while my word processor is saving, it needs to catch the fault and try saving again.
This is not necessarily the word processor responsibility to retry the action: after all, the word processor did ask the OS to write some data on the disk. Whether it took one or two tries for the OS to do it doesn't really matter to the word processor..
Don't you remember 'KHB: Recovering Device Drivers: From Sandboxing to Surviving' from the week before?