Quote of the week [LWN.net]

Be realistic

Posted Oct 12, 2006 23:28 UTC (Thu) by i3839 (guest, #31386) [Link]

Two options:

- Someone is going to backport the new driver to the kernel version you're using. Not only that, but also to all other old versions that people might still be using, or else not everyone is happy. This is a lot of work, not to mention that all those backports aren't as well tested as the new one with the new kernel, so the chance of working is much smaller. It will also get much less testing, so less bugs will be found. All that time and effort is better spend in making sure the new kernel is better than the old one.

- Have a stable binary API for modules. See stable_api_nonsense.txt in your kernel source, or at kernel git http://tinyurl.com/y724vj.

Any better ideas?

I don't see the problem, really. You prefer to hunt for individual drivers instead of having everything work automatically? If you don't want to download all those Mbs to get a few driver, then your distro should package the drivers seperate from the kernel, and automatically download missing drivers when needed (not too hard to do with udev and friends nowaday, as long as the network is running).

And how often does it happen that you get new hardware that isn't supported by your old kernel anyway? Few times a year?

A lot of people are working quite hard to improve the kernel everyday, why would they do that if the new kernel wouldn't be better than the old one, you think? Sure, something might have broken, but no one will know as long as the kernel isn't used and the chance of existing drivers breaking is smaller than the chance that a backported driver breaks. You lose some, you win some, but overall there should be improvement.

Stable API Nonsense nonsense

Posted Oct 13, 2006 0:57 UTC (Fri) by mrshiny (subscriber, #4266) [Link]

See, everyone keeps waving around that "stable API Nonsense" document as
if it's a magic wand. Just because it's written down doesn't make it
right. And just because it has some advantages over another approach
doesn't mean the disadvantages aren't real.

Face it: most people won't or can't update their kernel on a whim. There
are lots of reasons for this. One reason is that, since there are no
stable driver APIs/ABIs, they might have the situation where they need an
out-of-tree driver that doesn't work on the new kernel. Another reason
might be that the new kernel has a bug in an existing driver. Another
reason might be that said new kernel has a completely new scheduler, or
some otherwise potentially disruptive change. And in a mission-critical
environment, these risks are unacceptable.

The kernel devs can wave their magic "stable api nonsense" document around
all they want, but it doesn't change the real world. There's a darn good
reason why other platforms have stable APIs or ABIs. And unlike, say,
Microsoft, the kernel devs can be more ruthless about culling deprecated
APIs; nobody says they have to keep a broken or sub-optimal api around for
ever. But the current approach makes life nearly impossible for
out-of-tree drivers, and having no out-of-tree drivers makes life very
difficult for everyone who doesn't want to live on the bleeding edge.

I'm sorry but YOU can run a just-released vanilla kernel on your
production computers; I have work to do.

As for how often I get hardware that doesn't work on an old kernel? It
happens all the time, actually. Linux drivers often lag the hardware
release. USB has made life somewhat easier since there are more devices
with standard drivers. But the reality is that third-party drivers work
on a pitifully small number of kernels, and new hardware comes out
constantly that requires new or updated drivers. Heck, sometimes it's as
simple as a busted PCI ID in a driver which would be enough to turn an
expensive server into a door-stop. Want to fix that driver? You need a
new kernel. Madness.

Stable API Nonsense nonsense

Posted Oct 14, 2006 1:14 UTC (Sat) by i3839 (guest, #31386) [Link]

The alternative to the stable API nonsense is backport hell. If you know anything better then please say so.

So you're basically saying that running a shiny new, potentially buggy driver for your new hardware is acceptable for your mission critical system? And you buy new hardware on a whim too? Doesn't like a mission critical setup to me. Face it: you can't run any new software on your mission critical system. If you get new hardware and a new setup, then you at least need to test that setup for quite a while before changing to it. You probably also run a 2.4 kernel as that's more stable. Only when you're in a situation where you need some newer features which don't exist in older kernels do you need to upgeade to something newer. Bu then you're screwed anyway, as you can't keep your trusted stability anyway.

On one hand you plead for out of tree drivers, on the other hand you say

"But the reality is that third-party drivers work on a pitifully small number of kernels".

Maybe the fact that they are out of tree is the reason why they are supported for such few kernel versions? And you say that instead of merging them with mainline the kernel should keep its API more stable? Somehow that seems the wrong way around.

Instead of getting the new kernel you can also backport the fix for the driver. Or anyone else. But backporting all fixes to all so called stable versions is way too much work, and gets messy after a while. The question is if all the effort spend in doing that can't be better spend on keeping the latest version more stable, or on writing new drivers. And who's going to do all that backporting work.

Also keep in mind that people are trying to improve the kernel, they aren't just fiddling with it just for the heck of it. You'd want all the bugfixes, and the new features you need. The new features you don't need you don't have to use, so they won't cause instability. Only thing left are general improvements which might cause instability in the short term. But if your kernel panics because of that or because of a bug in a new driver doesn't really matter, does it?

Stable API Nonsense nonsense

Posted Oct 14, 2006 22:31 UTC (Sat) by mrshiny (subscriber, #4266) [Link]

You misunderstand me. I said that third-party drivers work with a
pitifully small number of kernel versions, but this is not a complaint
about the drivers inasmuch as it's a complaint about the kernel. The
reason those drivers don't work with more kernels is not because people
don't want to backport the drivers; it's because the kernel keeps
changing. You shouldn't HAVE to backport drivers.

And your comment about "untested" drivers misses the mark; I could have
two totally tested drivers that have been in production for months or
years but can't be combined on one box because they are for different
versions of the kernel. And even if I'm not running a five-nines server,
downtime is still expensive. If I run my business on my computer, and I
need to add or replace some hardware component that needs a driver newer
than my box, upgrading the whole kernel is a risk. There may be no whim
involved; maybe the old hardware isn't available. A new kernel is
potentially FULL of new bugs, especially in the drivers. I face this
problem all the time with Fedora Core, because they track the upstream
kernel pretty closely, and I've had kernel updates screw my network card
lots of times. Now, this box isn't "mission critical", but thank God, or
else I'd be risking lots of money every time there was a security update
or something.

As for the kernel devs, I understand they are working hard to improve the
kernel. The kernel is a great piece of software. I understand that it's
hard work, the bleeding edge of computer science. But it's not as simple
as "Don't use the new features". If those new features are "new
filesystems" or "new video4Linux2 drivers", fine, they may not impact me.
But there are lots of ways new features in the core could impact me even
if I don't want to use them. And even if the new features are beneficial,
the problem is I have to update EVERY driver I have when I update the
kernel. So I don't get ONE component that's changed, I get lots.

I don't understand why people don't seem to get the whole risk aspect of
this. It's basic software engineering.

Don't get me wrong, the kernel developers are great developers. But they
have chosen a methodology that I disagree with. I'm a software developer
myself, and I understand the pain of maintaining old interfaces when you
have a newer, better one. I know that it's nice to throw away old code.
But when you're writing a piece of software as important as the kernel,
you have a LOT of people depending on you, and the no-stable-API approach
is hostile to third-party developers.

Stable API

Posted Oct 15, 2006 0:05 UTC (Sun) by i3839 (guest, #31386) [Link]

I understand your point. I'm afraid there isn't an easy solution for the problem. The kernel devs want to move along, so don't want to keep the internal API stable. Keeping both the old and new API alive at the same time is problematic too, that may be harder to get right than making the new one bugfree. Having a dev branch and a stable branch of the kernel won't solve your problems either, and that's not what you asked for anyway.

What is your main grudge, that you need to upgrade the whole kernel for new drivers, or that keeping out-of-tree drivers up to date is hard? Probably both, and a stable API would solve both, no wonder you'd like it. ;-)

Would the following solution suit you: A stable kernel version where only bugfixes and new drivers or contained functionality are added, and nothing else.

Any other ideas?

Stable API

Posted Oct 16, 2006 13:17 UTC (Mon) by mrshiny (subscriber, #4266) [Link]

What I would like to see is a return, in part, to the approach used in the 2.2 and 2.4 kernels, where the kernel API was considered stable for the lifetime of the kernel. An API could be deprecated, indicating that it should no longer be used. After some period of time the API would be removed. Obviously this approach was not perfect but at least it allowed for out-of-tree drivers. I think it should be easier to separate the kernel from the drivers. With the current approach there is no deprecation period, and no warning to out-of-tree driver maintainers, thus making out-of-tree drivers "self-flaggelation".

Other projects are able to provide stable APIs (and sometimes ABIs as well) for long-lived releases. This problem is more complex in the kernel but I feel it should still be a design goal. However the kernel devs disagree and claim that not maintaining a stable API lets them improve the kernel faster. I think that if a stable API was seen as a requirement, like a stable user-space API is, they would just find a solution like they always do. Nobody seems concerned that user-space has to be stable for ever, while drivers may have incompatible APIs from point-release to point-release. I'm sure some middle ground could be found, where APIs could be deprecated for some time period and then removed eventually. Whether or not we need a stable branch with no new features, I can't say. I know lots of people would like to see that, but that isn't strictly necessary for a stable driver API or ABI.

Stable API

Posted Oct 16, 2006 21:10 UTC (Mon) by i3839 (guest, #31386) [Link]

As you say, the kernel devs don't have a stable API high on their wishlist. So instead of trying the impossible and let all of them work the way you want, it's more realistic to do the work yourself (with similar minded people), and to keep a stable API branch, one way or the other.

Personally I think it's best to just get all out of tree drivers merged into mainline, no matter how obscure the driver, as that's what most kernel devs seem to want. Also unrealistic, I know. But if they can't handle the flood, then they might change their mind.

Stable API (nonsense)

Posted Oct 22, 2006 6:25 UTC (Sun) by HalfMoon (guest, #3211) [Link]

Other projects are able to provide stable APIs (and sometimes ABIs as well) for long-lived releases. This problem is more complex in the kernel but I feel it should still be a design goal. However the kernel devs disagree and claim that not maintaining a stable API lets them improve the kernel faster. I think that if a stable API was seen as a requirement, like a stable user-space API is, they would just find a solution like they always do.

The www.kernel.org kernel is however not a "long lived release". And the reason it isn't is specifically because that's a complex problem that few of the kernel developers are concerned with.

What they're concerned with is bug fixing, user mode stability, and working on more hardware. That basically precludes the silly model you're promoting: freeze design bugs in concrete, for years, and pile workaround on top of workaround until it becomes unmaintainable.

As has been said before: if you want a stable binary API, go ahead and commit to one. Yourself, and whichever friends join you. (Or perhaps I should say, whoever you can pay -- get to pay? -- you to provide one.)

You're talking like someone who's never actually had to maintain a large software stack with multiple "frozen" layers of interfaces. And where many of those layers were misdesigned, and incapable of functioning well in a number of increasingly important cases ... maybe because they integrate poorly, and will always do so until the interfaces change. Speaking as someone who has been in that boat, I have no sympathy for your whining. If it really bothers you, you have the source code in hand and can try to do a better job. Once you really get into it you'll find that your whines were misplaced.

Just get those drivers merged to the mainstream kernel, then they'll get evolved and updated with everything else. No worries, that way.

Stable API (nonsense)

Posted Oct 22, 2006 18:24 UTC (Sun) by mrshiny (subscriber, #4266) [Link]

For your information, I do develop software and have to deal with stable
APIs and ABIs. And I love how when kernel developers complain that
maintaining an API is hard, and thus they don't want to do it, it's not
called whining, but when I (and others) point out the harm this does to
users, it's called whining.

Stable API

Posted Oct 22, 2006 9:49 UTC (Sun) by Duncan (guest, #6647) [Link]

> What I would like to see is a return, in part,
> to the approach used in the 2.2 and 2.4 kernels,
> where the kernel API was considered stable for
> the lifetime of the kernel.

What you want, it would seem, is the continuing stable 2.6.16.* series
kernels. The stable series kernels have been continuing only for a time,
but that one has a volunteer to continue it for much longer --
indefinitely at this point. It's now a continuing stable kernel with only
fixes (and a few new drivers but nothing interfering with the old stuff
but fixes, is how I understand it) backported.

I'm sorry I don't remember who it was that volunteered, partly because I
like the challenge of sometimes breaking change and run the -rcs and
sometimes daily snapshots so 2.6.16 is ancient history for me, but I do
remember reading the articles on it right here on LWN, so go look it up if
long-time-stable is what you are after.

Otherwise, the feeling is that many of the mission critical folks will
normally be running a distribution catering to that and their supported
kernels anyway, not anything as raw as the mainstream vanilla stuff. For
those that aren't, well, if you don't already have a tested backup server
at /least/ /ready/ to bring up when your "mission critical" one breaks,
and are relying on swapping in new hardware without testing it for a good
30-days first, well, you're obviously not treating it as so mission
critical after all, and if you aren't, why should the kernel devs?

Duncan

Stable API

Posted Nov 1, 2006 17:47 UTC (Wed) by i3839 (guest, #31386) [Link]

It was Adrian Bunk, see:

http://lkml.org/lkml/2005/12/3/55
http://kerneltrap.org/node/7164

Quote of the week

Posted Oct 13, 2006 16:54 UTC (Fri) by job (guest, #670) [Link]

I'm not sure if running just one driver from a completely different kernel together with all your existing drivers from another kernel would be preferable. It's absolutely guaranteed to be an untested combination, and the odds that it'll work without problems is probably lower. I know I would prefer to just install the latest kernel and be done with it.

Quote of the week

Posted Oct 13, 2006 17:24 UTC (Fri) by mrshiny (subscriber, #4266) [Link]

The thing is, by virtue of a driver being out-of-tree, it's always an "untested combination". However, if there were a stable API, this would be less of an issue because the API would (try to) guarantee that a properly-written driver would continue to work. But anyway, with out-of-tree drivers the drivers would be more likely to work with old kernels than new ones, by virtue of the old kernel having been around longer for testing by the driver writer.