LWN.net Logo

Quote of the week

Quote of the week

Posted Oct 12, 2006 8:39 UTC (Thu) by Frej (subscriber, #4165)
Parent article: Quote of the week

So what do I do when:
1) buy a webcam
2) find out there is no driver in my 3months old distribution but in upstream kernel

This system works great for the user!

Ofcourse I could have checked first, but come on. A driver that works for real people would be nice.

I'd rather have a crappy driver that does what I need, than a beautifully engineered one that isn't available to me.


(Log in to post comments)

Quote of the week

Posted Oct 12, 2006 9:14 UTC (Thu) by khim (subscriber, #9252) [Link]

Then you go and grab "upstream kernel". Really, it's no big deal. RMS is not joking when he says that "Linux is just a kernel" mantra. This is not a joke, that's reality on one system I'm using (extreme, I know, but there are also RedHat 9.0 with 2.6.17 and few versions of Slackware, it's just the oldest system still in use where I work):
$ uname -a ; cat /etc/redhat-release
Linux srv 2.6.17.6 #1 Fri Jul 21 17:41:50 MSD 2006 i686 unknown
Red Hat Linux release 6.2 (Zoot)

From the user perspective kernel is the collection of drivers. If you need new driver - just install new kernel!

Quote of the week

Posted Oct 12, 2006 15:22 UTC (Thu) by mrshiny (subscriber, #4266) [Link]

What a good idea! That's excellent from a quality assurance perspective. I'll upgrade my WHOLE KERNEL, including ALL drivers, just to get ONE driver. Absolutely NO way that might cause problems. There's ZERO chance that new drivers for my existing hardware might be broken in a new kernel, and ZERO chance that the new kernel might be less stable than the one I already have. Of course! I'm so glad you explained this to us.

Please excuse my snarky attitude but frankly, the whole "just upgrade the whole kernel" thing has got to stop. The kernel devs can do what they want, I'm not paying their salaries, but it's 100% clear that users want and need drivers separate from the kernel, and I, for one, am a little tired of people telling me I don't understand my own requirements.

Be realistic

Posted Oct 12, 2006 23:28 UTC (Thu) by i3839 (guest, #31386) [Link]

Two options:

- Someone is going to backport the new driver to the kernel version you're using. Not only that, but also to all other old versions that people might still be using, or else not everyone is happy. This is a lot of work, not to mention that all those backports aren't as well tested as the new one with the new kernel, so the chance of working is much smaller. It will also get much less testing, so less bugs will be found. All that time and effort is better spend in making sure the new kernel is better than the old one.

- Have a stable binary API for modules. See stable_api_nonsense.txt in your kernel source, or at kernel git http://tinyurl.com/y724vj.

Any better ideas?

I don't see the problem, really. You prefer to hunt for individual drivers instead of having everything work automatically? If you don't want to download all those Mbs to get a few driver, then your distro should package the drivers seperate from the kernel, and automatically download missing drivers when needed (not too hard to do with udev and friends nowaday, as long as the network is running).

And how often does it happen that you get new hardware that isn't supported by your old kernel anyway? Few times a year?

A lot of people are working quite hard to improve the kernel everyday, why would they do that if the new kernel wouldn't be better than the old one, you think? Sure, something might have broken, but no one will know as long as the kernel isn't used and the chance of existing drivers breaking is smaller than the chance that a backported driver breaks. You lose some, you win some, but overall there should be improvement.

Stable API Nonsense nonsense

Posted Oct 13, 2006 0:57 UTC (Fri) by mrshiny (subscriber, #4266) [Link]

See, everyone keeps waving around that "stable API Nonsense" document as
if it's a magic wand. Just because it's written down doesn't make it
right. And just because it has some advantages over another approach
doesn't mean the disadvantages aren't real.

Face it: most people won't or can't update their kernel on a whim. There
are lots of reasons for this. One reason is that, since there are no
stable driver APIs/ABIs, they might have the situation where they need an
out-of-tree driver that doesn't work on the new kernel. Another reason
might be that the new kernel has a bug in an existing driver. Another
reason might be that said new kernel has a completely new scheduler, or
some otherwise potentially disruptive change. And in a mission-critical
environment, these risks are unacceptable.

The kernel devs can wave their magic "stable api nonsense" document around
all they want, but it doesn't change the real world. There's a darn good
reason why other platforms have stable APIs or ABIs. And unlike, say,
Microsoft, the kernel devs can be more ruthless about culling deprecated
APIs; nobody says they have to keep a broken or sub-optimal api around for
ever. But the current approach makes life nearly impossible for
out-of-tree drivers, and having no out-of-tree drivers makes life very
difficult for everyone who doesn't want to live on the bleeding edge.

I'm sorry but YOU can run a just-released vanilla kernel on your
production computers; I have work to do.

As for how often I get hardware that doesn't work on an old kernel? It
happens all the time, actually. Linux drivers often lag the hardware
release. USB has made life somewhat easier since there are more devices
with standard drivers. But the reality is that third-party drivers work
on a pitifully small number of kernels, and new hardware comes out
constantly that requires new or updated drivers. Heck, sometimes it's as
simple as a busted PCI ID in a driver which would be enough to turn an
expensive server into a door-stop. Want to fix that driver? You need a
new kernel. Madness.

Stable API Nonsense nonsense

Posted Oct 14, 2006 1:14 UTC (Sat) by i3839 (guest, #31386) [Link]

The alternative to the stable API nonsense is backport hell. If you know anything better then please say so.

So you're basically saying that running a shiny new, potentially buggy driver for your new hardware is acceptable for your mission critical system? And you buy new hardware on a whim too? Doesn't like a mission critical setup to me. Face it: you can't run any new software on your mission critical system. If you get new hardware and a new setup, then you at least need to test that setup for quite a while before changing to it. You probably also run a 2.4 kernel as that's more stable. Only when you're in a situation where you need some newer features which don't exist in older kernels do you need to upgeade to something newer. Bu then you're screwed anyway, as you can't keep your trusted stability anyway.

On one hand you plead for out of tree drivers, on the other hand you say

"But the reality is that third-party drivers work on a pitifully small number of kernels".

Maybe the fact that they are out of tree is the reason why they are supported for such few kernel versions? And you say that instead of merging them with mainline the kernel should keep its API more stable? Somehow that seems the wrong way around.

Instead of getting the new kernel you can also backport the fix for the driver. Or anyone else. But backporting all fixes to all so called stable versions is way too much work, and gets messy after a while. The question is if all the effort spend in doing that can't be better spend on keeping the latest version more stable, or on writing new drivers. And who's going to do all that backporting work.

Also keep in mind that people are trying to improve the kernel, they aren't just fiddling with it just for the heck of it. You'd want all the bugfixes, and the new features you need. The new features you don't need you don't have to use, so they won't cause instability. Only thing left are general improvements which might cause instability in the short term. But if your kernel panics because of that or because of a bug in a new driver doesn't really matter, does it?

Stable API Nonsense nonsense

Posted Oct 14, 2006 22:31 UTC (Sat) by mrshiny (subscriber, #4266) [Link]

You misunderstand me. I said that third-party drivers work with a
pitifully small number of kernel versions, but this is not a complaint
about the drivers inasmuch as it's a complaint about the kernel. The
reason those drivers don't work with more kernels is not because people
don't want to backport the drivers; it's because the kernel keeps
changing. You shouldn't HAVE to backport drivers.

And your comment about "untested" drivers misses the mark; I could have
two totally tested drivers that have been in production for months or
years but can't be combined on one box because they are for different
versions of the kernel. And even if I'm not running a five-nines server,
downtime is still expensive. If I run my business on my computer, and I
need to add or replace some hardware component that needs a driver newer
than my box, upgrading the whole kernel is a risk. There may be no whim
involved; maybe the old hardware isn't available. A new kernel is
potentially FULL of new bugs, especially in the drivers. I face this
problem all the time with Fedora Core, because they track the upstream
kernel pretty closely, and I've had kernel updates screw my network card
lots of times. Now, this box isn't "mission critical", but thank God, or
else I'd be risking lots of money every time there was a security update
or something.

As for the kernel devs, I understand they are working hard to improve the
kernel. The kernel is a great piece of software. I understand that it's
hard work, the bleeding edge of computer science. But it's not as simple
as "Don't use the new features". If those new features are "new
filesystems" or "new video4Linux2 drivers", fine, they may not impact me.
But there are lots of ways new features in the core could impact me even
if I don't want to use them. And even if the new features are beneficial,
the problem is I have to update EVERY driver I have when I update the
kernel. So I don't get ONE component that's changed, I get lots.

I don't understand why people don't seem to get the whole risk aspect of
this. It's basic software engineering.

Don't get me wrong, the kernel developers are great developers. But they
have chosen a methodology that I disagree with. I'm a software developer
myself, and I understand the pain of maintaining old interfaces when you
have a newer, better one. I know that it's nice to throw away old code.
But when you're writing a piece of software as important as the kernel,
you have a LOT of people depending on you, and the no-stable-API approach
is hostile to third-party developers.

Stable API

Posted Oct 15, 2006 0:05 UTC (Sun) by i3839 (guest, #31386) [Link]

I understand your point. I'm afraid there isn't an easy solution for the problem. The kernel devs want to move along, so don't want to keep the internal API stable. Keeping both the old and new API alive at the same time is problematic too, that may be harder to get right than making the new one bugfree. Having a dev branch and a stable branch of the kernel won't solve your problems either, and that's not what you asked for anyway.

What is your main grudge, that you need to upgrade the whole kernel for new drivers, or that keeping out-of-tree drivers up to date is hard? Probably both, and a stable API would solve both, no wonder you'd like it. ;-)

Would the following solution suit you: A stable kernel version where only bugfixes and new drivers or contained functionality are added, and nothing else.

Any other ideas?

Stable API

Posted Oct 16, 2006 13:17 UTC (Mon) by mrshiny (subscriber, #4266) [Link]

What I would like to see is a return, in part, to the approach used in the 2.2 and 2.4 kernels, where the kernel API was considered stable for the lifetime of the kernel. An API could be deprecated, indicating that it should no longer be used. After some period of time the API would be removed. Obviously this approach was not perfect but at least it allowed for out-of-tree drivers. I think it should be easier to separate the kernel from the drivers. With the current approach there is no deprecation period, and no warning to out-of-tree driver maintainers, thus making out-of-tree drivers "self-flaggelation".

Other projects are able to provide stable APIs (and sometimes ABIs as well) for long-lived releases. This problem is more complex in the kernel but I feel it should still be a design goal. However the kernel devs disagree and claim that not maintaining a stable API lets them improve the kernel faster. I think that if a stable API was seen as a requirement, like a stable user-space API is, they would just find a solution like they always do. Nobody seems concerned that user-space has to be stable for ever, while drivers may have incompatible APIs from point-release to point-release. I'm sure some middle ground could be found, where APIs could be deprecated for some time period and then removed eventually. Whether or not we need a stable branch with no new features, I can't say. I know lots of people would like to see that, but that isn't strictly necessary for a stable driver API or ABI.

Stable API

Posted Oct 16, 2006 21:10 UTC (Mon) by i3839 (guest, #31386) [Link]

As you say, the kernel devs don't have a stable API high on their wishlist. So instead of trying the impossible and let all of them work the way you want, it's more realistic to do the work yourself (with similar minded people), and to keep a stable API branch, one way or the other.

Personally I think it's best to just get all out of tree drivers merged into mainline, no matter how obscure the driver, as that's what most kernel devs seem to want. Also unrealistic, I know. But if they can't handle the flood, then they might change their mind.

Stable API (nonsense)

Posted Oct 22, 2006 6:25 UTC (Sun) by HalfMoon (guest, #3211) [Link]

Other projects are able to provide stable APIs (and sometimes ABIs as well) for long-lived releases. This problem is more complex in the kernel but I feel it should still be a design goal. However the kernel devs disagree and claim that not maintaining a stable API lets them improve the kernel faster. I think that if a stable API was seen as a requirement, like a stable user-space API is, they would just find a solution like they always do.

The www.kernel.org kernel is however not a "long lived release". And the reason it isn't is specifically because that's a complex problem that few of the kernel developers are concerned with.

What they're concerned with is bug fixing, user mode stability, and working on more hardware. That basically precludes the silly model you're promoting: freeze design bugs in concrete, for years, and pile workaround on top of workaround until it becomes unmaintainable.

As has been said before: if you want a stable binary API, go ahead and commit to one. Yourself, and whichever friends join you. (Or perhaps I should say, whoever you can pay -- get to pay? -- you to provide one.)

You're talking like someone who's never actually had to maintain a large software stack with multiple "frozen" layers of interfaces. And where many of those layers were misdesigned, and incapable of functioning well in a number of increasingly important cases ... maybe because they integrate poorly, and will always do so until the interfaces change. Speaking as someone who has been in that boat, I have no sympathy for your whining. If it really bothers you, you have the source code in hand and can try to do a better job. Once you really get into it you'll find that your whines were misplaced.

Just get those drivers merged to the mainstream kernel, then they'll get evolved and updated with everything else. No worries, that way.

Stable API (nonsense)

Posted Oct 22, 2006 18:24 UTC (Sun) by mrshiny (subscriber, #4266) [Link]

For your information, I do develop software and have to deal with stable
APIs and ABIs. And I love how when kernel developers complain that
maintaining an API is hard, and thus they don't want to do it, it's not
called whining, but when I (and others) point out the harm this does to
users, it's called whining.

Stable API

Posted Oct 22, 2006 9:49 UTC (Sun) by Duncan (guest, #6647) [Link]

> What I would like to see is a return, in part,
> to the approach used in the 2.2 and 2.4 kernels,
> where the kernel API was considered stable for
> the lifetime of the kernel.

What you want, it would seem, is the continuing stable 2.6.16.* series
kernels. The stable series kernels have been continuing only for a time,
but that one has a volunteer to continue it for much longer --
indefinitely at this point. It's now a continuing stable kernel with only
fixes (and a few new drivers but nothing interfering with the old stuff
but fixes, is how I understand it) backported.

I'm sorry I don't remember who it was that volunteered, partly because I
like the challenge of sometimes breaking change and run the -rcs and
sometimes daily snapshots so 2.6.16 is ancient history for me, but I do
remember reading the articles on it right here on LWN, so go look it up if
long-time-stable is what you are after.

Otherwise, the feeling is that many of the mission critical folks will
normally be running a distribution catering to that and their supported
kernels anyway, not anything as raw as the mainstream vanilla stuff. For
those that aren't, well, if you don't already have a tested backup server
at /least/ /ready/ to bring up when your "mission critical" one breaks,
and are relying on swapping in new hardware without testing it for a good
30-days first, well, you're obviously not treating it as so mission
critical after all, and if you aren't, why should the kernel devs?

Duncan

Stable API

Posted Nov 1, 2006 17:47 UTC (Wed) by i3839 (guest, #31386) [Link]

It was Adrian Bunk, see:

http://lkml.org/lkml/2005/12/3/55
http://kerneltrap.org/node/7164

Quote of the week

Posted Oct 13, 2006 16:54 UTC (Fri) by job (guest, #670) [Link]

I'm not sure if running just one driver from a completely different kernel together with all your existing drivers from another kernel would be preferable. It's absolutely guaranteed to be an untested combination, and the odds that it'll work without problems is probably lower. I know I would prefer to just install the latest kernel and be done with it.

Quote of the week

Posted Oct 13, 2006 17:24 UTC (Fri) by mrshiny (subscriber, #4266) [Link]

The thing is, by virtue of a driver being out-of-tree, it's always an "untested combination". However, if there were a stable API, this would be less of an issue because the API would (try to) guarantee that a properly-written driver would continue to work. But anyway, with out-of-tree drivers the drivers would be more likely to work with old kernels than new ones, by virtue of the old kernel having been around longer for testing by the driver writer.

Quote of the week

Posted Oct 12, 2006 18:44 UTC (Thu) by bronson (subscriber, #4806) [Link]

khim... You're joking, right?

Quote of the week

Posted Oct 19, 2006 17:25 UTC (Thu) by cbcbcb (guest, #10350) [Link]

> Then you go and grab "upstream kernel"

... and an "upstream udev" (although recently this seems to have stopped being necessary)

Next, you need to decide whether you need any of the new CONFIG options. If you're smart you'll use make oldconfig, but that won't help when you get asked if you want, for example, DMA engine support. (My hard disk etc uses DMA, but is this a new option for that code, or new code for some new device I don't have?)

What happens when ipchains compatibility is removed from Netfilter? I'll have to rewrite my firewall rules, and that is a big deal.

I also need 10's of megabytes of development tools if I haven't installed them already.

Quote of the week

Posted Oct 12, 2006 10:14 UTC (Thu) by shapr (guest, #9077) [Link]

Long ago and far away DOS drivers were the standard and you had to check carefully to see if a piece of hardware would work with Windows 3.1

At some point Windows became popular enough that DOS drivers were rare.

Until hardware sold on the shelves commonly comes with a penguin compatibility logo, it'll be simpler to check for Linux support ahead of time.

C'mon, you know that your make, model, and year of car doesn't have much compatibility even with other years, why would you expect the Windows monoculture to try to be compatible with Linux?

You want a driver that works for "real people"
meaning
you want to be able to walk into a store and buy hardware without having to surf the web to see if it has a Linux driver.

The solution is simple, go ahead and buy the hardware. If it's not supported in Linux, call the company who makes it and complain that they're not supporting your OS of choice, and that the drivers would be written for them if they'd release sufficient specs and docs.

Then take that hardware back and buy that same flavor of device from another company. Repeat until you're happy with the situation.

Windows drivers became common because customers complained to the companies that made the hardware.

Quote of the week

Posted Oct 13, 2006 16:58 UTC (Fri) by job (guest, #670) [Link]

I was actually in a store shopping for a new printer the other day. One model on a shelf had a proud penguin sticker just beside the Works-with-Windows one. I think it was a Samsung. So we're actually closing in on your vision. It made my day anyway...

Quote of the week

Posted Oct 15, 2006 12:04 UTC (Sun) by tlw (subscriber, #31237) [Link]

> One model on the shelf had a proud penguin sticker
> just beside the Works-with-Windows one.

Unfortunately it looks like I'm going to have to un-make your day. :-)

First of all there is no governing body (that I am aware of) which tracks the use of any such "works-with-linux" logos or statements. Vendors are therefore free to make any such claim and slap as many penguin logos on a product without any sort of outside confirmation, testing, or clarification.

In order to put a "works-with-windows" logo or statement on a product, the vendor must download a huge test-bench application, run it, and submit the results to Microsoft. This suite attempts to test the device in many different situations such as high-load, and different sleep states. There are different tests for video cards, printers, hard-drives, and so on. Only after the results have been confirmed can a company make any such claim. Obviously this system has many flaws too, but for some people is slighly more reassuring than taking the vendor's word; the vendor is in a conflict of interest here.

Secondly, just because the vendor claims a product works with Linux does not mean it works with your Linux setup or the one you intend to use with their product. It just means the vendor allegedly got this product to work with whatever version of Linux they had floating around the shop internally.

Have you actually ever bought a product with a penguin logo thinking that it would work on your Linux setup? Don't be surprised if you should open the packaging and find a little slip of paper that says in order to make this device work on Linux you have to compile an out-of-tree module found on the CD yourself. Oh yea, the module was written against 2.4.15, and the driver code is just a wrapper which loads a binary blob... and that's if you're lucky. Sometimes all you get is a URL to an un-named IP address where you'll find a binary module quietly whispering "install me, I dare you, I promise I won't do anything naughty".

But hey, they got it working with 2.4.15 so they put a penguin on the box. Good luck getting it to work with the SMP system you bought last week running the latest Linux kernel!

Putting a penguin on a box means nothing because nobody's regulating it. The presence of the penguin has no implied legal meaning or promise. It can mean everything from "there is an in-kernel open-source driver which is actively maintained for this product" to "we haven't even bothered to see if it works with any known flavour of Linux". It's true! If some bean-counter told the vendor they could improve sales 10% by putting a penguin logo on the box the vendor's first question is: "How much does it cost to put that logo on the box?". The answer would be: "Nothing. There are no test suites to pass, no governing body to pay dues to, and no legally implied meaning." How many sleepness nights do you think the vendor will endure?

Who knows, maybe some day the penguin logo on vendor's boxes will become such a sham the Linux community will be forced to regulate it in order to head-off the negative publicity generated by hundreds of vendors shamelessly slapping penguin logos on anything they sell and the resulting disappointed Mom&Pop Linux users suddenly finding themselves with a product they've paid lots of money for but can't get working until they figure out how to compile and load a kernel module.

Works with Linux sticker

Posted Oct 21, 2006 11:53 UTC (Sat) by anton (guest, #25547) [Link]

In this concrete case this most likely means that the printer supports Postscript, no special kernel or CUPS drivers needed.

Quote of the week

Posted Oct 12, 2006 12:09 UTC (Thu) by pizza (subscriber, #46) [Link]

More to the point, what happens if an out-of-tree driver isn't mergeable and won't likley ever be merged?

There are many legitimate reasons for this; including many cases where "just use a new kernel" is NOT an option.

One driver I maintain currently has to work with everything from 2.4.0 through 2.6.18.

Quote of the week

Posted Oct 12, 2006 13:18 UTC (Thu) by ebiederm (subscriber, #35028) [Link]

Well if a driver isn't mergeable the most likely reason is the code quality is such that it should not be run anyway.

In most cases backports can be handled by a simple compat_mac.h that defines the new apis on top of the old kernel.

Even if you have to maintain the backports merging a driver into the new kernels is a good idea so other developers can see what kind of things your driver relies on, so they don't remove the functionality you depend on.

This also makes it easier for new users to get your driver as after a point the driver is available in the kernels all of the distros ship and you don't have to do anything.

Beyond that David's comment about flagellation is apt. You may choose to do it but the price you pay for not working with the other developers is that they don't work with you.

Quote of the week

Posted Oct 12, 2006 14:29 UTC (Thu) by pizza (subscriber, #46) [Link]

No; it's not mergeable because it is structurally considerably different from the HolyWayOfLinux, and to make it AcceptableForMerging would strip away most of the reasons for using it (rich userspace configuration API), not to mention a great deal of work for little apparent gain.

Meanwhile, I don't think the kernel people go out of their way to make out-of-tree drivers difficult to maintain ("don't care one way or another" is a good way of putting it), but dismissing all out-of-tree drivers as somehow inferior, unsuitable for use, or "self-flagellating at best" is grossly unfair, for the precise reasons that Linus mentioned:

> it's the damn drivers maintained outside the tree, and
> people who want to maintain the same driver for multiple different
> versions of the kernel.

If your driver needs to support multiple kernel versions, then you need to maintain it out of tree, whether or not there is an in-tree variant to keep track of as well.

While there is an extensive compatibility header already in place, there are still many bits and pieces littered all over the codebase for things that are simply incompatible. Even within the 2.6-series, for example, pcmcia changed radically for both 2.6.16 and 2.6.17.

Quote of the week

Posted Oct 12, 2006 18:34 UTC (Thu) by Thalience (subscriber, #4217) [Link]

I think "self-flagellating" was meant to describe the self-inflicted pain of maintaining the driver out-of-tree, and not using it.

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds