out of tree [LWN.net]

out of tree

Posted Aug 4, 2009 8:06 UTC (Tue) by farnz (guest, #17727)
In reply to: out of tree by mikov
Parent article: A tempest in a tty pot

The thing is, and this is from direct personal experience, companies maintaining out of tree drivers don't put their driver through the Linux code review process; as a result, they ignore standard Linux interfaces in favour of inventing their own.

Add to that the fact that they lose interest in maintaining their driver at all when they've moved onto the new shiny (in the case of one vendor I've dealt with, the driver stops being maintained as soon as they have a new chip out, even though they're still selling the old chip), and the fact that I don't have the time to do more than bugfix drivers that have already been ported to new APIs, and I find myself thinking that all the advantages you claim for out of tree drivers are disadvantages from my perspective.

I want one place to go to for drivers. Not twenty trees, but the one tree on kernel.org. I have found it easier to backport drivers from 2.6.30 to 2.6.23 than to integrate vendor drivers, so I would prefer in- aiming to move into the tree (before GregKH's staging area), and that was quite enough pain.

I'm also afraid that your admission that you're stuck on an old kernel due to an out-of-tree driver from your board vendor reminds me of a subset of nVidia graphics card owners, who complain bitterly (regardless of technical reasons) when X.org releases a new version of the X server that isn't backwards compatible with the ancient version of the nVidia drivers that supports their older hardware; I do feel that you should be exerting pressure on your vendor to make this happen, either by getting their drivers into the kernel, or getting them to step up and do the work of maintaining a stable in-kernel API. It's not going to come for free, so why shouldn't the people who want such an API do the work of maintaining it?

(Log in to post comments)

out of tree

Posted Aug 4, 2009 20:51 UTC (Tue) by mikov (subscriber, #33179) [Link]

> The thing is, and this is from direct personal experience, companies
> maintaining out of tree drivers don't put their driver through the Linux
> code review process; as a result, they ignore standard Linux interfaces
> in favour of inventing their own.

I have no opinion about this because I haven't personally experienced it, but I believe you. But why isn't this a problem in Windows?

> Add to that the fact that they lose interest in maintaining their driver
> at all when they've moved onto the new shiny (in the case of one vendor
> I've dealt with, the driver stops being maintained as soon as they have
> a new chip out, even though they're still selling the old chip), and the
> fact that I don't have the time to do more than bugfix drivers that have
> already been ported to new APIs, and I find myself thinking that all the
> advantages you claim for out of tree drivers are disadvantages from my
> perspective.

Agreed, but what is your point? You are describing exactly the problems caused by a lack of stable API. Vendors do not want to maintain drivers for hardware which they no longer sell, because it is a never ending cycle of completely pointless but expensive work (from their standpoint). That would not be the case if there was a stable API.

First, it would be very cheap for them to continue the maintenance, so they would likely do it. Second, even if they didn't, the driver should just continue to work. And lastly, a volunteer can pick it up at any moment.

> I want one place to go to for drivers. Not twenty trees, but the one
> tree on kernel.org. I have found it easier to backport drivers from
> 2.6.30 to 2.6.23 than to integrate vendor drivers, so I would prefer in-
> aiming to move into the tree (before GregKH's staging area), and that
> was quite enough pain.

Well, that is you but you are the exception. 99.9% of the Linux users and most of the developers can't backport a driver. While I personally also could do it, there is more than that - for example I don't really have time or support such backported driver in a business setting.

There is definite value in a central repository for drivers. The problem with the current approach is that the release cycles of all drivers are tied together. Ideally I would like to see a central repository where vendors are free to upload their drivers whenever they want.

> I'm also afraid that your admission that you're stuck on an old kernel
> due to an out-of-tree driver from your board vendor [...]

I have never said that. I mentioned a problem where newer webcam drivers are only available for new kernels. This has nothing to do with in-tree vs out-of-tree drivers; it is precisely a matter of lack of stable API.

It seems that many people who responded negatively to my comments here are confusing a couple of related by separate issues:

* Stable API. It is equally beneficial to in-tree and out-of-tree drivers because in either case it decreases the maintenance burden. In fact contrary to the common wisdom, a stable API gives more freedom to core developers to make changes because they can keep them isolated behind the facade of the stable API.

* In-tree vs out-of-tree drivers. It is true that a stable API decreases the incentive to submit drivers to the kernel because the "free maintenance" red herring is gone. But if people perceive value in having a central point for driver development, there is nothing stopping them.

* Closed source drivers. This is completely unrelated to a stable source level API. They are prohibited by the kernel license and suffer from their well known problems.

> reminds me of a
> subset of nVidia graphics card owners, who complain bitterly (regardless
> of technical reasons) when X.org releases a new version of the X server
> that isn't backwards compatible with the ancient version of the nVidia
> drivers that supports their older hardware;

I would like to see an open source Nvidia driver as much as the next guy, but I have to disagree with you. How did Microsoft manage to keep their graphics driver API the same from NT 3.1 to Windows XP? To think that it is a bad thing is to ignore reality.

> I do feel that you should be
> exerting pressure on your vendor to make this happen, either by getting
> their drivers into the kernel, or getting them to step up and do the
> work of maintaining a stable in-kernel API. It's not going to come for
> free, so why shouldn't the people who want such an API do the work of
> maintaining it?

Agreed in theory. But you are forgetting that these vendors don't need to support Linux at all. It is less than 1% of their market, but it takes more work to support that Windows. So, why the hell would they do it at all?

So, they don't have to do anything. It is not their problem. If the Linux community wants to bring these vendors fully on board, it should make it easy for them. Easier than Windows. The major factor for "easiness" is stable APIs.

out of tree

Posted Aug 5, 2009 10:32 UTC (Wed) by farnz (guest, #17727) [Link]

> The thing is, and this is from direct personal experience, companies > maintaining out of tree drivers don't put their driver through the Linux > code review process; as a result, they ignore standard Linux interfaces > in favour of inventing their own. I have no opinion about this because I haven't personally experienced it, but I believe you. But why isn't this a problem in Windows?

It is a problem in Windows, too; for example, very few graphics card drivers support Windows native screen rotation in XP - you need to use a vendor utility instead. The difference is that Windows users are used to this, and don't complain as loudly about it. It's becoming less of an issue as things like USB standardize more and more device classes, and manufacturers use the standard classes, because that means that the device is Plug And Play under currently shipping Windows systems (Vista, future Windows 7 installs).

> Add to that the fact that they lose interest in maintaining their driver > at all when they've moved onto the new shiny (in the case of one vendor > I've dealt with, the driver stops being maintained as soon as they have > a new chip out, even though they're still selling the old chip), and the > fact that I don't have the time to do more than bugfix drivers that have > already been ported to new APIs, and I find myself thinking that all the > advantages you claim for out of tree drivers are disadvantages from my > perspective. Agreed, but what is your point? You are describing exactly the problems caused by a lack of stable API. Vendors do not want to maintain drivers for hardware which they no longer sell, because it is a never ending cycle of completely pointless but expensive work (from their standpoint). That would not be the case if there was a stable API.

Oops - my bad. I forgot to mention that the case in question was migrating from NT4 to XP in an embedded product. We could still buy the chips in question, but the NT4 driver didn't work in XP, even after we recompiled it against the new DDK (don't know why - I thought there was a stable API?). We ended up having to buy new ASI capture chips, and scrapping the older hardware.

> reminds me of a > subset of nVidia graphics card owners, who complain bitterly (regardless > of technical reasons) when X.org releases a new version of the X server > that isn't backwards compatible with the ancient version of the nVidia > drivers that supports their older hardware; I would like to see an open source Nvidia driver as much as the next guy, but I have to disagree with you. How did Microsoft manage to keep their graphics driver API the same from NT 3.1 to Windows XP? To think that it is a bad thing is to ignore reality.

They didn't keep the graphics driver API the same from NT 3.1 to Windows XP. Newer OSes cannot run graphics cards using older drivers, pure and simple. No different to Linux, in that respect.

And note that vendors do need to bring the Linux community on board - my employer alone buys a significant number of TV capture cards a month, on condition that they are supported under Linux. We pay a slight premium to buy Hauppauge cards because of the Linux support; other vendors are £5/card cheaper, but cost huge amounts of money in expensive developer time keeping the card going.

out of tree

Posted Aug 5, 2009 18:05 UTC (Wed) by mikov (subscriber, #33179) [Link]

It is a problem in Windows, too; for example, very few graphics card drivers support Windows native screen rotation in XP - you need to use a vendor utility instead.

I don't want to defend Windows, but IIRC that is technically wrong. Unless I am mistaken, XP does not support native screen rotation. It was introduced in Vista.

I do agree with your general point that you can't rely on the vendors to "do the right" thing. Vendors do suck. I have seen really really horrible disgusting things working with closed source drivers. But this is an impossible battle. You can't require all developers for your OS to be perfect. People will do whatever they want to do. The best you can do is give them tools encouraging them in the right direction. Plus, hopefully GPL will shame them into better code.

Look at Google Android. They are basically doing whatever they want with the kernel and not bothering to submit anything back.

They didn't keep the graphics driver API the same from NT 3.1 to Windows XP. Newer OSes cannot run graphics cards using older drivers, pure and simple. No different to Linux, in that respect.

I am sorry but that is not quite true either. In a previous life I used to do Windows kernel development, and graphics drivers specifically. I have ported drivers from NT4 to W2K and WXP. The API remained backwards compatible and even binary compatible.

In NT4 they did a fundamental architectural change by moving the drivers into the kernel, but the source level API remained compatible with NT 3.51.

And note that vendors do need to bring the Linux community on board - my employer alone buys a significant number of TV capture cards a month, on condition that they are supported under Linux. We pay a slight premium to buy Hauppauge cards because of the Linux support; other vendors are £5/card cheaper, but cost huge amounts of money in expensive developer time keeping the card going.

I don't quite follow. Can you please clarify a bit? Why do the other cards need huge amounts of money? Also, you are saying that Hauppauge is more expensive because it has to support Linux. So, how is that a good thing?

out of tree

Posted Aug 5, 2009 18:15 UTC (Wed) by johnflux (subscriber, #58833) [Link]

> I don't quite follow. Can you please clarify a bit? Why do the other cards need huge amounts of money? Also, you are saying that Hauppauge is more expensive because it has to support Linux. So, how is that a good thing?

I think his point was that his company was willing to pay extra just for linux support. So Hauppauge was able to charge more money, and still get his custom. Whereas the companies that didn't support Linux didn't get his business.

From a business point of view, if the money that a company loses because of their lack of linux support is greater than the cost of opening the specs or writing the drivers, then the company should write Linux drivers. The poster was providing anecdotal evidence to indicate that they believe that this is the case for some (most?) companies.

On a personal note, I think companies perceive the cost of opening their drivers to be far higher than it really is.

out of tree

Posted Aug 5, 2009 18:45 UTC (Wed) by farnz (guest, #17727) [Link]

Hauppauge cards tend to work out of the box with a recent enough kernel; when they don't, Hauppauge employees are happy to help you make the cards work. Other vendors cards require developer time to make work, and attempts to get the vendor to help tend to be fruitless (they have their own out of tree driver that doesn't work well, due to reinvention of interfaces, and don't understand why you won't simply rewrite your userspace to match their driver).

Developer time is hugely expensive, as we don't have much to spare; given the choice between a small amount of extra money per card (to the vendor), or developer time, we'd rather pay. If you're a vendor, looking to expand, this is a market, and it has money to spend on you.