The sad story of the em28xx driver

By Jonathan Corbet
November 11, 2008

Over the last year or two, the kernel development process has been changed in a deliberate attempt to make the addition of new drivers easier. It has become clear that out-of-tree drivers often do not get any better until they are merged; meanwhile, users want those drivers and distributors are shipping them. So it would seem that everybody's interests are served by getting those drivers into the mainline tree. Experience with drivers merged under this policy has generally been positive; once those drivers head for the mainline, they get more attention and tend to improve quickly.

Given that, one might well wonder why Markus Rechberger's recently submitted "empia" driver series is encountering so much resistance. This driver works with a number of video acquisition devices based on Empia chips; many of those are not supported by the kernel now. As an Empia Technology employee, Markus has access to the relevant data sheets and is, thus, well placed to write a fully-functional driver. There are users who will attest that the drivers work, and that Markus provides good support for them. But, as things stand now, it would appear that this driver is not headed for the mainline.

What we have here is a classic story of an impedance mismatch between a developer and the development community. In the process, this long story has helped to give the Video4Linux development community a bit of a reputation as a dysfunctional family - a perception which those developers are only now beginning to overcome. The sad truth would seem to be that, while working with the community is something that a couple thousand developers do with little trouble every year, there will always be a few who have difficulties.

A quick review of some of the history is in order here. Markus was one of the authors of the original em28xx driver, first merged for the 2.6.15 kernel. His efforts to enhance that driver quickly ran into trouble, though, when he tried to make substantial changes to the low-level tuner interface - changes which affected a number of other drivers. These changes were not popular in the Video4Linux community, and there were fears that they could break unrelated drivers. So this code was not merged.

In response to this rejection, Markus claimed ownership of the em28xx driver and asked that it be removed from the mainline kernel. He then continued development of the code, hosting it on his own server. There was even a period where the code was relicensed to the MPL, apparently as part of an attempt to prevent it from being taken into the mainline. Eventually, Markus came back with a new approach which moved much of the tuner code into user space. That solution, too, failed to pass review; nobody else could really see much advantage in moving that much driver code out of the kernel. The fact that Markus clearly intended to have some of that code appear in the form of binary-only blobs did not help his case. So the user-space approach, like its predecessor, was not merged.

While Markus was working on his own version of the code, others were putting patches into the mainline em28xx driver. At times, Markus tried to block those changes. The tone of the discussion is, perhaps, best seen from this note sent to Video4Linux maintainer Mauro Carvalho Chehab:

Best would be to replace you as a maintainer since you don't have any respect of others work either. Companies should be aware that if they try to submit any code to you they will loose the authority over _their_ work.

Of course, losing "authority" over code is inherent in releasing that code under a license like the GPL. This attempt to exercise control over freely-licensed code was slapped down by Andrew Morton and others, but it left unpleasant memories behind.

Now Markus is back with a driver that, to all appearances, duplicates the functionality of a driver which is already in the mainline kernel. It is not hard to see this submission as an attempt to retake control of that driver and, perhaps, restart the discussions from past years. So it is not entirely surprising that this driver has not been received with a great deal of enthusiasm. In short, Markus has been told to go away until he is prepared to submit his work in the form of a series of small patches to the in-tree em28xx driver.

The advantages of improving the current driver, rather than duplicating some of its functionality in a new code base, are clear. It would avoid the confusion which can come from having two drivers for the same hardware in the tree, and it would minimize the risk of losing important fixes which have been applied to the in-tree code. This is, also, the way that kernel developers are normally expected to do their work. On the other hand, video developer Hans Verkuil reviewed the new driver and concluded:

In my opinion it's pretty much hopeless trying to convert the current em28xx driver into what you have. It's a huge amount of work that no one wants to do and (in this case) with very little benefit.

This review notwithstanding, Mauro has indicated that he is not interested in accepting this patch. But rejecting Markus's new driver out of hand might just be a mistake. There seems to be little doubt that it has developed well beyond the in-tree driver; it supports a wider range of devices. Failure to merge it risks losing the work that has been done, and, perhaps, losing the future work of a developer who, for all his faults, is clearly trying to provide a better experience for Video4Linux users.

Having multiple drivers for the same hardware in the kernel is not an ideal situation, but it is also not without precedent. The IDE and parallel ATA subsystems provide redundant support for a wide range of hardware. The e1000 and e1000e drivers had overlapping coverage for some time. In such cases, the long-term goal is usually to work toward the removal of one of the drivers.

So one could make the case for merging the new driver and, eventually, removing the older one. In the process, the new driver could receive some much-needed attention from other developers. It has coding style and copyright attribution problems; a quick review has also left your editor wondering about locking issues. But such problems are common to drivers which have spent a lot of time out of tree; they are simply something to fix. Meanwhile, this driver contains the result of years of work and access to the relevant data sheets; freezing it out may not be in the best interests of kernel developers or users.

Index entries for this article
Kernel	Development model/Driver merging

The sad story of the em28xx driver

Posted Nov 11, 2008 23:24 UTC (Tue) by Hanno (guest, #41730) [Link] (6 responses)

So nothing has changed since over a year?

http://www.linuxtv.org/pipermail/linux-dvb/2007-July/0188...

Sad. As a user, it's unbelievable to me that these guys have so much time on their hand for childish flamewars.

The sad story of the em28xx driver

Posted Nov 12, 2008 0:45 UTC (Wed) by JoeF (guest, #4486) [Link]

Well, politics and clashes of personalities happen in every organization, even in a loosely organized community like an open source development group.
It is up to the leaders/steering group/maintainers to handle these kinds of situations. Obviously, this hasn't worked out very well in this instance.
Maybe Jon's article can be the crystallization point for everybody to take a step back and go on a path of compromise instead of confrontation (hope springs eternal, I know...)

The sad story of the em28xx driver

Posted Nov 12, 2008 13:07 UTC (Wed) by mrec (guest, #41847) [Link] (3 responses)

Oh I'm able to reply here great :-)

First of all let's start at the beginning and not in the middle of everything.

I agree with Jonathan that many things finally went that way BUT there's some kind of reason for it.

First of all some clarification:
The ownership of the em28xx driver, I have a couple of devices here and I know what's required to get those things work, there are also some requirements about backward compatibility. I don't want to own the em28xx driver, I would like to maintain it since I did that for 3 years already, accepting useful patches giving reviews etc. It could easily be that this was mistaken though.

The userspace driver attempt .. there we go it start's right in the middle of the whole story not taking into account what happened before.

So let's start and add it here:

The whole driver started at the end of 2005, I started to reverse engineer the protocol and worked on it with Luca Risolia(hope I got the name right). We got video, tuning and audio work. At that time nothing was a problem because there were only analog devices available.
At the beginning of 2006 hybrid devices came up and I continued to work on the driver. At this time I faced the problem that the analog and digital TV framework was separated, and there was certainly also some flamewar going on between v4l and the dvb people.

There have been a couple of approaches to solve the technical problem to share 1 component between analog and digital TV (the tuner) back then. My first approach was limited to a specific parameter setting which only allowed to receive some TV channels, I realized that and started over again digging further into the V4L and DVB framework.
I came up with the idea to refactor the tuner stuff in order to be able to use the same tuner code between both v4l and dvb, to make the story short this was declined. During the time while working on it the driver already started supported more and more devices, and that small gap at the beginning became as a base for the entire code, everytime I tried to rework some bits in it it broke alot devices.

So at that point there was already a large codebase depending on it, and a merge request has been sent.
http://mcentral.de/v4l-dvb/
(note this has nothing to do with userspace drivers).

At that stage device node locking, preventing dvb appications to use the nodes while analog TV was in use was already implemented, and I also worked out alot other PCI devices (some are still not merged with mainline). Also quite a few bugs have been fixed in the kernelframework (eg tuner-core) it cost alot time all together and the work didn't get valued at all. So at the end the code basically just got left behind and also during the development noone of those people who were against it was interested to join the core development - I asked several time on IRC, the ML posts didn't get further anyway either.

Just one, I have to write that Michael Krufky was also involved into why it turned bad. When I once asked him about joining that stuff he just told me well but you have it already done.. I wanted to do it.
The whole issues at that time gave him the opportunity to work on his own stuff even more to let the existing em28xx code left behind.

Finally his stuff got merged, bringing my code totally out of sync, and patches which got into mainstream were mostly derived from my code too.
If someone looks at the audio driver, and the patches for the audio driver it's the code I've written, bugfixes to read() for the videoframes the same story but never taking into account all the devices which were supported by the em28xx driver which I worked on constantly.

So now the userspace stuff comes in thinking about how to open up that project again so that work can be used with the existing kernel.
I got around i2c-dev which allows the access of those shared components in userland, the only outstanding gap in order to remain compatible with existing tv apps was to add a small wrapper to resubmit the controls to userspace (sick isn't it?) well partly there are projects out there for BSD which do tuning in userspace. Another point here could be what is a kernelspace driver worth which requires firmware and doesn't come with firmware. At that time there was no single binary driver available from my side anyway, everything has always been opensource (this also has to be considered).

Now continue reading with the article ontop of it.

Markus Rechberger

The sad story of the em28xx driver

Posted Nov 12, 2008 22:30 UTC (Wed) by liljencrantz (guest, #28458) [Link] (2 responses)

I mean no offense, but it would seem to me from this message that the good old language barrier is likely to be a significant factor in the problems described - I found your post extremely hard to understand, and I suspect I'm not alone in this.

The sad story of the em28xx driver

Posted Nov 12, 2008 23:57 UTC (Wed) by mrec (guest, #41847) [Link] (1 responses)

The main problem is that the whole story is much longer, even though what I wrote right there it's just a small piece of what was going on before. I just want to avoid spending hours on it again.

Someone could dig into the whole topic at least for a week (fulltime).
Jonathan's article is overall ok, but it starts right after the bad things happened when the mood of everyone was already below something good.

Markus

The sad story of the em28xx driver

Posted Nov 13, 2008 1:35 UTC (Thu) by JoeF (guest, #4486) [Link]

Well, let the past be the past. There is no point really to rehash all the old stuff, and to blame person a or person b or person c. I am sure the other people involved have their own recollections which may differ from your's (and memory fades, anyway.)
That's all water under the bridge. Some time, some sociologist may want to use that material in a thesis about group dynamics in open source development... (no offense to sociologists)

Let's just agree that you guys disagreed, and move forward.

The sad story of the em28xx driver

Posted Nov 14, 2008 23:30 UTC (Fri) by giraffedata (guest, #1954) [Link]

... it's unbelievable to me that these guys have so much time on their hand for childish flamewars.

It takes very little time to conduct a childish flamewar. In the referenced posting, you suggest that time could be used better to fix bugs. I'll bet they couldn't fix 5% of a device driver bug in the time it takes to write a series of childish flames.

So I think your real objection isn't to the developers' allocation of their time, but to the way they choose to conduct their lives.

The sad story of the em28xx driver

Posted Nov 12, 2008 6:34 UTC (Wed) by cjayachandran (subscriber, #45677) [Link] (4 responses)

Is there an issue in having two drivers for the same device?

I distinctly remember many devices having legacy as well as newer drivers supporting more devices, it does not hurt anybody to get his new driver in as an alternate driver and give the choice to the user.

It's same as with Reiser4

Posted Nov 12, 2008 9:11 UTC (Wed) by khim (subscriber, #9252) [Link] (3 responses)

If you are abusing developers's and users's trust it becomes suddenly much harder to integrate code in kernel. Take look on the story with Reiserfs and Reiser4: Hans promised to work with the community on Reiserfs and so after long deliberation it was included in kernel. After few years he started to close bugs and feature requests with "use reiser4" (effectively abandoning users) and stopped reiserfs support (when something went wring he pointed out that the same thing worked perfectly with previous version of kernel and basically refused to debug and fix problems thus abandoning developers). This made inclusion of reiser4 more-or-less impossible: since it was pretty clear that reiser4 will be abandoned once it'll be included all problems which usually don't act as merge blockers (because kernel developers know they can be fixed later) were declared as such (because it was not clear if Hans and his command will try to fix them after merge and history showed that he'll probably not bother).

And in most cases newer drivers support more hardware, not less. Kernel developers are quite reluctant to include driver which works on subset of hardware just because driver developer is not interested with community work. If there are technical reasons to have two drivers it's Ok, but if the reasons are political... it's different story. Multiple drivers are easily accepted as temporary solution, but then someone must merge support for other hardware - and then we have "reiserfs situation" where developer works for it's own feature and against all others...

It's same as with Reiser4

Posted Nov 14, 2008 23:25 UTC (Fri) by giraffedata (guest, #1954) [Link] (2 responses)

If I follow your analogy, you're saying that it would be bad to include Markus's new driver alongside the existing one because Markus would eventually abandon it in favor of working on yet another new driver.

Is it actually a requirement for driver inclusion that future support be lined up? That seems to be the opposite of the open source process. Rather, a developer puts code into the mainline so that the whole rest of the world can maintain it. In particular, users of the code — people with a material interest in it — maintain it. If the code doesn't get maintained and becomes unusable, we just drop it back out.

The article mentions the confusion factor of having two similar drivers. I can think of another drawback: division of development and test effort. All of these are weak.

Mauro doesn't explain his reason for not wanting both. He just says:

Both upstream and the 4 duplicated drivers have similar functionality. Also, the upstream driver is actively maintained. So, there's no sense on accepting those duplicated drivers.

That's non sequitur to me.

It's same as with Reiser4

Posted Nov 14, 2008 23:32 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

I read it not as an analogy, but as another example of a developer who had trouble working with people and therefor had trouble getting code merged.

there have been many examples of alternate drivers for hardwar being in the kernel, very few of them have worked out well in the long run, so kernel devs don't want to do more of it.

It's same as with Reiser4

Posted Nov 15, 2008 5:30 UTC (Sat) by mrec (guest, #41847) [Link]

It's not about a developer having problem to work together with people.
It's about that this code has been available for years

http://mcentral.de/hg/~mrec/em28xx-new/shortlog

just check the list of contributors, and the maintainer hard fights against it now. The reason for the big patch has always been that the code had to be rebased because the maintainer basically didn't care about anything there.

Another example

http://linuxtv.org/hg/v4l-dvb/rev/dc22320bb695

I don't see any pull request on the mailinglist, Mauro does it offlist excluding everyone else from it. Ok I acknowledge that is what's wanted and how it should be done.

Markus

The sad story of the em28xx driver

Posted Nov 12, 2008 9:26 UTC (Wed) by hverkuil (subscriber, #41056) [Link]

I'd like to clarify my position on the empia driver since the quote from
my review in this article is a bit misleading.

I am *not* in favor of adding the empia driver as-is to the kernel as the
out-of-context quote suggests. As I mentioned in my review the empia
driver *first* needs to be converted to use the current tuner drivers and
infrastructure. Only then would I be in favor of merging the empia
driver. Until now I have seen no attempts to make these changes, alas.

It's all a very sad story, especially compared with the success stories
in v4l in the recent years: ivtv, gpsca, uvcvideo just to name a few were
all added without any problem and in good harmony. v4l-dvb is now the
third-largest driver subsystem (after scsi and net) in the linux kernel
which is all thanks to the hard work of the many dedicated v4l-dvb
developers.

Hans Verkuil

The sad story of the em28xx driver

Posted Nov 13, 2008 11:22 UTC (Thu) by mjcoder (guest, #54432) [Link] (5 responses)

One huge problem is that the V4L community is not willing to advance (or rewrite) their API to ease driver development (just take a look at the DVB-S(2) mess).

They fight for (at least) three years now and they still don't have a useful infrastructure to support DVB-S2 devices.

The sad story of the em28xx driver

Posted Nov 13, 2008 18:35 UTC (Thu) by mrec (guest, #41847) [Link] (4 responses)

that's over since Mauro took over some people didn't agree either, one maintainer stopped to support his project too because of that, the competitive S2 API got merged already.

http://www.mail-archive.com/vdr@linuxtv.org/msg07539.html

although guess what one argument for not taking my code was we should first get multiproto in before taking my first patches. It's all connected, including multiple flamewars. Now one group (the v4l group) has taken over without taking other dvb people who managed the project before as a small group before... although this group wasn't defined either.

It's all about friendship there, A likes B instead of C even if C has better stuff and supporters, so A will wait till B completes his stuff, and C will be pushed away as long as possible without clean reviews or something else, this is just the way how it works out. Basically there are too many good people involved already. The project is moreover looking for lower skilled people who can be taught what to do (do it as A or B says otherwise they'll run against C). It's not a project for innovative people to work on own ideas that for other open operating systems are better in that case.

The sad story of the em28xx driver

Posted Nov 16, 2008 10:17 UTC (Sun) by Ze (guest, #54182) [Link] (3 responses)

When friendship rules over code quality then that's a sad state of affairs but not an uncommon one.

Cliques are a hard problem to beat and generally impossible , we can see so many examples of this screwing projects over.

In this case it's bloody disgusting that code was put on the backburner in favour of waiting for somebody else to complete their alternative solution just because A liked B better than C.

The sad story of the em28xx driver

Posted Nov 17, 2008 20:35 UTC (Mon) by cde (guest, #46554) [Link]

This story reminds me of the Con Kolivas scheduler melodrama. Here we have again one guy against a supposed "clique" of kernel developers. Reality is probably a bit more complex though.

The sad story of the em28xx driver

Posted Nov 26, 2008 18:30 UTC (Wed) by makomk (guest, #51493) [Link] (1 responses)

As I recall, it was more complicated than that (though this was a while
ago). There was some prolonged feuding relating to changes to the core
code. They ended with Markus submitting some code which made non-backwards
compatible changes to that code (something some of the other developers
didn't like); one of them proposed a simple change that would add better
backwards compatibility, but Markus basically said "take it as it is or
leave it".

What happened next was that Marcus developed his own (partially user-space)
replacement for a large chunk of the Video4Linux/DVB framework, including
forking a driver or two that hadn't been written by him, had been in-kernel
for a while, and was used by other cards. Meanwhile, the other developer
added the main required feature (support for hybrid analog-digital tuners)
to the Video4Linux code in roughly the same way as they'd proposed earlier,
but written from scratch, and this was accepted. (It also eventually lead
to some nice code cleanups.)

Well, Marcus' userspace stuff was rejected, due to (a) duplicating existing
code in a way that'd make life difficult for developers and (b) the only
real "benefit" being the ability to have closed-source blobs in drivers.
(Messages to me from Markus threatening to take his code closed-source if
he didn't get his way didn't exactly encourage me to support it either.)
There were code quality issues too, of course, but I'm not sure anyone got
as far as looking at those.

In the end, some of the developers decided to just improve the version of
the driver currently in-kernel rather than continuing to deal with Markus.
Only after this effort was well underway did Markus begin developing yet
another in-kernel driver. (This one also seemed to fail to meet kernel code
quality standards last time I checked, mainly due to incorporating large
chunks of third-party code.)

The sad story of the em28xx driver

Posted Nov 27, 2008 15:39 UTC (Thu) by mrec (guest, #41847) [Link]

(b) the only real "benefit" being the ability to have closed-source blobs in drivers.

This was one possibility yes, but I also pointed out that the already available i2c-dev infrastructure could directly be used with the driver for setting up chips.

http://mcentral.de/mrec/analogscan.png

This player parses the sysfs tree and opens the corresponding i2c device directly without any additional module. It can do some setup steps which the API currently cannot handle.

So in order to avoid that someone has to disallow the usbfs and i2c-dev interface. Back then Greg Kroah pointed out that binary drivers could use that interface, it was accepted because he basically didn't clash with another developer. So binary drivers here or there is basically the same.
This was basically my try to make development liberal and not driven by one single person.

The userspace drivers used a couple of self written drivers. The drivers which everything was about are still not available in the kernel, it basically was about Micronas chips back then.
The approach aimed to have some compatibility level to the available BSD tuning modules.

Building a virtual front between developers who actually supported linux for a long time isn't healthy. But well that way I don't see any chance in officially supporting and telling people to use video on linux unless they want to depend on the video4linux Maintainer. This will be my last comment about everything unless someone starts to think clear about the whole situation.

I never dropped off anyone contributing to that project, rather helped them to get further into it and the code authors as well as the support mailinglist are a proof of that.

The sad story of the em28xx driver

Posted Nov 20, 2008 18:19 UTC (Thu) by donbarry (guest, #10485) [Link] (1 responses)

Reminds one of that person who went by the nom de plume of "Nemosoft".

No possibility they are the same person, is it?

Same disagreements, same tone, same part of the kernel..

The sad story of the em28xx driver

Posted Nov 24, 2008 10:19 UTC (Mon) by zor (guest, #55304) [Link]

Or could it be, different people, same unresolved management problem ?

Can't a higher-up maintainer intervene?

Posted Nov 24, 2008 20:02 UTC (Mon) by nicodoggie (guest, #55309) [Link]

I'm not really clear about the kernel development process, but when it comes to a popularity vs a quality-of-code vote, can't a higher-up maintainer just evaluate, make decision and put his foot down?

It's might be quite a problem for Linux as a platform when developers with such access to the hardware specs as Markus can't even properly contribute to the mainline kernel driver.