Obstacles to contribution in embedded Linux
The problem, Tim said, is not small. A typical system-on-chip (SoC) requires 1-2 million lines of out-of-tree code to function. Keeping that code separate obviously hurts the kernel, but it is also painful for the companies involved. There is a real cost to carrying that much out-of-tree code. Sony (where Tim works), for example, was managing 1,800 patches for each phone release — and that was just kernel code.
Fixing this problem requires engagement between embedded developers and the development community. "Engagement" means more just using the code and complying with its licensing; it means explaining requirements to the community, interacting in the community's forums, and merging changes upstream. A lot of companies still find that idea scary — they don't want to be producing code that will be used by their competitors. That is how our community works, though.
Obstacles
The idea behind the survey (conducted in September 2014) was to identify the obstacles to engagement with the community. So the group set out to locate developers who are not contributing code upstream and figure out why that isn't happening. The top reasons turned out to be:
Reason % agreed Developing against an older kernel 54% Work depends on out-of-tree code 50% It's too hard 45% Unable to test 41% Employer does not allow time 40% Patch not good enough 35% Afraid of rejection 33%
One thing that is not lacking is the desire to contribute: 92% of developers said that they think upstreaming code is important. But 21% of them said that management explicitly disapproves of such work, and 40% said that their management did not allow the necessary time. Issues that didn't matter included the need to use English (only 9% agreed) or developers feeling that upstreaming isn't their responsibility (6%). A bit more significant was the 26% of developers who said that their company's internal processes made contribution too hard.
At the top of the list of obstacles was "version gap" — developing code against an older kernel. Companies developing kernel code tend to start with whatever kernel was provided by the SoC vendor rather than the mainline; this is especially true for Android products. A typical Android kernel starts at Google, passes to the SoC vendor, and finally ends up at the equipment manufacturer; by that time, it is far behind the mainline.
As an example, Tim mentioned the Sony mobile kernel, which started as the
3.4 release when Google settled on it. By now, this kernel is 16
releases and three years behind the mainline. Some 26,000 commits, adding
up to 1.8 million lines, have been added to Sony's kernel. The
mainline kernel, of
course, is in a state of constant change. Distance from the mainline makes
![[Tim Bird]](https://static.lwn.net/images/conf/2015/lcj/TimBird-sm.jpg "Tim Bird")
contribution harder, since the kernel community needs patches against
current kernels. As a result, it is unlikely that those 26,000 changes to a
16-release-old kernel can be easily upstreamed.
Another problem is the perceived difficulty of contributing to the kernel; to many, the process seems cumbersome and fraught with pitfalls. There are some documents on how to contribute, but those documents do not fully cover the community's social issues, timing constraints, procedures, and more. That said, Tim noted that some things are getting better; the availability of tools to find trivial issues with patches is helpful.
Kernel subsystem maintainers tend to be strict and terse with contributors, mostly as a result of overload; they simply don't have the time to fix up problematic patches. If a contributor gets a reputation for submitting bad patches and wasting maintainer time, their life will get worse. Silly mistakes can cause a patch to be quickly rejected or ignored. The problem here is that embedded developers are not full-time contributors, so they have a hard time maintaining a proficiency in dealing with the process. It is also hard for them to respond to requests for changes after their project has moved on.
Adding to the problem is the fact that much embedded code simply is not at the required level of quality for upstreaming. This code is often low-quality and highly specialized; it features the sort of workarounds and quick hacks that tend to get a poor reception on the kernel mailing lists.
Dependencies on other out-of-tree code also make things worse. Tim raised the example of the out-of-tree Synaptics touchscreen driver; Sony had developed a number of fixes and enhancements for this driver, but Synaptics had no interest in taking them. So where should these patches go? It is, he noted, not at all fun to be in the business of mainlining a supplier's code.
Developers complained that management does not give them the time they need to work with upstream communities. Product development teams work on tight schedules with the object of producing a "good enough to ship" solution. This is true throughout the industry; it is not unique to open-source software. These developers have no time to respond to change requests from maintainers. The kernel community, meanwhile, is not interested in any particular company's product timelines.
Things get worse, of course, when companies wait until late in the development process to release their code — something everybody does, according to Tim. When it comes time to mainline the code, the developers discover that major changes will be needed, which is discouraging at best. It would be far better to get input from the community or an internal expert early in the process.
Overcoming the obstacles
If a company wants to overcome version-gap problems, Tim said, the best place to start is to get current mainline kernels running on the hardware it is working with. One development team can then work on the mainline, while product engineers work with whatever ancient kernel is intended for the shipping product. The two-team approach can also help with the product-treadmill problem; if a small team is dedicated to mainlining the company's code, it can operate independently of the deadlines that drive the product engineers.
Companies should employ internal mentors and train developers to work with the wider community. Tim also stressed that it is important to use the community's processes internally. When working with kernel code, keep patches in Git; using a tool like Perforce to manage patches will not ease the task of engaging with the community.
With regard to low-quality code, Tim admitted that he had no silver bullet to offer. We all have to do hacks sometimes. The best that can be done is to examine the code to determine whether it should be maintained going forward; it should be kept in Git and reviewed (and possibly improved) at forward-porting time.
To avoid the highly specialized code problem entirely, don't use specialized hardware if at all possible. Manufacturers should, Tim said, require mainline drivers from their suppliers. The cost of software development should be figured into the bill of materials when the hardware is selected. Figuring in the cost of out-of-tree drivers is "an important next step" for the industry.
Unfortunately, companies following Tim's advice are likely to run into a new issue, which he called the "proxy problem." Having a special team dedicated to mainlining code can ease interactions with the community, but it also creates a situation where the community-facing developers are not the subject-matter experts. When they try to upstream code that they did not write, they cannot quickly answer questions or test changes. There is no avoiding the need for the original developers to help the proxies in situations like this.
Why bother?
Tim closed out the session by asking: why should companies bother with upstreaming their code in the first place? He pointed out that Sony has 1,100 developers who have made patches to its kernels in the last five years; many of them are applying the same patches over and over again. Sony would like to decrease the amount of time going into that sort of activity; mainlining its changes is the obvious way to do that.
Getting code upstream has a significant financial benefit for companies: it reduces the maintenance cost of that code and allows others to share the work. Even more importantly, having code upstream can reduce the time to market for future projects. Going through the community process improves the quality of the code. It can also fend off the need to migrate over to a competing company's implementation in the future. Finally, upstreaming is a reward for developers; it is something they want to do, and it will turn them into better developers.
These are all completely selfish reasons, Tim said; they are an entirely sufficient justification for working upstream even without getting into the ethical issues.
To further this effort, the "device mainlining project" is working within the CE Linux Forum. This project will continue to analyze the obstacles to contribution and attempt to address them by promoting best practices, collecting training materials, and publishing code analysis tools. There is also work ongoing to justify community engagement to management and quantify the costs of using out-of-tree code. This group will have its next meeting at the Embedded Linux Conference Europe in October.
[Your editor would like to thank the Linux Foundation for funding his
travel to LinuxCon Japan]
| Index entries for this article | |
|---|---|
| Conference | LinuxCon Japan/2015 |
Posted Jun 9, 2015 21:58 UTC (Tue)
by Jonimus (subscriber, #89694)
[Link] (32 responses)
Posted Jun 9, 2015 23:04 UTC (Tue)
by SEJeff (guest, #51588)
[Link]
See:
Posted Jun 9, 2015 23:58 UTC (Tue)
by jg (guest, #17537)
[Link] (29 responses)
These blobs mean that we have a security nightmare underway in embedded systems, even worse than elsewhere.
This is currently a self-perpetuating culture.
The blob problem is particularly true for communications chips and graphics devices: chip vendors, you know who you are....
Posted Jun 10, 2015 3:43 UTC (Wed)
by dlang (guest, #313)
[Link] (3 responses)
Vendors need to stop thinking about the "life of this device" but rather the "life of the parts and how many different devices they may end up in"
There are very few parts that are only used in one device. Once it is in the companies supply chain and inventory, it becomes an easy choice for other devices over a much longer timeframe.
This is also why Tim said that companies should push their part vendors to provide mainline drivers. The part manufacturer is in a much better position to make the driver match the hardware, and stands to gain the most benefits over time (by being able to sell/support the part more broadly)
If a part really is only going to be used in one product for a couple of years, it really isn't worth mainlining the support for it. Where the value of mainlining the support shows up in when you have to revisit support for the device to make it work with a new kernel (either to update the first device, or to make a new device). You could argue that the effort of doing this once is comparable to mainlining the support (both involve moving to a new kernel version as the base, but when you mainline you have the advantage of the expertise of the kernel devs available and the 'drawback' of them forcing you to have a maintainable design), but if you have to do this twice, the option of mainlining the support when you first touched it (and had the most knowledge about it assembled) is very clear.
But if all you are looking at is this quarter's balance sheet, or this version of this product, you won't see these advantages.
It's technical debt, each time you 'just make it work' rather than mainlining it, you are adding to your future work until the debt builds to the point that the product (or company) fails and gets replaced by something else.
Posted Jun 11, 2015 4:49 UTC (Thu)
by rhekman (guest, #102114)
[Link] (2 responses)
I realize that scenario doesn't necessarily apply to SoC, ASIC or other lower level embedded vendors, but I think there's an element of truth for example with phones or home routers. If the only way for an owner of said device to avoid a future Heartbleed, Shellshock, etc. is to throw out their old device and buy a new one, what incentive is there for a vendor to enable their hardware to run the latest software?
Posted Jun 11, 2015 13:50 UTC (Thu)
by jg (guest, #17537)
[Link]
Posted Jun 11, 2015 23:24 UTC (Thu)
by andrel (guest, #5166)
[Link]
I don't know whether there are enough of us to make a difference...
Posted Jun 10, 2015 8:18 UTC (Wed)
by edeloget (subscriber, #88392)
[Link] (24 responses)
In my own experience, it's even worse: to be able to build your product you'll have
(In France, NDAs are covered by penal laws: not only you'll be fired, but you can
Posted Jun 10, 2015 12:09 UTC (Wed)
by ballombe (subscriber, #9523)
[Link] (23 responses)
Posted Jun 10, 2015 16:51 UTC (Wed)
by jospoortvliet (guest, #33164)
[Link] (22 responses)
Posted Jun 11, 2015 13:45 UTC (Thu)
by krake (guest, #55996)
[Link] (21 responses)
Only employers that do not care about their employees let NDAs pass through to them.
Posted Jun 14, 2015 2:00 UTC (Sun)
by giraffedata (guest, #1954)
[Link] (20 responses)
But that doesn't make the explanation of why people sign NDAs for less than a million euros any less valid. These people gotta eat, and don't have "good" employers by this definition.
But the non sequitur aside, I don't understand your point. Are you saying the employer should negotiate to have the datasheets made public so its engineers don't have to sign NDAs? And if so, are you saying the employer should do so at a cost to its stockholders because of an ethical duty to care about its employees?
Posted Jun 14, 2015 2:26 UTC (Sun)
by raven667 (subscriber, #5198)
[Link] (2 responses)
I think we should stop normalizing the idea that it is OK to behave unethically if you are working in service of your company or organization. At some point "the stockholders might be cross with me" shouldn't be a justification to act unethically.
Posted Jun 14, 2015 2:37 UTC (Sun)
by dlang (guest, #313)
[Link] (1 responses)
Why should the opinions of the stockholders be ignored? Stockholders are people who put their money into the company. If they don't like what's going on, they will pull their money out and the company will suffer. I'm not saying that chasing the best stock price is the right thing to do, but angering the people who are funding you is clearly not a good idea.
I also think that you are wrong in your assumption that the definition of "acting Ethically" is so objectively and universally definable that everyone agrees on exactly what it means.
Posted Jun 14, 2015 2:43 UTC (Sun)
by giraffedata (guest, #1954)
[Link]
Posted Jun 14, 2015 9:22 UTC (Sun)
by krake (guest, #55996)
[Link] (16 responses)
No. I am saying that if a third party requires an NDA, it is the employers responsibility.
The engineers don't need these datasheets for fun or out of personal interests, they need them to for whatever task their employer has set them on.
Posted Jun 14, 2015 16:31 UTC (Sun)
by giraffedata (guest, #1954)
[Link] (15 responses)
I still don't know what you're saying is the employer's responsibility. We must be talking about different things.
An NDA is a non-disclosure agreement. In an NDA between A and B, A gives B some information and in exchange B commits not to disclose it to anyone else. Nvidia is a third party that requires an NDA, and that means Nvidia will not give a datasheet to an engineer unless the engineer commits not to disclose the information in it to anyone else.
Let's say Engineer's employer has tasked Engineer with writing some software which requires Engineer to have an Nvidia datasheet. I believe you're saying the employer is responsible for doing something so that the engineer can have the datasheet but not commit to Nvidia not to disclose its contents to anyone else. What is the employer responsible for doing?
Posted Jun 14, 2015 16:39 UTC (Sun)
by krake (guest, #55996)
[Link] (14 responses)
The company employing the engineer is the legal entity that deals with the supplier.
Posted Jun 14, 2015 16:59 UTC (Sun)
by giraffedata (guest, #1954)
[Link] (13 responses)
Companies with trade secrets don't want anyone to disclose them for any reason, so they do make it a condition of divulging those trade secrets to a person that the person promise not to disclose them. Giving a data sheet to an engineer and getting promises only from his current employer not to disclose the information is the same as publishing the data sheet.
Posted Jun 14, 2015 17:07 UTC (Sun)
by krake (guest, #55996)
[Link] (8 responses)
It is not the engieers fault if the company's lawyers mess up and provide a legal document that the other party then does not accept.
Does such a useless company require their engineers to personally make guaratees to their customers? Indemnify, etc?
Posted Jun 15, 2015 0:08 UTC (Mon)
by giraffedata (guest, #1954)
[Link] (7 responses)
Why does it matter if the engineer is at fault for not having the data sheet? Do you believe an employer should terminate employment only when the employee does something wrong? What if there just isn't any feasible work for him to do?
To make this easier, we could imagine that the employee hasn't actually been hired yet. He's given the choice of signing an NDA and taking the driver-writing job or not signing it and not taking the job.
Posted Jun 20, 2015 12:38 UTC (Sat)
by krake (guest, #55996)
[Link] (6 responses)
Well, you had constructed a weird scenrio where an employee would be fired for something he or she had no part in.
Sure, if the company fails to acquire assets they need to build their product, they might have to shut down and let go of their employees. But there is nothing the individual employee could have one anything about that.
> He's given the choice of signing an NDA and taking the driver-writing job or not signing it and not taking the job.
Why would anyone even consider working for such an exploitive company that makes you sign NDAs on their behalf but keeping you as the party carring the legal risk?
Taking part of a company's risk is what you might have to do as a partner, a co-owner, depending on the type of company it is registered as.
> No, because the employer can just do that itself, unlike keeping a trade secret secret.
So that hypothetical company can be trusted to fulfil its obligations towards its customers but not torwards its suppliers?
Posted Jun 20, 2015 14:16 UTC (Sat)
by anselm (subscriber, #2796)
[Link] (3 responses)
This whole discussion is strange because what will happen in the real world is that the company will sign the NDA if they think doing so helps them to do whatever it is that they're doing, and whoever works for that company will have agreed in their employment contract to keep any confidential stuff that they encounter in the course of their work (like any manuals and example code the company received under NDA) confidential.
Posted Jun 20, 2015 15:45 UTC (Sat)
by krake (guest, #55996)
[Link] (2 responses)
I cannot even imaging why a company would allow a third party to put its employees in legal jeopardy, let alone require its employees to do so.
Posted Jun 20, 2015 17:42 UTC (Sat)
by giraffedata (guest, #1954)
[Link] (1 responses)
No, not exactly. That is not the discussion we were having. We were talking about the case someone mentioned (whether it ever really happens or not) that an employee must sign an NDA in order to see a datasheet (and therefore can't ever contribute his driver to open source), and the following statement of wonder that any employee would sign that.
But even if the original commenter had meant to refer to the situation where the employer signs an NDA (a different NDA, by the way) instead of its employee signing one, it doesn't seem to me it changes any of the issues. The employee still can't tell anyone, still has to pay the consequences if he does, and still can't contribute to open source. (Because the employer essentially propagates its nondisclosure obligations to its employee).
So the possibility that the whole thing is hypothetical is a red herring.
Posted Jun 21, 2015 8:41 UTC (Sun)
by krake (guest, #55996)
[Link]
Exactly, because the employer would do that, as anselm and others have pointed out.
> and therefore can't ever contribute his driver to open source
That will largely depend on the NDAs phrasing and the information provided by the third party.
But the covered information could also be about architecture of the device and description of its components, enabling the driver developer to use the device in a way that fits their goal, producing a driver that does not contain any of the NDA covered information.
In the first case the driver might not be redistributable as part of a product anyway without violating the either the NDA or the GPL.
Posted Jun 20, 2015 17:31 UTC (Sat)
by giraffedata (guest, #1954)
[Link] (1 responses)
Sure, if the company fails to acquire assets they need to build their product, they might have to shut down and let go of their employees. But there is nothing the individual employee could have one anything about that.
So we agree: it's not necessarily wrong for an employer to terminate an employee for something she had no part in and could not have done anything about, so the fact that an employer terminates an employee because a third party won't supply a datasheet does not make it wrong. You seemed to be implying otherwise before.
I still think we must be talking about different things, because there's no significant risk. A person promises to keep something secret. The miniscule risk of accidentally telling the secret and thereby owing someone money is exactly the same as the risk of accidentally disclosing one's employer's own trade secrets. Are you OK with an employee assuming that risk?
And while we're at it, how do you feel about an employer requiring his employee to have a government security clearance to work on a military project when his customer (the government) requires it? How about a trucking company requiring its employees to have a commercial driver's license? I'm still trying to discern what you think is special about an employee agreeing to sign an NDA as a condition of employment.
Posted Jun 21, 2015 8:21 UTC (Sun)
by krake (guest, #55996)
[Link]
Right. It would be wrong if the company put the employee into a situation where beign terminated or not dependend on them agreeing to a contract with a company they don't have anything to do with, e.g. a supplier's NDA.
> I still think we must be talking about different things, because there's no significant risk
So we are talking about an NDA that does not contain any repercussions should the disclosure happen, eg. by accident.
> And while we're at it, how do you feel about an employer requiring his employee to have a government security clearance to work on a military project when his customer (the government) requires it? How about a trucking company requiring its employees to have a commercial driver's license?
These sounds like job requirements that are known to an employment seeking person, not something that miraculously appears during employment.
> I'm still trying to discern what you think is special about an employee agreeing to sign an NDA as a condition of employment.
I was not referring to NDAs as part of a person's employment contract. I was referring to requiring employees to sign random NDAs withn companies they do not work for, for assets they have no personal need of but which their employer requires. If the employer needs it, they (i.e. a company legaly representative) sign the paper work.
They don't expect individual employees to sign e.g. delivery contracts either, do they?
> For some people (my guess is at least 90% of engineers), that risk, and the political objection to being exploited, is outweighed by the salary being offered, so it is well worth considering.
I would think that this depends a lot on country.
I am sure that there are people willing to do it even in countries of the second category if substantial compensation is offered, but finding a volunteer is very different from making something a requirement.
But I don't really see why a company would want to go through that if they could just handle a business to business contract on the inter-company level.
Posted Jun 15, 2015 0:06 UTC (Mon)
by dlang (guest, #313)
[Link] (3 responses)
Posted Jun 15, 2015 0:13 UTC (Mon)
by giraffedata (guest, #1954)
[Link] (1 responses)
They accept them and require them; they just don't accept them in place of an NDA signed by someone else. If a company insists on John promising not to disclose as a condition of giving a data sheet to John, the company will not accept an NDA from Mary instead.
I've heard of a variation where the company with the secret requires the employer to treat the secret as its own and pay for any damages if its employee ever divulges the information, but that just means the engineer is prevented by his own employer from ever divulging the information. That's exactly the same thing in its effect on an engineer's ability to publish the information in a kernel patch.
Posted Jun 15, 2015 1:28 UTC (Mon)
by dlang (guest, #313)
[Link]
If you are somehow seeing this as unethical, I don't know how to start discussing the issue with you.
Now, if you are willing to bypass the "NDAs are unethical" discussion and talk about how datasheets should not be so secret, that's a discussion we can have, and I'll agree with you (after all, if a datasheet is secret, how does an engineer know enough to pick your product?) With the increase of things being designed by small companies, I think the datasheet secrecy is going to eventually collapse as people pick products that they can get info on over ones they have to sign NDAs just to learn about. but It has taken us several decades to get to the point where secret datasheets are considered the norm, and i think it will take a while (but probably not as long) to move back.
Posted Jun 20, 2015 13:20 UTC (Sat)
by krake (guest, #55996)
[Link]
A sensible company would not even want one specific employee to be the NDAs legal entity on their side, it would make it impossible to have other people team up with them, lose access to the information if the employee quit, etc.
Posted Jun 11, 2015 10:58 UTC (Thu)
by broonie (subscriber, #7078)
[Link]
They're also not free - it's not a big deal for the SoCs themselves but for off-SoC devices on larger processes the cost of the digital logic for a meaningful discovery mechanism can be noticeable, and if you try to standardize all the other stuff around the bus it rapidly gets very limiting for things like system wide power management. For a lot of applications it's just not the area where there are practical problems.
There *are* efforts at standard hardware platforms for v8 (SBSA being the main one, part of a bigger effort which also includes firmware interfaces and so on) but they're targeted at the server space where the cost/benefit is different and much more like x86.
Posted Jun 9, 2015 22:03 UTC (Tue)
by jstultz (subscriber, #212)
[Link] (1 responses)
The other aspect is often vendors don't see much value in fighting with upstream for 6mo to a year, to get their code merged, when many devices have such short lifetimes (maybe 2 years). Staging is great for this case too, since users get the immediate benfit, and if after a year or so no one really cares, upstream can easily drop support. But if folks continue to see use from code in staging year after year, the value of getting it cleaned up and merged for good increases.
Posted Jun 9, 2015 23:48 UTC (Tue)
by gdt (subscriber, #6284)
[Link]
I think that one of the solutions is mailing the patches to LKML regardless. A lot of the patches for a phone are of interest even if they aren't in a state to be integrated into the kernel. For example, you could easily imagine that some of the fixes are for drivers -- someone striking the same bug might then go to the trouble of having that fix integrated into the kernel: at least then their task is mere integration and not the horror which can be document-less driver bug exploration.
Posted Jun 9, 2015 23:11 UTC (Tue)
by rodgerd (guest, #58896)
[Link] (1 responses)
So no actual answer then.
Posted Jun 23, 2015 12:36 UTC (Tue)
by nye (subscriber, #51576)
[Link]
Companies can be pretty good at cost/benefit analysis, but it requires that the costs are visible up-front, so a real practical answer is to stop pretending that certain costs don't exist.
It's easiest to assume that marginal cost is all that matters, and that development costs are amortised into something close to zero per device - easy, but often wrong. The development costs can be significant, so for anything but the highest volume devices they should be factored in, at least within an order of magnitude or so, rather than looking at nothing but the per-unit cost for the widget and imagining that that's a good indicator of the *actual* cost to the company.
Posted Jun 10, 2015 3:03 UTC (Wed)
by JIghtuse (guest, #95703)
[Link] (1 responses)
Posted Jun 10, 2015 16:55 UTC (Wed)
by jospoortvliet (guest, #33164)
[Link]
But most important: be patient. Open Source and how we work is incredibly hard to understand for people on the 'business' side, harder than you can imagine. It can easily talk half a year to get reasonable managers who already understand much of open source to see there's a problem - years for those who are further away...
No easy answers here, no.
Posted Jun 10, 2015 3:34 UTC (Wed)
by neilbrown (subscriber, #359)
[Link] (1 responses)
I guess I already knew that, but this article serves to make it rather explicit. And depressing.
Posted Jun 10, 2015 10:19 UTC (Wed)
by NAR (subscriber, #1313)
[Link]
Posted Jun 10, 2015 5:21 UTC (Wed)
by asaz989 (guest, #67798)
[Link] (2 responses)
Having worked in one of these environments until very recently, this is not a workable solution; or at least, it's not a complete one. When we're pegged to old kernel versions by a vendor only supporting that old kernel for their drivers, our decision tree looks like this:
1. Update to a new kernel. Merging to mainline is now easy!
a. Whenever we get patches with bugfixes and new features from the vendor, spend time forward-porting to our shiny new kernel before shipping them.
2. Don't update to a new kernel. Merging vendor changes is now easy!
a. Whenever we want to merge to mainline, spend time forward-porting our changes to a new kernel version before sending the pull request.
I leave it as an exercise to the reader which of these minimizes the cost/product-bugginess ratio.
In my experience, porting from unsupported kernels is cost/time-prohibitive; you need to have code build for new kernels *from the start* to minimize this cost (even if you're stuck porting stuff forward by one year worth of mainline development, that's a lot better than the status quo). This starts with vendors - the number one determinant in how recent of a kernel my former employer shipped was how recent of a kernel our upstream drivers supported. When we wanted to get onto new kernels (or at least, ones that were EOLed within 6 months instead of ones that had been EOL for multiple years) the big problem was in vendor relations - putting as much pressure as possible on vendors to update their drivers to new kernels, or sometimes getting them to support our older hardware in newer versions of their drivers.
Posted Jun 12, 2015 11:00 UTC (Fri)
by HIGHGuY (subscriber, #62277)
[Link]
Posted Jun 16, 2015 7:52 UTC (Tue)
by Felix.Braun (guest, #3032)
[Link]
A. Ask competing component vendors whether they track mainline development
However, for this to happen, companies need to learn that this is a factor. The average consumer doesn't care, which kernel version their gadget runs on. They care about a lower bug rate / higher reliability. It's not entirely obvious how these two dimensions correlate though.
Posted Jun 10, 2015 11:54 UTC (Wed)
by NAR (subscriber, #1313)
[Link] (1 responses)
Posted Jun 14, 2015 16:05 UTC (Sun)
by Hauke (guest, #103131)
[Link]
Posted Jun 10, 2015 13:16 UTC (Wed)
by ncm (guest, #165)
[Link] (7 responses)
The law may be an ass, but naked capitalism as we see it here is worse.
Posted Jun 10, 2015 13:24 UTC (Wed)
by jeff@uclinux.org (guest, #8024)
[Link] (1 responses)
Posted Jun 10, 2015 21:50 UTC (Wed)
by ncm (guest, #165)
[Link]
Posted Jun 11, 2015 9:11 UTC (Thu)
by NAR (subscriber, #1313)
[Link] (4 responses)
The law-related solution would be legislation that would outlaw "no warranty"-type licenses. Of course, as collateral damage, it would also outlaw most (if not all) free and open source licenses...
Posted Jun 11, 2015 18:11 UTC (Thu)
by xtifr (guest, #143)
[Link] (2 responses)
* To the extent that RH is selling a product, rather than a service. The lines can be blurry.
Posted Jun 12, 2015 7:34 UTC (Fri)
by NAR (subscriber, #1313)
[Link] (1 responses)
And I guess it will still operate as a phone even if let's say the VLC app has a "security bug", so no basis to sue the phone maker.
Posted Jun 12, 2015 23:30 UTC (Fri)
by xtifr (guest, #143)
[Link]
Depends. If it was marketed as just a phone, and not, say, a smartphone, then yeah, you should be surprised that the VLC app does anything. Otherwise, well, if the VLC app was first-party, you might have a basis to sue (assuming they don't fix it in a timely manner); if it was third-party, you may have a basis to sue the third-party, but might not be able to recover more than the cost, which, if it was a free app, may not be worth much. :)
All of this depends on all sorts of things, like jurisdiction, and god knows what else. Warranty disclaimers are not necessarily as broad as they might try to sound, but they're also not nothing.
Posted Jun 13, 2015 0:52 UTC (Sat)
by dlang (guest, #313)
[Link]
Posted Jun 10, 2015 21:17 UTC (Wed)
by tpo (subscriber, #25713)
[Link] (8 responses)
It seems like often the developers can not choose the kernel upon which to build on, their solutions are hacks, it is too much work to upstream the patches and they are missing the required time as well.
Having a central, searchable and assorted place to put HW vendor trees and patches to already seems like an improvement to the current situation of hidden, private and non-discoverable linux trees and patches and carries the hope that maybe interested parties (potentially from the commercial embedded world too) could pick up code and run with it and thus bring it closer to the mainline kernel and to required quality standards...
It is clear that having an OS with very high quality standards is a very valuable good, but on the other hand code that "just works for me corp X on hardware Z" is also valuable - after all, maybe it could spare someone else some hard work too.
Posted Jun 11, 2015 1:13 UTC (Thu)
by nevets (subscriber, #11875)
[Link] (1 responses)
Isn't that what "staging" is for?
Posted Jun 11, 2015 3:12 UTC (Thu)
by dlang (guest, #313)
[Link]
Posted Jun 11, 2015 11:35 UTC (Thu)
by faramir (subscriber, #2327)
[Link] (2 responses)
This wouldn't even have to require any active support from the vendors. To the extent that
Posted Jun 14, 2015 16:15 UTC (Sun)
by Hauke (guest, #103131)
[Link] (1 responses)
One thing that would also help the open source community would be if the SoC vendors would release the code into the public when their costumers get it. This would be the same code one year later will be found in these GPL tars.
Posted Jun 14, 2015 16:50 UTC (Sun)
by faramir (subscriber, #2327)
[Link]
True. However, to the extent that a vendor releases multiple iterations of the same (or similar) silicon;
Posted Jun 12, 2015 16:30 UTC (Fri)
by mm7323 (subscriber, #87386)
[Link] (1 responses)
This site of 'crap' could also be interesting to kernel developers to see which subsystems are historically being patched the most and perhaps why. Commonalities may even be identified to give hints at how newer kernels could handle things differently or better e.g. creating new apis or interfaces or just by adding a few extra configuration knobs.
Posted Jun 14, 2015 16:57 UTC (Sun)
by faramir (subscriber, #2327)
[Link]
Posted Jun 19, 2015 4:10 UTC (Fri)
by Mook (subscriber, #71173)
[Link]
What would be the upside for the vendors dumping their code into the crap-repo?
The upside of upstreaming their changes is that the maintenance required to forward-port to newer versions is done by the community (i.e. not the vendor). If they land their changes into a repo nobody looks at, that work isn't going to get done, and the vendors will still have to do the porting themselves. At that point, it's effectively the same as not pushing the code at all.
Posted Jun 14, 2015 16:54 UTC (Sun)
by dirklwn (subscriber, #80581)
[Link]
First, regarding 'above central dumping ground for "crap"': Working since years with ARM devices, I wonder if it wasn't the goal of Linaro to at least improve this situation? And if not, maybe they could help regarding this? On the other hand, there are examples where SoC Vendors, initially working quite well with Linaro, stopped their Linaro engagement and continuing with their custom HW vendor tree, now. So there seems to be some room for improvement, indeed.
Second, regarding the 'two-team approach': It might work for others, but I don't have the feeling that it would work for everybody. You need a lot of qualified development resources to do the products, already. Where should the second team come from? Try to find some qualified Linux kernel developers familar with mainlining code you don't need sooner or later for the product team. At least in Europe and India this sounds quite hard!? Our main developing team (having only one) is just improving the mainlining efforts. I see similar configurations like > 1800 patches on top of an older mainline kernel (as mentioned in the article). I'd guess that improving our mainlining activities would result in < 100 patches mainlined per year. But yes, better than nothing. On the other hand, we have to be aware that a lot of these > 1800 patches are either 3rd party proprietary driver stuff which will never be mainlined. Or back ports of patches already in mainline, needed in the older product kernel.
But back to automotive topics. In cars, you have some special requirements. Especially, your hardware and software needs to be able to survive any random power loss. Usually SD card like storages or hard disks are used for the root and data file systems. Embeddable MMC/SD cards called eMMC, mountable directly on the PCB, are quite popular. As normal SD cards are already known to fail, the environment in cars is even harder: You need your root file system stable and undamaged over a temperature range of -20°C to +80°C, with random power loss while write, over the car’s life time. Several attempts are made to improve the support for this environment: You have to detect power failure conditions trying to keep the system stable for several milliseconds until the power for the storage device drops. With eMMCs, this time can be used to send high priority interrupts to interrupt the current work of the device and shut down it properly. Similar to this is the support for reliable writes.
Talking about systems with eMMCs and removable SD cards, SD cards are typically used for user data like audio or system navigation data. Then you have to ensure your eMMC with the root file system is visible under the same device name always. This has to be independent of the SD card being inserted at boot time, or not. There are several attempts and requests from different use cases to get a solution for persistent MMC block device names merged mainline. These attempts are still ongoing, since several years, now.
Furthermore, the overall system stability requirements for IVI Linux are quite high. Have you ever used the ‘reboot’ button in your car infotainment system? Is there even such a button? From your Android device you are used to it, no? How long do you use your Android device? More than 10 years like you use your car? There is the expectation that your car infotainment system runs stable while you drive, no Oops or reboot is allowed. And this for the lifetime of the car. Even if it’s powered by nearly the same SoC like your Android mobile, using nearly the same operating system, the Linux kernel. To achieve this, the usual long time to market is used for heavily testing. Automated test benches in conjunction with under voltage scenarios record every Oops. Resulting in e.g. hard to discover race conditions never seen before.
There are several other issues, too. As always, boot time. If you want to boot fast (and in the automotive world you have to), you have to optimize, optimize and optimize again the system. Most probably you optimize it for your SoC. But these optimizations (hacks?) often conflict with the generic and compatible mainline kernel approach once you want to mainline your optimizations.
Then there are several other topics like board configuration, device trees, multi platform kernels, LTS kernels and their selection, legal aspects of mainlining etc. But most probably this should go to an own article, instead ;)
Posted Jun 29, 2015 20:44 UTC (Mon)
by tbird20d (subscriber, #1901)
[Link] (5 responses)
Three of the above ideas are around ways to make the out-of-tree code more accessible: 1) use staging, 2) mail to LKML, 3) make a dumping ground. Some of the ideas are related to exposing the cost. I hope to follow up on both of these broad strategies with concrete proposals or work shortly. I'll be having a BOF at LCE/ELC Europe in October. Please come to the BOF if you are at the event, for more discussion. Thanks.
Posted Jul 15, 2015 7:41 UTC (Wed)
by pabs (subscriber, #43278)
[Link] (4 responses)
Posted Jul 15, 2015 9:06 UTC (Wed)
by BlueLightning (guest, #38978)
[Link] (3 responses)
IIRC Greg KH used to have a wiki page listing something like that for the Linux Driver Project, sadly the page seems to be gone (perhaps it was only meant to be around for the initial investigation) - either that or my google-fu needs work. A centralised tracker for such code that is kept up-to-date would indeed be very useful.
Posted Jul 16, 2015 8:33 UTC (Thu)
by pabs (subscriber, #43278)
[Link] (2 responses)
Posted Jul 17, 2015 12:43 UTC (Fri)
by BlueLightning (guest, #38978)
[Link] (1 responses)
I'm not a kernel person (just an interested observer), but FWIW I think it would probably be worth asking Greg KH since he has done this kind of thing before.
Posted Jul 20, 2015 6:52 UTC (Mon)
by pabs (subscriber, #43278)
[Link]
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
http://www.devicetree.org/Main_Page
http://elinux.org/Device_Tree
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
> that only provide binary blob drivers/kernel modules. This "freezes" such systems
> on often quite old versions of the Linux kernel and keeps them there .. (snip)
to sign an NDA - this way, you might be able to gain access to datasheets that are
not public. But then, you don't have the right to publish these trade secrets in any
form - including source code - meaning that you can't even write a replacement
driver that you'd like to mainline. In some cases, you're prevented to contribute
fixes and enhancements to drivers that have been mainlined years before.
also have to pay damages to both your employer and the founder, and that could
theoretically amount to millions of euros ; this is quite an effective deterrent).
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Any good employer will handle NDAs before the reach the individual engineers.
Obstacles to contribution in embedded Linux - NDAs
Only employers that do not care about their employees let NDAs pass through to them. Any good employer will handle NDAs before the reach the individual engineers.
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
And this is all really getting ahead of the discussion, because no one in this thread has yet said it's wrong for a company to help its employees at the expense of its stockholders. I know I didn't - all I did was ask for a clarification of a post I did not understand.
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
I am saying that if a third party requires an NDA, it is the employers responsibility.
Obstacles to contribution in embedded Linux - NDAs
Ah. That would not result in Engineer getting the data sheet. Company A would throw that NDA signed by Company B away and refuse to supply the data sheet. Engineer, being thus useless to Company B, would lose his job. So we're back to engineers signing NDAs because they gotta eat.
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
Why would the engineer lose their job when they had no part in the dealings?
Obstacles to contribution in embedded Linux - NDAs
It is not the engineer's fault if the company's lawyers mess up and provide a legal document that the other party then does not accept.
I don't believe there is any document the employer's legal department could provide that would get Nvidia to provide a data sheet to a person who does not promise to keep it secret. No one messed up in this scenario. A lawyer can't get blood from a stone.
Does such a useless company require their engineers to personally make guarantees to their customers? Indemnify, etc?
No, because the employer can just do that itself, unlike keeping a trade secret secret.
Obstacles to contribution in embedded Linux - NDAs
But as an employee?
Would such an unreliably firm be something you would even want to work for?
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
what will happen in the real world is that the company will sign the NDA
...
Exactly!
Obstacles to contribution in embedded Linux - NDAs
If the driver requires data tables or firmware, etc. covered by the NDA then obviously not.
Obstacles to contribution in embedded Linux - NDAs
Why does it matter if the engineer is at fault for not having the data sheet?
Well, you had constructed a weird scenrio where an employee would be fired for something he or she had no part in.
Why would anyone even consider working for such an exploitive company that makes you sign NDAs on their behalf but keeping you as the party carrying the legal risk?
Why would anyone even consider working for such an exploitive company that makes you sign NDAs on their behalf but keeping you as the party carrying the legal risk?
For some people (my guess is at least 90% of engineers), that risk, and the political objection to being exploited, is outweighed by the salary being offered, so it is well worth considering.
Obstacles to contribution in embedded Linux - NDAs
Why have such an NDA in the first place?
If said trucking company hired people without licenses it can of course (nicely) ask its employees to get licensed on the company's expenses, but require it?
In countries with weak or no employee rights protections people might be more likely to concede, while in other countries they would probably just shake their heads in astonishment.
After all that is why companies are legal entities, with certain individuals being designated legal representatives.
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
since when do companies not accept NDAs signed by other companies? That's a pretty routine thing to have take place
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux - NDAs
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to purchasing devices containing embedded Linux
I guess you know the relevant Alan Shepard quote...
Obstacles to purchasing devices containing embedded Linux
Obstacles to contribution in embedded Linux
b. Don't ship new patches from the vendor, leaving bugs in place.
b. Don't merge to mainline.
An additional constraint is that even if you want to upstream a driver for a device attached to your SoC, you probably cannot test it since the SoC doesn't run mainline and that's the only board you have with that device on it.
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
B. Factor in reply into the cost / benefit analysis for choosing which components to include into your product
There's an interesting thought in the article: many of them are applying the same patches over and over again. Sony would like to decrease the amount of time going into that sort of activity; mainlining its changes is the obvious way to do that. I guess if the kernel parts these patches modify don't change (much), then applying these patches are not much work. On the other hand if the kernel changes much, then applying the patches over and over could be substantial work. Mainlining would mean that the kernel community would do this substantial work - maybe not at once when these patches are applied, but each time the kernel changes. Why would the kernel community do this work for Sony? Could the kernel community actually do this work without access to the hardware?
Why do the work?
Why do the work?
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Can you actually buy a device which doesn't say in its license that "There is NO WARRANTY, to the extent permitted by law."? I mean even the RHEL license states that "in no event will red hat, inc. be liable to you for any damages, including lost profits, lost savings, or other incidental or consequential damages, arising out of the use or inability to use the software programs, even if red hat, inc. or a dealer authorized by red hat, inc. had been advised of the possibility of such damages". Of course, the license shouts it in bold capitals, in order to make it hard to read it. How do you plan to sue?
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Can you actually buy a device which doesn't say in its license that "There is NO WARRANTY, to the extent permitted by law."?
The key there is the "to the extent permitted by law" clause. In a lot of jurisdictions, there are severe limitations on how much you can refuse to warrant something, especially if money or other considerations are involved. Of course, this is often limited to refunds or the like, so free software that isn't explicitly being sold is still relatively safe (although RH* and phone vendors and such don't get this out). And how you market something matters too--for example, I'm currently trying to offload something I'm describing as a "broken centrifuge", so the fact that it doesn't work shouldn't be an issue, but something packaged and found on store shelves marked as "phone" pretty much has to operate as a phone, no matter how many disclaimers the license includes. Otherwise, you're pretty clearly into fraud territory.
but something packaged and found on store shelves marked as "phone" pretty much has to operate as a phone
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
Obstacles to contribution in embedded Linux
A place to collect abandonware kernel trees?
?
*t
A place to collect abandonware kernel trees?
A place to collect abandonware kernel trees?
A place to collect abandonware kernel trees?
they abide by the GPL and make source available (usually as the equivalent of a tarball), it should be possible to diff that kernel against the underlying mainline kernel. It could then be made a single giant git pullable changeset. Reverse engineering the underlying functional changes from the giant patch would take more work, but is probably doable in a somewhat automated way by subsystem. I could see people who work on third party Android ROMs or router firmware benefiting from something like this. I would hope that eventually common changes/drivers by SOC or other hardware would become apparent and could be broken out into their own changesets/trees. Vendors might even contribute to this since the bar for
being included would simply be that it was shipped in a product. Admittedly this is work that the vendors should really be doing themselves, but it might be worth the efforts of outsiders to
get things started.
A place to collect abandonware kernel trees?
A place to collect abandonware kernel trees?
having software for a year old interation of the actual silicon device (not whatever marking name is
slapped on it) could still be helpful.
A place to collect abandonware kernel trees?
A place to collect abandonware kernel trees?
Somebody's patches to TCP, MM, or .... Just a web page with links to all the places to go look for this
kind of thing would be interesting. Maybe some kind of wiki by subsystem where people could do
drive by edits when they run across (or release) new code. While not as useful as public git trees,
it seem to be a step in the right direction.
A place to collect abandonware kernel trees?
Embedded automotive Linux
Thanks everyone for the discussion on this article. There are many good ideas and suggestions in your responses.
Here are some that I think are interesting, and worth considering for action (sorry not to acknowledge the contributor of each idea):
Feedback is really appreciated
On this last point, I started a wiki page to save the links to different mobile phone kernel source download sites, that I found during my analysis. It's at http://elinux.org/Phones_Processors_and_Download_Sites
Feedback is really appreciated
Feedback is really appreciated
Feedback is really appreciated
Feedback is really appreciated
> pages for general categories or for subsystems? Who do we contact at kernel.org about this?
Feedback is really appreciated