Linux in mixed-criticality systems [LWN.net]

Linux in mixed-criticality systems

Posted Dec 14, 2018 9:27 UTC (Fri) by nim-nim (subscriber, #34454) [Link] (8 responses)

A car is an expensive thing to produce, but the automobile industry is very competitive, so margins are minuscule (IIRC, profit is a few hundred of €/$ for a car that sells for tens of thousands). It works because volumes are high, and the car industry real earnings come from repair and loans to buyers.

So basically, consolidating the car information systems, can easily result in savings at least equal to the profit the manufacturer currently makes on the car as a whole.

Linux in mixed-criticality systems

Posted Dec 14, 2018 15:18 UTC (Fri) by marcH (subscriber, #57642) [Link] (7 responses)

Just like any other type of product: it depends. The margins are low for low-end cars (the ones manufacturers want to stop selling, see recent GM layoffs for instance) but they're not low for trucks and SUVs and not the ones with... a gazillion electronic gadgets. We all heard countless stories of buyers negotiating harder and for longer and paying thousands of dollars less than the next guy for the same car.

Linux in mixed-criticality systems

Posted Dec 15, 2018 10:30 UTC (Sat) by drag (guest, #31333) [Link] (6 responses)

Manufacturers are dependent on car dealerships to push their products onto the buying public. Dealerships depend heavily on post-sale support/maintenance/parts/repair and financing for a huge portion of their profits.

So there is a conflict of interest there. If manufacturers produce 'open' technology that is easy and simple for anybody to repair, provide parts for, and support then they risk alienating the sales goons. So they design vehicles specifically to be anti-serviceable and keep diagnostic tools and software extremely proprietary.

As the technology improves cars should be becoming simpler and easier to repair. The technology and engineering that goes into modern vehicles is incredible, very expensive, and hugely difficult... but the actual execution is still really basic. Hall effect sensors, fuel injectors, temperature sensors, O2 sensors, fuel meters, fuel pumps are all late 1970's early 1980's technology just highly evolved. A effective modern fuel injection system is much simpler and easier to support and repair then a old carbureted system. Parts are cheaper to manufacture on a large scale, too. Which is some of the major reasons why cars moved away from the old tech in the first place.

So, yes, piling on tons of gadgets and features to otherwise basic form of transportation is a big way they can help plump up the profits of their dealer networks.

As is tying computer systems together into big massive proprietary lumps and protecting access and tools required to diagnose with DRM in order to trigger DMCA legislation and making it illegal, or at least extremely difficult, for people to produce third party diagnostic and support tools.

https://www.wired.com/2015/04/dmca-ownership-john-deere/

> General Motors told the Copyright Office that proponents of copyright reform mistakenly “conflate ownership of a vehicle with ownership of the underlying computer software in a vehicle.”

Because of that sort of crap I am extremely weary of any attempt to combine entertainment and vehicle management systems into one big lump. Besides the fact that it's just poor engineering. I just don't trust these people.

Next thing you know they will have to have all sorts of backdoors and encrypted control over firmware that prevents user serviceability because of 'security'. Can't have somebody hack your bluetooth speaker and then updating the firmware for your anti-lock brake system... right? Right?

For tying these systems together in a secure manner it should be very possible to setup one-way communication so that information and logging and other things can flow from vehicle management into the display and networking computers. The protocols should be documented and 'best effort' standards should be established so that third party tools and software can be used to reduce the cost of ownership and increase the value of vehicles for the customers.

I have hopes for companies like Tesla will move the industry away from car dealerships and such things. And I think that things will improved provided the Government resists the temptation of strengthening IP protection legislation for the sake of these major corporations. (which are all already extremely damaging at all levels of society). But until that happens I really am going to stick with buying the most basic cars possible.

Linux in mixed-criticality systems

Posted Dec 18, 2018 13:28 UTC (Tue) by nilsmeyer (guest, #122604) [Link]

Great comment. I think adding a lot of Software in embedded devices also drastically increases the loss in value because of course you can't replace them once they stop receiving updates - although this may be discovered as an additional revenue stream instead of trying to get people to buy new cars.

There really is no good reason for an in-vehicle entertainment system providing anything beyond sound or display output and perhaps networking, since passengers usually have a powerful device like a smartphone or tablet with them which they replace more often than the car. The same also goes for airplanes.

I use the train a lot here in Germany, and while the railway company gets a lot wrong in the software / digitalization realm at least they resisted the temptation to outfit the trains with an entertainment system like you have on some airlines. Instead you can log-in to the on-board wifi over which they also offer a streaming service (persumably with lots of content cached on the train), since people bring their own devices anyways.

> So, yes, piling on tons of gadgets and features to otherwise basic form of transportation is a big way they can help plump up the profits of their dealer networks.

You can diversify a lot in the realm of sales here by offering choices so no two car buys are similar and then upselling the hell out of it - "want the soundsystem? You also have to get the undercoating." I never personally bought a car but from what I hear it's a very unpleasant experience, especially if you don't like to haggle.

Linux in mixed-criticality systems

Posted Dec 20, 2018 0:40 UTC (Thu) by rahvin (guest, #16953) [Link] (3 responses)

You should not trust them and there are significant risks to the convergence model. Look no farther than what's happened with Toyota's unintended acceleration issues after they moved to drive by wire. A code error created a situation where a car would accelerate indefinitely, and the transmission would look the shifter so it couldn't be slipped into neutral and the result was a bunch of fatalities. Toyota is going to pay out a LOT of profits by the time all those lawsuits are over.

And the remarkable thing about these suits was just how long it took and how much legal effort was expended just to prove it. IIRC it took almost 3 years of legal work before a set of lawyers were able to get discovery granted to the source code to the systems that caused the problem and a proper analysis was done.

With the move to drive by wire systems we need government regulations and code quality and testing requirements that protect life safety possibly even a requirement for a switch that turns the whole drive system off while preserving braking and steering or some other failsafe. One of the reasons cars were so bloody reliable from 1990 to about 2008 and didn't have these type of issues was because everything was designed previously as single controllers and ASIC's for every function so that a computer failure would only take out a single system not the whole drivetrain. This convergence is very scary to me and was vindicated by the finding on the Toyota acceleration issue where it was a code quality issue. Life safety needs very high standards and multiple failsafes.

I design roadways for a living and I'm legally required to consider safety before any other variable at the risk of my possible incarceration for failure to do so. I believe vehicle source code should be held to a similar standard.

Linux in mixed-criticality systems

Posted Dec 21, 2018 7:35 UTC (Fri) by murukesh (subscriber, #97031) [Link] (2 responses)

Do you have some reading material on this? I looked through https://en.wikipedia.org/wiki/Sudden_unintended_accelerat... and that article seems to claim that the problem was not with electronics or code, but with the pedal design (and the floor mat). Or are you describing another SUA incident?

Linux in mixed-criticality systems

Posted Jan 3, 2019 16:02 UTC (Thu) by sdalley (subscriber, #18550) [Link] (1 responses)

Lots of gripping stuff for the embedded software engineer here. I found it compulsive reading.

https://embeddedgurus.com/barr-code/2013/10/an-update-on-...
http://www.safetyresearch.net/Library/Bookout_v_Toyota_Ba...

Toyota's Unintended Acceleration

Posted Jan 6, 2019 1:02 UTC (Sun) by marcH (subscriber, #57642) [Link]

Fascinating, thanks.

Whether these bugs are real or not and no matter who "wins" this eventually, Toyota and others should already have learned a lesson the hard way: every time human life is at stake legal costs alone far outweigh any development cost saved by sloppy programming. While a rather low bar it's still a somewhat "good" bit of news.

People cope with getting screwed over again and again by $BIGCORP and the 0.1% but funny enough there's always this line that you really can't cross: they don't like to... die.

Now rebooting my smartphone to work around some memory leak and other random issues...

Linux in mixed-criticality systems

Posted Dec 20, 2018 20:12 UTC (Thu) by kamil (guest, #3802) [Link]

> I have hopes for companies like Tesla

Well, those hopes may be misplaced. Tesla, innovative though it is, has a well-deserved reputation of being extremely proprietary. There are plenty of stories flying around about the difficulties of reverse engineering what its cars do internally, etc.

Heck, recently they announced that they will stop working with third-party body shops and that they want all such repairs to be handled in-house. Compared to Tesla, traditional car manufacturers are a model of openness...

Linux in mixed-criticality systems

Posted Dec 14, 2018 11:32 UTC (Fri) by smurf (subscriber, #17840) [Link]

There are reasons besides shaving a few $$ off the cost of a car why you'd like tighter integration between safety critical systems and outside-accessible code.

The typical car (i.e. basically anything that's mid-range or better) consists of a veritable heap of embedded processors with no way to log anything substantial if/when something goes wrong, and with no way to update them from the outside. Tesla excepted …

Linux in mixed-criticality systems

Posted Dec 14, 2018 12:02 UTC (Fri) by nhippi (subscriber, #34640) [Link] (8 responses)

OTOH having a separate microcontroller for everything means a lot more wires and connectors, which are usually the places of faults on modern cars. Putting things on the same die might actually make things more reliable.

Internet connectivity and cloud services OTOH are much big risks.

Linux in mixed-criticality systems

Posted Dec 14, 2018 15:11 UTC (Fri) by marcH (subscriber, #57642) [Link] (7 responses)

> OTOH having a separate microcontroller for everything means a lot more wires and connectors,

Not for everything; for safety-critical systems. Cars used to be certified and pretty safe before micro controllers so there can't be that many safety-critical micro controllers in a car today.

Reducing the number of micro controllers and wires is great and all and sure you also want to add some partitioning and "soft real-time" there but that's different.

Linux in mixed-criticality systems

Posted Dec 14, 2018 15:24 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

By the way safety critical ethernet already exists and is already in use in planes for instance. Reducing the number of wires doesn't require reducing the number of microcontrollers, those are two different problems with several orders of magnitude difference in complexity.

Linux in mixed-criticality systems

Posted Dec 14, 2018 17:01 UTC (Fri) by ibukanov (subscriber, #3942) [Link]

Older planes still use ARINC 429 protocol which is one way serial link where one source can talk to multiple clients. But since it is used only in aviation, the boards are really expensive. It is cheaper just to use Ethernet and require special verified routers where software is formally verified.

Linux in mixed-criticality systems

Posted Dec 14, 2018 16:54 UTC (Fri) by smurf (subscriber, #17840) [Link] (3 responses)

> Cars used to be certified and pretty safe before micro controllers

Yeah, but that was before brake assistants and airbags and automatic transmissions and mandatory seatbelt warnings and central locks and theft protection and whatnot. All of these, and more, either require µCs outright or are more expensive if you build them without one.

Today? No way. Even less way if you want a car that doesn't burn gasoline.

Linux in mixed-criticality systems

Posted Dec 14, 2018 17:33 UTC (Fri) by marcH (subscriber, #57642) [Link] (2 responses)

mandatory != safety critical

Central locks and rear cameras are "real-time" (for some definition of it) but really not safety critical.

Linux in mixed-criticality systems

Posted Dec 20, 2018 6:26 UTC (Thu) by JdGordy (subscriber, #70103) [Link] (1 responses)

pretty sure there is a requirement for rear cameras to turn on within ~2s of power, and quite likely there is actual safety critical requirements on central locking (if fitted obviously) around how they work in accidents

Linux in mixed-criticality systems

Posted Dec 20, 2018 6:39 UTC (Thu) by marcH (subscriber, #57642) [Link]

My car doesn't have a rear camera, so it must be critically unsafe.

Check what it takes to open a damaged car door.

ABS when fitted is a good example considering its job is to ... release the brakes. Cruise control and any other sort of self driving feature are other obvious ones. Can't wait for the fun of seeing these interact with security holes and other bugs in the media player. As usual the only winners will be the lawyers.

Linux in mixed-criticality systems

Posted Dec 14, 2018 17:38 UTC (Fri) by mpr22 (subscriber, #60784) [Link]

Safety-critical microcontrollers in production automobiles have (with initially narrow market penetration) been with us for nearly fifty years, in the form of anti-lock braking systems.

Linux in mixed-criticality systems

Posted Dec 14, 2018 15:43 UTC (Fri) by marcH (subscriber, #57642) [Link]

We all here know our skills are in short supply and we've worked with people who lack some of the most basic software/hardware engineering practices yet are assigned business- (not safety-) critical tasks anyway. Our industry is well known for its... "youth".

You'd think car makers are paranoid about safety and you'd be right. The problem is: that doesn't magically make them able to attract and hire good computer professionals (bar super rare exceptions like Tesla).

This is where I find the security "circus" extremely valuable: hacking a phone is one thing, hacking a car makes much bigger headlines and shows the emperor is naked.

I've also heard a couple software engineering horror stories from the inside and it's actually not pretty.

Please consider this presentation for what it is: a very interesting but *long term research* project and wish none of it shows up in your car tomorrow.