LWN.net Logo

Security updates for embedded systems

The Avaya S8500 Media Server is a product which "allows for a distributed enterprise over an IP infrastructure in the mid-market space (up to 3200 ports)." Whatever that means. It fits in a 1U rack space; it also, as it happens, runs Linux - Red Hat Enterprise Linux, in particular. So, when Red Hat recently produced a security update for its kernel, Avaya sent out an advisory of its own. As it turns out, however, Avaya has classified this set of vulnerabilities as being of "low" concern, so, by the company's posted policy, there will be no software update coming anytime soon. Instead, the fixes will be packaged up with the next regular operating system update.

In the mean time, however, Avaya has a helpful suggestion:

For all system products which use vulnerable versions of the kernel, Avaya recommends that customers restrict local and network access to the server. This restriction should be enforced through the use of physical security, firewalls, ACLs, VPNs, and other generally-accepted networking practices until such time as an update becomes available and can be installed

Restricting network access will certainly make a network server product more secure, but it might just interfere with the tasks said server was purchased to perform in the first place.

In a separate episode, your editor was recently wandering around on the net in search of a fix for some obnoxious behavior exhibited by his DSL router. As it turns out, this router runs Linux. One can telnet into it and wander around. It's always amusing to discover that one is running even more Linux systems than had been previously thought. This one is built upon a MontaVista distribution, and is running a 2.4.17 kernel.

As LWN readers may have noticed, there have been a few security issues discovered in the 2.4 kernel after 2.4.17 was released. Quite a few. The support services sold by MontaVista to its customers must certainly include security updates, but there does not appear to be any mechanism for getting those updates through to the end customers who will actually be running vulnerable software. That is true even in your editor's case, where the router was obtained directly from the local huge telecom company, which should have good records regarding the equipment at its customers' homes. Said large telecom company tends to be held in rather low esteem by its customers, but, even so, one might expect that it would make a minimal attempt to keep those customers (who are, in the end, connected to its network) secure.

The end result is that your editor's DSL router - a Linux system with all the power BusyBox can deliver - almost certainly contains known security holes. It has writable flash storage, and can run programs uploaded to it. This is a rather discouraging situation when one considers that, for many users, this router will be the front gate to their home or small business network. The potential for mass mayhem is real.

In both cases, we are seeing situations where Linux systems have been deployed into security-relevant roles, but the security update mechanism has not kept up with them. As Linux pushes its way into more low-end consumer-grade devices, this problem will multiply. Who thinks about applying security updates to their telephone? And which manufacturers of cheap consumer electronics will concern themselves with pushing security updates to their customers?

Linux systems can be quite secure, especially when they are pared down to a minimal set of functions. But one of the things that keeps Linux secure is the quick closing of known security holes, and the quick dissemination of those fixes to deployed Linux systems. Without that support structure in place, Linux systems (like all others) become vulnerable to holes discovered after they were built.

Embedded systems tend to lack that support structure. When the system is, say, a music player with no connection to the wider world, there is no particular cause for concern. Network-connected devices, however, are subject to attack. Fortunately, network-connected devices should also be able to detect and install security updates - though setting up such a mechanism in a way which does not create privacy concerns can be a challenge. It should be a solvable problem.

The use of Linux in embedded systems is a cool thing - especially if those systems are designed to allow improvements by their users. It is one more step toward World Domination. But that cause could be set back significantly by a single Linux-based router or cellphone worm. We do not yet know how to create systems which will remain secure indefinitely into the future. Until that problem is solved, we must maintain structures which can close vulnerabilities as they are discovered. Purveyors of embedded systems ignore that need at their peril.


(Log in to post comments)

Security updates for embedded systems

Posted Sep 5, 2006 17:24 UTC (Tue) by HenrikH (guest, #31152) [Link]

Well, to be honest this is not an Linux issue since the situation is the same regardless of OS used on the device, wheter it by Symbian or Embedded Windows. The real issue is that the companies that distributes these devices sees them as hardware and hardware cannot have security issues right :)

Security updates for embedded systems

Posted Sep 5, 2006 19:42 UTC (Tue) by sepreece (subscriber, #19270) [Link]

I think they see them as hardware and think "hardware is what it is". That is, they don't generally think about the possibility of upgrading devices after they're shipped, unless they turn out to have something that is clearly a defect (like catching fire). Security vulnerabilities would typically not be perceived as at that level.

Of course, many of these devices are configured in ways that remove at least some of the typical vulnerabilities (not running all the normal daemons, not listening to all the sockets that a laptop would, etc.), so you'd have to look pretty closely at a particular device to see whether any individual vulnerability applied.

Also, lack of upgrades typically applies at the lowest tier of Linux-based devices, like routers. More complex/expensive devices (like set-top boxes and DVRs) tend to get periodic upgrades, at least if they're provided by a service provider. Don't know, though, whether those upgrades typically go all the way down to the OS, or only upgrade the application software. And, if you buy one without a service provider, you're probably on your own.

I completely agree that the scene is ripe for a major incident that will diminish Linux's reputation...

Security updates for embedded systems

Posted Sep 8, 2006 7:36 UTC (Fri) by simlo (subscriber, #10866) [Link]

I remember from a tradeshow that Windows XP Embedded comes with an automatic update function. It simply connects to a server and download the needed patches.

GPLv3 and Security updates for embedded systems

Posted Sep 5, 2006 17:42 UTC (Tue) by atai (subscriber, #10977) [Link]

How is this viewed in light of the GPLv3 and DRM debate, that the users may not have ways of updating the software themselves, maybe by intent?

GPLv3 and Security updates for embedded systems

Posted Sep 6, 2006 18:13 UTC (Wed) by landley (guest, #6789) [Link]

A) Never attribute to malice what can be adequately explained by
stupidity. With cheap embedded devices they're generally pretty happy if
you buy them and smash them for sculpture materials; it means you BOUGHT
them. Modding them violates the warantee, but that's just because they
don't want to support it. Support costs money. Updates are support.
Updates cost money.

B) The terms of GPLv3 will never be enforceable against BusyBox. We're
never dropping GPLv2 (although we might go GPLv2 only once GPLv3 ships and
we have something specific to reject).

C) Show me a way of phrasing GPLv3's "enforced upgradability" requirements
that does NOT require the World of Warcraft people to give me root access
to their servers if they're using BusyBox on it somewhere and I sign up
for a WoW account.

FUD

Posted Sep 7, 2006 0:32 UTC (Thu) by stevenj (guest, #421) [Link]

Show me a way of phrasing GPLv3's "enforced upgradability" requirements that does NOT require the World of Warcraft people to give me root access to their servers if they're using BusyBox on it somewhere and I sign up for a WoW account.

Please, spare us the ridiculous scare scenarios. GPLv3 would require no such thing.

Servers USE software

Posted Sep 8, 2006 21:58 UTC (Fri) by sepreece (subscriber, #19270) [Link]

If the WoW servers run BusyBox, that's considered use of the software, which GPLv3 explicitly acknowledges requires no license, even if they chose to modify the version they run on their software.

Now, IF they sold a hardware client - say a controller running Linux with the WoW client software running on it, then if you got one, GPLv3 would give you the right to replace the version of BusyBox running on it, with the resulting client functioning as it did before, other than any changes made by your changes.

That requirement would be an argument against their choosing Linux for the client box, since they presumably want to require fair clients and can only assure that if they control the client software. However, I don't think it's an issue, since I doubt they plan to build hardware clients, anyway...

Servers USE software

Posted Sep 14, 2006 7:48 UTC (Thu) by anandsr21 (guest, #28562) [Link]

So what you are saying is that if they sold their hardware, and if Linux sported GPLv3 then they will have to give the source to their server software. That is ridiculous.
They can do all they want with their proprietory server code. They don't need to make them open. Their code is not GPL and is not part of the kernel. If they don't want modded clients they can make use of the following scheme.
1) Let the client register the first time it connects to the server.
2) they provide the buying code, which should come with the box and should be different for each box.
3) Ask for a name, from the client. Use the code, the name, the IP subnet, and a random number to make a identifier. The name should not be changed on subsequent registrations.
4) Use this identifier to identify the client.
5) Whenever the user changes his IP subnet, they must register again. At this time remove the older identifier.

This would be mostly foolproof. Ofcourse people could share in the same subnet. But at least user will not be able to post it on the internet. Which is the biggest problem. And you can easily disallow one ID connecting twice. Since people who know each other like to play with each other. Sharing will not be a satisfying solution.

6) If somebody steals a users id, it should be easy re register.
The software should not keep the code or the name anywhere on the system. And should tell the user not to do so either. This will make guessing the code and name simultaneously nearly impossible.

Yes this is more difficult than making sure that the hardware is secure. But for the hardware that a company sells, they shouldn't get the right to keep it closed. For leased hardware there may be a case.

GPLv3 and Security updates for embedded systems

Posted Sep 7, 2006 3:32 UTC (Thu) by cventers (subscriber, #31465) [Link]

> Show me a way of phrasing GPLv3's "enforced upgradability" requirements
> that does NOT require the World of Warcraft people to give me root
> access to their servers if they're using BusyBox on it somewhere and I
> sign up for a WoW account.

If you feel that this is a legitimate concern, perhaps contacting the FSF
about it might be appropriate? They at least claim they are taking
community input, and appear to me to be doing so.

GPLv3 and Security updates for embedded systems

Posted Sep 7, 2006 11:55 UTC (Thu) by coriordan (guest, #7544) [Link]

Yes, this is an important comment which has to be made repeatedly this year. If people submit their complaint to gplv3.fsf.org, it can be acted on.

Here's the newsforge article which suggests the comments are being listened to.

And here's FSFE's GPLv3 page which has info and links to info such as eight presentation transcripts.

GPLv3 and Security updates for embedded systems

Posted Sep 7, 2006 18:56 UTC (Thu) by dwheeler (guest, #1216) [Link]

Poster said: "C) Show me a way of phrasing GPLv3's "enforced upgradability" requirements that does NOT require the World of Warcraft people to give me root access to their servers if they're using BusyBox on it somewhere and I sign up for a WoW account."

I've read all the major versions of the GPLv3, and NONE of them require any such thing. Where in the world did you get such an interpretation?!? If the WoW are sold a device with BusyBox, then under GPLv3 the WoW would have to be given the right to upgrade the BusyBox they were using. But that doesn't mean anyone ELSE gets to upgrade your software.

I think you've been misinformed. Please check your sources again, and if you still think there's a problem, send a comment to the GPLv3 process. But I think this is a gross misunderstanding of GPLv3.

GPLv3 and Security updates for embedded systems

Posted Sep 14, 2006 9:08 UTC (Thu) by forthy (guest, #1525) [Link]

Typical anti-GPLv3 soundbite. Why do people do this? IMHO, GPLv2 did already clearly express the intent of updateability, though not as precise as GPLv3. TiVo-izing GPLv2 software IMHO is a violation of the license, because the intent of the license is already clear, and the way TiVo-izing works hasn't been forseen by the licensor. Remember that copyright licenses require an additional license for unforseen usage. All you could do (e.g. as Linus Torvalds does) is state your opinion about it, but it's your opinion, and not the opinion of other developers of the system. So people can't take it as legal advice.

Security updates for embedded systems

Posted Sep 5, 2006 19:26 UTC (Tue) by tzafrir (subscriber, #11501) [Link]

It is not that difficult to build an updates system for the embedded system that will securely nd automatically apply fixes from the vendor. It is feasible to test the fixes beforehand, as there aren't too many possible configurations.

The problems are:

* Users are afraid of the device "phoning home" occasionally and automatically installing software from there.

* The vendor will eventually stop maintaining the device.

Security updates for embedded systems

Posted Sep 5, 2006 20:01 UTC (Tue) by sepreece (subscriber, #19270) [Link]

While I agree with the editor that vulnerabilities like this are eventually going to cause a major problem and blacken Linux's reputation, at least a little, I think the danger is more at the level of the DSL router he mentioned than the Avaya Media Server.

First, the long list of vulnerabilities are almost all local-user exploits, which probably wouldn't apply to a media server.

Second, all but one of the remote-user vulnerabilities are DoS cases - causing a panic or infinite loop. Those would be severely irritating, but not so much so as opening access to data or allowing remote access to the device.

And, third, the recommendation from Avaya is probably mostly reasonable. Most people with a box like this are going to have it on a network, but they're going to have some network protection out in front of it. Requiring physical security and controlled access to the network would not be an issue for most such users. They didn't say take it off the network, they said to protect it from untrusted access, which most businesses would do routinely (I hope).

However, again, I completely agree with the description of the general problem of unpatched Linux running in lots of little devices, like the DSL router, all over the internet, most of them much less protected than that Avaya box is likely to be. As Linux grows, the number of attacks tailored to Linux vulnerabilities will increase and the targets will include those devices...

Clarification - Linux isn't the problem

Posted Sep 5, 2006 20:10 UTC (Tue) by sepreece (subscriber, #19270) [Link]

Just to be clear - I'm not accusing Linux of being at fault here! The situation would be exactly the same if the devices were running some other OS. The problem is the unpatched, potentially vulnerable devices. Linux at least makes it possible, in some cases, to update the software at the user's end. [Not that that matters much in practical terms - the vast bulk of DSL routers are in the homes of people who would have no idea how to upgrade them even if they somehow became aware of the problem.]

I think the danger to Linux's reputation is higher because it has a much higher profile than the real-time OSes it replaced in devices like this. Nobody outside the trade would have noticed if, say, VRTx had a similar problem, but the Linux name is getting enough general-audience traction now to be vulnerable.

Clarification - Linux isn't the problem

Posted Sep 5, 2006 21:06 UTC (Tue) by HenrikH (guest, #31152) [Link]

And not to mention that it would be easier for the vendor to blame Linux for this since it is a well known name in contrans with VRTx who nobody in the general public would ever have heard of.

The DSL routers is a special sad case since it would be so easy for the ISPs to implement the upgrade due to that they own and controll the very network that these routers are connected to.

Clarification - Linux isn't the problem

Posted Sep 7, 2006 14:59 UTC (Thu) by jschrod (subscriber, #1646) [Link]

The problem with Linux is also that it might make a target for script kiddies in the future, as it is a brand-name that's popular. Attacks on well-known brands are more likely, after all.

Concerning routers -- I don't want to know how many CISCO routers are on the internet that are not patched. (E.g., I know that the CISCO router that I got from my ISP, and is owned and managed by the ISP, does not get patched.) But this doesn't seem to bother anyone, except me. Since no wide-spread worms target CISCO systems, most customers let it go through; and those who care, get told that "it's not part of the process".

Well, I wouldn't trust them anyhow, for me that router is an external untrustworthy system on the Internet.

Cheers, Joachim

Security updates for embedded systems

Posted Sep 6, 2006 3:54 UTC (Wed) by tzafrir (subscriber, #11501) [Link]

Here's a real-life example for the sake of discussion:

There's a certain common ADSL modem in Israel that can also be configured to be used as a NAT router. It uses the same 2.4.17 kernel and monte-vista-based system.

Two well-known problems of it:

1. The builtin caching DNS server (an old version of dproxy) seems to have an issue with Linux clients. Maybe this is something to do with ipv6. However the modem will always give the address of itself as the DNS server.

2. It is very linux-guru-friendly: you can login with ssh and/or telnet and get a shell. This requires password. However the web interface also has a nice loophole to run an arbitrary shell command without even requireing authentication. Seems like an intended "technician's loophole".

Luckily that web interface is only availble from the LAN.

The system seems to date to 2004.

Security updates for embedded systems

Posted Sep 6, 2006 10:12 UTC (Wed) by DennisJ (subscriber, #14700) [Link]

The only thing where Linux has a potential to make this problem larger is monoculture. I don't think there is a reason to think that non-Linux embedded systems are any more secure or any better upgraded than the Linux ones.

But there is *a lot* of work to fix this from the suppliers of the embedded systems. As an example of the level of security awareness of the DSL router field: I recently got a new ISP, they supplied a router, and a card showing wiring, and then it should "just work".

Before the line came up, I hooked up the router, and took a look at it's web administration tool. Everything was indeed set up to "just work". So the builtin wireless was started, unencrypted, and open for all. Remote administration was available through telnet, www, and ftp without restrictions of sides of the router or IPs.

And of course the administration page itself was protected by the default password of: "1234", which it was good enough to tell me itself. To be fair: It instantly prompted me to change it. But how many home users would have even gotten there when the system "just works"?

My estimate is very few indeed, considering the names of wireless networks I have access to from my own apartment: "Belkin54g", "default" and "Linksys" which are all kind enough to "just work" for me as well as their owners and whoever else might be around.

Security updates for embedded systems

Posted Sep 6, 2006 10:15 UTC (Wed) by warmcat1 (guest, #31975) [Link]

Yes there's a serious tension between the requirement for security updatage and the consequent potential of box breakage.

Not everyone in the hierarchy around a product can understand the importantance of a security issue which is only theoretical until exploited, but everyone can understand meddling with boxes at unwittingly happy customers and breaking or bricking them. Since not updating costs nothing until there is an actual crisis, that is the path of least resistence. I guess the "fix" will be to make security updates available, but make it the customer's problem and action to go get them.

Security updates for embedded systems

Posted Sep 6, 2006 13:29 UTC (Wed) by nedrichards (subscriber, #23295) [Link]

If the telecoms company had thought about it properly they would have put a remote upgrade path for the box they gave you in there.

DISCLAIMER: I work for the same multinational (although a different bit) as this story.

Orange Broadband (one of the larger ISPs in Europe) lends to all its customers (the deivce is still owned by Orange) a WiFi ADSL modem running Linux. This 'Livebox' is made by Inventel [http://www.inventel.com/]. Normally you'd expect the same problems as mentioned above to occur but becuase the device is *only* going to be plugged into an Orange network we just send out firmware updates for it, automagically rebooting the box after its installed. Now this isn't a perfect system (and doesn't make it any easier for Linksys etc.) but for any network operator where the vulnerable device is connected into their network really should have a system like this.

Security updates for embedded systems

Posted Sep 6, 2006 16:49 UTC (Wed) by ortalo (subscriber, #4654) [Link]

Security updates, security updates, security updates...
This is an endless race, one you cannot win. Maybe I am getting too old or maybe I am simply getting bored at monitoring updates and their potential failures or even the updates to the previous updates (the last case, admittedly, never for Linux systems...).
However, I am definitely getting bored with security updates. I am a real-life computer security officer so I need to use systems that do not have security faults.
And no, I am not dreaming. This update frenzy started somehow 7-8 years ago.

Before that, everyone in the security field was looking after eliminating security bugs, gathering security assurance, using mandatory access control, extending code audit, using formal methods, discussing law enforcement, modifying contracts to incorporate requirements, educating users, and I certainly forget a lot of other nice and/or tricky ideas.

Nowadays, much of the energy surrounding systems with security requirements is targeted at this useless update race while it could be much more useful elsewhere in the security field. Do you really think patching a (t)ftp server will ever be truly useful for its security if your requirements simply imply that you cannot use the (T)FTP protocol? What about *not* using an application? Oh, btw, anyone has heard about the money problems of the OpenSSH project developpers?

Maybe it is time to face our responsibilities: switch *off* updates, *shutdown* vulnerable systems and go after those that deliberately refuse to introduce security requirements when they were necessary (either to fire them, to ask for reparation or simply to flame them).

You know what? This is one of the main reasons I still think that Linux cannot compete with OpenBSD with respect to security. And, this is not a problem: the reverse is certainly true for other kind of requirements.

You know, I don't ask for any change at all (albeit a minor one). Personnally I really enjoy using and loving 2 very good different systems!
Sometimes, as a computer administrator or developer, it seems to me that anyone needs to learn that he needs to use something *else*.
Oh, and switch off those damn security updates to replace them with software upgrades (cryptographically signed of course).

Security updates for embedded systems

Posted Sep 6, 2006 17:43 UTC (Wed) by tzafrir (subscriber, #11501) [Link]

So could you please comment on the example I gave above? (DSL router with two issues: (1) bug in the dns proxy server, not security hole, but hurts some of the users, and (2) a silly loophole left intentionally by the system's builder (or not removed unintetionally).

If you start to make too strict requirements for a small gadget, it will fail to work. E.g: you need a simple method for the user to recover the master password (or equivalent) in case it was forgotten.

Nither of those issues have anything to do with the basic OS. Your favourite OS does not seem to share your sentiment: http://openbsd.org/security.html#39

Security updates for embedded systems

Posted Sep 6, 2006 20:18 UTC (Wed) by ortalo (subscriber, #4654) [Link]

Concerning (1), I'd like to know easily if I am one of the users hurted. In fact, I'd like much more information on the risks associated with a security issue than on the administrative actions I "must" do "immediately". (Even my boss does not use these words!) This is of course due to the fact that risk management is essential to decide what to do (either spend time on the administration and patching or elsewhere) and sometimes it depends on the overall architecture - something the distribution maintainer or the developper cannot know in advance.
Note that proposing a good risk evaluation is certainly pretty difficult. Currently, the general attitude (especially outside of open-source software) seems to be: no risk until the bug is fixed ("no disclosure") and maximum risk afterwards ("apply all fixes immediately"). Of course, this is absurd. System administrators are humans, they *can* cope with uncertainty pretty easily.

Concerning (2), this is a clear indication to me that the system builder should be avoided due to unresponsible behavior. You need guarantees and the presence of a loophole is an indication that your guarantees are void because you cannot trust the furnisher. You shouldn't have bought the system in the first place so you probably need to review your purchase procedure too... You can continue to use the device, but maybe not exactly to do all you wanted to do...

I do not really agree with your second paragraph. Requirements should be good or exact (adapted to the covered risk), but they may be very strict without compromising the success. If you look at mobile phones for example: the bluetooth stack security is usually pretty funny (at least, not very strict and not very convenient isn't it?), but the GSM security protocol with the operator is older and very good (and you *can* recover the PIN code). Maybe that's because the monthly billing amount was involved?

Finally, yes, it does not really have something to do with the kernel itself. More with the entire system. Note I do *not* have a favourite OS - I really like both Linux and OpenBSD. (Security requirements are not always in first place for me, of course.)

The OpenBSD "updates" are also pretty interesting in their form don't you think: they are *only* available in source code patch form. It implies a lot of non-obvious things: they are nearly useless on a running system installed only with binaries, they open by default with a text editor (and you can see the extent of the problem and the complexity of the solution, e.g. comparing the impact of 2006-006 with 2006-009: 30s), dependencies with other source code is dealt with on your *own* computer when you rebuild the (entire) system. In fact, much of the security management is left to the end user and he is provided with information more than with replacement programs. Linux systems have much in common with this (a good thing imho) but they tend to "like" binary updates too.

Most other commercial systems only offer binary "patches" and I wonder if anybody really cares about what's inside. Nowadays no one questions anymore the general consensus "it must do more good than harm". My problem is that I question this consensus when I am on my provocative day...

Security updates for embedded systems

Posted Sep 7, 2006 6:04 UTC (Thu) by grahammm (guest, #773) [Link]

For recovering the master password, the solution is relatively simple. You have a hardware way (eg keeping the reset button pressed rather than just press and release) to locally perform a 'reset to factory defaults'. OK, this not only resets the password but also the user has to re-input any configuration etc.

Automatic updates for embedded devices

Posted Sep 6, 2006 17:04 UTC (Wed) by sdoyon (subscriber, #4221) [Link]

It's no excuse for a vendor not to provide firmware updates, but it's
interesting to see how well embedded free software distributions do.
I know a little bit about OpenWRT and Familiar. I know the OpenWRT
folks will not stay put when problems arise such as a vulnerability in
the dropbear SSH server. But updating often means reflashing and
reintegrating your customizations. It's definitely not as seemless as
when a workstation pulls updates off of a Debian security feed in the
middle of the night. You would want your PDA running Familiar to be as
secure as your laptop, but the Familiar distribution doesn't have the
resources and infrastructure of a big distro. At least it's based off
of Debian too. Of course doing per-package updates has to be tricky
when you have very limited storage space on flash, and when geek users
tend to customize their setups a lot, and when just about every
program under the sun is available as an add-on package.

Automatic updates for embedded devices

Posted Sep 7, 2006 9:03 UTC (Thu) by appie (subscriber, #34002) [Link]

This issue is, as stated in other comments, a generic one (i.e. not Linux, *BSD, Windows, embedded-OS-of-choice specific).

A lot of people view appliances as "hardware". Forgetting that almost all hardware for a long time now needs to run software too. Certainly a lot of vendors sell it as hardware, as a product. An el-cheapo DSL router comes with an attractive price attached to it. Mind you: the vendor is able to deliver it to the market for that price because they don't take into account the maintenance.

Maintenance is not a product, it's a SERVICE.

A lot of people seem to compare software to cars. I personally think that's wrong. Apart from the fact that the various components that make up a car are easier to prove (yes, as in Proving software) it's comparing apples and rollercoasters (to get the point across :) anyway.

I'm not happy nor indifferent with/to the fact that us humans creating the software make errors and/or aren't given enough time to check for errors. But that's what it is: a fact. Software doesn't roll out of the "factory" mostly proven correct.

Software needs maintenance (as does a car right ? but we shouldn't compare them :-)
It feels wrong to pay for fixing mistakes you didn't make. After all it's not mileage that brings out the breakage in components, it's the actual design that was wrong.

I guess one can blame vendors that make huge profits and could be held responsible for not spending some of that money on better quality control.
Out of responsibility a vendor should at least mention that people could be at risk if they don't take into account the maintenance side of things.
And if people actually pay for support and updates they should get them too.

Automatic updates for embedded devices

Posted Sep 8, 2006 16:57 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

It's not clear what distinction you're drawing between cars and software. And also why you're comparing the two, since this thread is really about devices that contain software, and as such, cars are clearly included. The article does emphasize Internet-accessible devices, so maybe we can stick to those.

You seem to imply a router with Linux inside is more complex than a car, therefore should be expected to have bugs, and so we should expect to apply updates to the router, while not to the car.

A router is not more complex than a car, even with millions of lines of code in it. What makes the difference between a car and a router is that the car is mostly hardware, which makes it more expensive to upgrade in the field. Therefore, producers spend more on removing the bugs during design. With the router, it's in society's best interest to leave some bugs in and fix them in the field, thus getting more function for less money overall.

There could be some dispute over the risk of bugs in a router, leading a producer to ship one with bugs and also without field upgradability, and that's really the point of this article. But that applies to cars too.

Maintenance is not a product, it's a SERVICE

BTW, this uses a strange definition of "product." Normally, a product is whatever a company produces, and lots of products are partly or entirely service.

Free software on embedded systems

Posted Sep 7, 2006 11:58 UTC (Thu) by coriordan (guest, #7544) [Link]

I saw yesterday that Rockbox has been ported to the iriver H10. I did some investigation on that device and have decided to buy one later. Hopefully I'll have time to do a blog entry on what I had to do and what Rockbox is like.

(And "Linux" isn't important - the Tivo has linux, but it also has spyware you can't remove and it purposefully lacks features. Free software is cool.)

Cost and time is the problem, not Linux

Posted Sep 7, 2006 12:30 UTC (Thu) by plougher (subscriber, #21620) [Link]

For many years LWN seems to have concentrated its efforts/reportage on enterprise level Linux. It's good to see some more focus on smaller embedded systems.

From my personal experience, the biggest issue that embedded Linux suffers from WRT security is the cost and time pressures that consumer embedded systems are designed under. There is rarely enough flash or RAM to properly design secure firmware update methods, or enough time to properly audit for security of the system being released (let alone future problems). One system I'm aware of included an automatic firmware download system not because of security update concerns, but because they knew the software was full of bugs, and it would have to be updated in the field at a later date. In this scenario security is of a low concern. As long as Linux is adopted in embedded systems due to its cheapness over proprietary licensed systems, this will only get worse.

On a slightly different topic, it's nice to see from the link that the router is yet another system using SquashFS. Slightly disappointing that this wasn't mentioned in the article even though its use of MontaVista Linux and Busybox was mentioned. Also because Squashfs is read-only, the filesystem isn't directly writable, which does lessen security issues.

Cost and time is the problem, not Linux

Posted Sep 8, 2006 11:08 UTC (Fri) by wookey (subscriber, #5501) [Link]

Indeed.

It is perfectly possible to include an update mechanism. My snom VOIP phone ,for example, has such a mechanism; accessible from the menus on the device and from the web interface via an 'update firmware' button. It can update both the system image and the bootloader. This mechanism allows you to change the URL so in theory can still function after snom have stopped supporting the device, and it works painlessly whilst snom have updates in the default location. It isn't automatic, in that I get to choose when to put in an update, not snom. This does of course mean that it won't get done by most users. It was so easy to use that I actually used it.

The automatic vs. user-instigated choice is a tricky one. People don't like surprise upgrades (especially if it breaks anything) or 'phone home' devices, but most DSL routers are plugged in and ignored by their owners, so defaulting them to get upgrades is probably good for both customers and the ISP/Cable/phone company. Said company usually owns the device in these cases and thus is entitled to set the update policy. A well-designed one would let you disable this feature if you wanted to retain a veto.

But as Mr Lougher said, it is not trivial to implement a proper update scheme on a severely resource-constrained device: (do you have room to store a whole spare system image before flashing it? - if not do you have room for a package database and management system, are you absolutely sure your update or the update mechanism isn't going to brick the box?). These things tend to get ignored or left till later rather than fundamentally designed in. They are not important enough to receive proper design attention during the system design so that there _is_ enough space for a spare image. They probably should be, but it'll take a major case of infected/DOSed installed routers for manufacturers to take it seriously, especially at the low end where cost is everything.

The smart ones amongst us should use stuff we know will remain supported and be updateable (WRT54G+openWRT, iRiver+rockbox etc). I know I never buy anything anymore that I can't at least theoretically maintian independently, even if I can't necessarily be arsed in practice.

Copyright © 2006, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds