Bitten by old bugs [LWN.net]

Michal Zalewski recently publicized a couple of denial of service problems with the Postfix mailer. Distributors responded quickly; here's a quick look at who released updates and when:

Distributor	Updated versions	Response time (days)
Conectiva	7.0, 8	1
Debian	3.0 (woody)	0
EnGarde	Community 1.0.1, 2 Professional 1.1, 1.2, 1.5	1
MandrakeSoft	8.2, 9.0, Corp. Svr. 2.1 Firewall 8.2	1
Red Hat	7.3, 8.0, 9	1
SuSE	7.2, 7.3, 8.0, 8.1...	1
Trustix	1.2, 1.5	3

(See the LWN vulnerability entry for current information on distributor updates). Here, "response time" is calculated as the number of days between the posting of Michal's advisory and the distributor update. Distributors clearly had a bit of advance notice with which to produce their updates, which is a good thing. There was very little delay before updates were made available to users.

The only problem is that, as Postfix creator Wietse Venema pointed out, both of the vulnerabilities had been fixed a long time ago. One of the problems was fixed in version 1.1.12, released in November, 2002. The other (fixed in 1.1.13) does not exist in Postfix 2.x, which has been available since February. But even relatively modern distributions (such as Red Hat Linux 9) are built with version 1.1.11, which dates back to May, 2002. It is laudable that the distributors were so quick to make updates available. But if they had stayed a little closer to the current release of Postfix, much of this scramble might have been unnecessary, at least for more recent distribution releases.

One can always come up with possible reasons for the shipping of such old software. For most distributions, only a small minority of users run Postfix, so it is probably relatively low on the prioritized list of packages to update. Switching to a new major release (2.0) is always a bit of a scary move; distributors tend not to rush into that sort of change. And, then, there is the little fact that neither fix was marked by the Postfix developers as a security fix. As we have seen in this case, distributors move quickly when a security issue is outstanding, but slowly otherwise.

The fixes were not advertised as being security related for a simple reason: the developers did not know - in either case - that a security bug was being fixed. One fix just sort of happened during a big (2.0) code reorganization, and the other fix looked like just another bug fix at the time. The end result is that, as a result of inaction on the part of both developers and distributors, users have been running vulnerable code for months when a fix was available.

to post comments

Bitten by old bugs

Posted Aug 14, 2003 5:01 UTC (Thu) by ncm (guest, #165) [Link] (3 responses)

All bugs are security holes until proven otherwise.

That's why OpenBSD doesn't announce security holes they find when they fix bugs in inherited packages. Instead, they advise users to run the latest versions of everything. They also advise anybody who cares about the security of those packages to watch (and, one hopes, apply) OpenBSD's patch list.

Last year I fixed a bug in Gnome libxml2 -- a double-free error that could be evoked, if I recall correctly, by supplying bad XML text. Exploits of double-free errors have appeared in the wild, although they are a little harder to make work, and more dependent on exact versions, than your typical buffer overflow. Nobody announced a security hole in libxml2. Should they have? Bugs are getting fixed all the time. No exploits have been reported yet. Arguably, though, libxml2 is used widely enough now that it demands the level of auditing thus far reserved for mail transport agents and secure remote login services. XML from all over gets parsed by all kinds of programs, some of which run privileged, and all of which (anyhow) run on systems that are easier to crack from a login than remotely.

As it is, any time somebody competent wants to break into a host somewhere, it suffices to look at the list of bugs fixed in package versions released after the host was installed. Only the tiniest minority of those packages will have had security alerts (which get ignored by maintainers of 40+% of exposed hosts anyhow). While the majority of bugs probably turn out not to have been exploitable, it's remarkable how many are. (Where no fixed bug serves, new features often bring easily-detected bugs of their own, so change logs for older versions offer black hats a second chance.)

This suggests that the Debian Project policy of preferring to keep old versions of packages around, and backporting only patches reported as security problems, might be unwise. For security-critical programs and the libraries they use, it might be better to maintain forks that get all applicable bug fixes, not just those attached to security advisories. While bug fixes bring their own risks, those risks don't generally involve marauders behind your firewall.

I don't know of any case of a security hole deliberately introduced along with a bug-fix source code patch. Although that is conceivable, it would be risky and anyway unnecessary; there are plenty of bugs already there to exploit. The ready supply of old bugs is probably also the reason that we have so rarely seen attacks on code repositories such as just happened to the FSF's ftp site. They just haven't been necessary. If we ever start treating all bugs (and new features) as potential security holes, these other avenues might become more attractive.

Bitten by old bugs

Posted Aug 14, 2003 6:17 UTC (Thu) by proski (guest, #104) [Link] (1 responses)

While the majority of bugs probably turn out not to have been exploitable, it's remarkable how many are.

However, the attacker needs only very specific vulnerabilities. If the attacker has no local account, s/he needs a remote exploit. Many holes require social engineering to be exploited - they may be useless against smart users. Most software is not directly listening to the ports - it needs to be run by other software or by one of the user (often by root). Double free in a piece of software that nobody runs on untrusted data is useless.

The ready supply of old bugs is probably also the reason that we have so rarely seen attacks on code repositories such as just happened to the FSF's ftp site.

As far as I know, it was a brand new bug that didn't have a fix released at the time it was exploited.

Bitten by old bugs

Posted Aug 14, 2003 16:13 UTC (Thu) by ncm (guest, #165) [Link]

Libraries used by programs that do keep open ports are rarely treated with the same care as those programs, even though they contribute equally to vulnerability. Also, programs running at reduced privilege are not treated as seriously even though vulnerabilities there lead to local shell access and to the much richer exploit opportunities available that way.

The FSF ftp server compromise occurred not because a new hole opened up, but because somebody found a hole that was there and used it. Probably there are dozens more, and it might have happened dozens of times before, but not so many people are interested yet. As it is, the best that can be done is to reduce the number of people who have logins.

Bitten by old bugs

Posted Aug 15, 2003 13:57 UTC (Fri) by pimlott (guest, #1535) [Link]

This is absolutely true, and I'm glad you've explained it so well. This idea of backporting only "security" fixes sounds like a prudent engineering principle, but the reality is far less pretty. As you said, there's just no way that all fixed bugs can be evaluated for their potential exploitability, and missing just one is enough.

Furthermore, running a patched older version leaves you running a configuration that nobody upstream is even thinking about. It's like taking a boat out without telling anyone. With the current version, you have a whole community watching your back. (Say a bug is found in the current version that happens not to be exploitable because of some other change since your version. Is anyone going to tell you?)

In practice, the best software development teams (and I'm not just talking about OpenBSD) have a simple "don't leave bugs open" discipline. You take some risk by continuously fixing bugs, but with a little intelligence, the risk is very small and entirely acceptible. And the benefits are, of course, in more than security: every bug hit is an alienated user.

Frankly, many small projects don't even really need separate stable branches, because even "development" releases are highly unlikely to break anything. That said, bigger, more complex projects should have stable branches on which bugs are fixed aggressively. If they don't, the distributions should pick up the slack, but they should encourage upstream to do it, so there are more hands and eyes involved, and the whole user base benefits.

The OpenBSD guys ought to be among the most qualified to distinguish "security" bugs from other bugs, and even they don't do it. Debian shouldn't either.

Bitten by old bugs

Posted Aug 14, 2003 11:06 UTC (Thu) by erich (guest, #7127) [Link]

Which Trustix Date is correct?
Trustix is listed with 3 days delay here, but in the advisory list below it is on the same day as Redhat, SuSE, Mandrake.

I'm a happy Debian user, and for my servers i am really happy about their "conservative" strategy for updates. Of course with high-quality software such as postfix it is ok to track the latest versions with just a couple of days delay. In fact i'm running postfix 2 on my servers, so i wasn't bitten by the bugs.
But this doesn't apply to all software. Most applications show this feature-additions/bugfixes cycle, and it is very good to stop at the less-features-less-bugs point. In fact most software has too many features IMHO, more that you will need, and more that you will verify to work ok.
The Gnome project also noticed this and does a good job IMHO trying to keep the features to the needed ones. Whenever i launch the KDE Control Center i'm scared by the amount of - mind if this sounds rude - useless crap you can do there. It's like being able to open the read windows in your car too from the control panel at the drivers position. It's nice, but you also more easily forget to close all windows.

Affordability of QA

Posted Aug 14, 2003 14:57 UTC (Thu) by cdmiller (guest, #2813) [Link]

Unfortunately most distributions do not have the resources to run full QA on the latest version of all software packages every time a new version of the software is released. As any seasoned SysAdmin will tell you, stability comes first. Upgrade for the sake of upgrade is generally not a good approach. Of course, companies like Microsoft or Sun would have you upgrade every time they manage to make their binaries 50% larger.