LWN.net Logo

Cascading security updates

By Jake Edge
February 27, 2008

When following the distributions' security updates on a daily basis, as we do at LWN, certain days are more work than others. Two weeks ago we had a rather full update with no less than 28 packages updated for Fedora (most of those for both F7 and F8), along with a handful of updates from other distributions. It turns out that the majority of the Fedora updates had a single cause: a set of serious vulnerabilities in Mozilla Firefox.

How does a single update to an application ripple so far that more than a dozen packages have to be rebuilt? One would think there would be shared libraries that would get updated, with applications picking up those changes the next time they are run. That is, in theory, how things are supposed to work, but in this case, the underlying libraries have no fixed application binary interface (ABI). So, changes to those libraries require any applications that use them to be rebuilt and retested.

Gecko is the rendering engine used by Mozilla in their products to display HTML. Various other packages have started using it as well because of its speed and standards compliance. Because Mozilla sometimes breaks the ABI between releases, even minor releases, distributions may be stuck rebuilding those applications when a new version of the library is released. Normally, that only happens when packaging a new version of the distribution—or when serious security flaws are found.

Mozilla's solution for this problem is XULRunner which will provide a stable ABI for applications. As XULRunner and its companion libxul become more widely available, the applications that currently link to the Gecko libraries will presumably switch to avoid these kinds of problems in the future. It is highly unlikely that we have seen the last security problem in the Gecko engine, so reducing the cascade that results from finding one would be welcome.

Because of problems with the ABI changing in the past, Fedora chooses to make the applications' library version number exactly track the Mozilla release number. Some other distributions do not do that, so unless the ABI does change, they do not need to update each package that uses the libraries. This has some advantages, but could lead to broken applications if an ABI change goes unnoticed.

We have also seen similar cascades of updates, most notably from the xpdf PDF viewer. Unlike Gecko, there is no library for xpdf, leading multiple applications to include its source into their own. When a flaw is found, several different applications (cups, gpdf, etc.) across all distributions need to be updated immediately, leading to a similar effect as was seen with the Gecko vulnerabilities. Hopefully, over time, the development of the poppler library will mitigate this problem somewhat.

There are lots of good reasons to separate code into components where possible, but security is an important one. Creating and maintaining an ABI is sometimes difficult, but generally worth the trouble. Imagine the chaos that could result from a security vulnerability requiring an ABI change in glibc.


(Log in to post comments)

Cascading security updates

Posted Feb 28, 2008 4:23 UTC (Thu) by magnus (subscriber, #34778) [Link]

A good example of ABI backwards-compatibility is GTK+, which is still compatible back to
version 2.0, released 6-7 years ago. 

I wish more effort was made on ABI forward-compatibility (symbol versioning), which very few
libraries seem to care about. Without it, you have to build your program against the oldest
version of the library that you depend on to get a portable binary. This would also make
things like distribution upgrades a lot safer and more flexible (like being able to upgrade
one package at a time).

Cascading security updates

Posted Feb 28, 2008 10:56 UTC (Thu) by nix (subscriber, #2304) [Link]

Symbol versioning doesn't help there unless your applications explicitly state which versions
of symbols they use (which none do because it's so annoying).

Instead they implicitly rely on a GNU extension whereby a symbol with a @@ between name and
version (rather than a single @) becomes the default version and is used if none is specified.
This is invariably the latest symbol version available for that library. So you don't gain
backcompat: what you gain is forward-compat (until you relink, anyway).

Cascading security updates

Posted Feb 29, 2008 20:14 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

Symbol versioning actually makes the backward compatibility problem worse. Take an old program foo.c and compile it on your new system. Transport the program to an old system, and it won't run. That's because it calls baz(), which at compile/link time turned into a reference to the new Version 2 of baz().

Without symbol versioning, baz() designers would instead introduce a new baz2(), and maintain compatible header files for baz(), and the old foo.c would be unaffected.

Furthermore, a coder of a new foo.c might see in the manual that baz2() is a recent invention, and thus write foo.c to use the more ubiquitous baz(). It's hard to expect a coder to use explicit symbol versions the same way.

Last, but not least, the error message you get when you try to run that newly compiled foo.c on the old system is very cryptic, even to a seasoned software engineer.

Cascading security updates

Posted Feb 28, 2008 13:10 UTC (Thu) by davecb (subscriber, #1574) [Link]

There's a classic discussion of ABI maintainability
by Paul Stachour (which I typed in) at Multicians.ORG:

http://www.multicians.org/stachour.html

--dave

Cascading security updates

Posted Feb 28, 2008 7:15 UTC (Thu) by jimparis (subscriber, #38647) [Link]

One example of cascading chaos was with Zlib vulnerabilities such as CAN-2005-2096 and CAN-2005-1849. So many packages statically link to zlib that binary scanners were developed to try to find them all. If I remember correctly this has resulted in a strong preference (at least within Debian) to always link zlib dynamically whenever possible, to avoid such problems in the future.

Cascading security updates

Posted Feb 28, 2008 8:32 UTC (Thu) by edschofield (guest, #39993) [Link]

I was one of the people puzzled about the recent avalanche of Gecko-related updates. Thanks for the good explanation.

Cascading security updates

Posted Feb 28, 2008 10:50 UTC (Thu) by nix (subscriber, #2304) [Link]

Times really have changed when people are using Gecko because of its *speed*. :)

(of course, this is largely because it's sped up, although computers getting faster has played
some part.)

(btw, KHTML *does* have a published API and stable ABI. ;} )

Cascading security updates

Posted Feb 28, 2008 12:50 UTC (Thu) by Frej (subscriber, #4165) [Link]

Hmm, is there really anything that uses xpdf and not libpoppler today? (Other than xpdf
itself...)

Cascading security updates

Posted Feb 28, 2008 14:48 UTC (Thu) by nix (subscriber, #2304) [Link]

Way too many things: I think I still have more copy-of-xpdfers on my local system than poppler
users. One which springs to mind (because I upgraded it yesterday) is CUPS.

Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds