Debian and Suhosin

A recent proposal for Debian to stop shipping PHP with the Suhosin security patches has been controversial. There are a number of reasons behind the proposal—manpower, sticking to the mainline, performance, and more—but others responding in the thread consider the security mitigations that Suhosin provides to be very important for the web application language given its less than stellar security track record. What most would like to see is that those protections make their way out of the Suhosin patches and into the PHP mainline, but that does not seem to be in the offing. In the meantime, users may find that the PHP protections they have depended on will disappear from Debian.

Debian PHP maintainer Ondřej Surý posted a message to several lists noting that the Suhosin patches have been disabled in the unstable repository and tries "to summarize the reasons why I have decided to disable suhosin patch" in the message. Over time, he has changed his mind about Suhosin, so he is documenting the reasons and looking for other opinions. The Debian PHP team is evidently understaffed, and the work to add in the Suhosin patches (and module) eats up some of that time. Surý is not convinced that the extra time is necessarily well-spent because PHP has "improved a lot".

By shipping only a Suhosin-enabled PHP, Debian is diverging not only from the mainline, but also from what other Linux distributions do. That means that users coming from other distributions (like Fedora which doesn't ship Suhosin or openSUSE where it is optional) may run into problems they don't expect. In addition, he said, bugs reported upstream from the Debian version are often met with a request to reproduce it in vanilla PHP. There are also performance and memory usage impacts from Suhosin that some find excessive.

Suhosin grew out of the PHP hardening patch that was developed in 2004. The basic idea is to add protections against bugs in the PHP core (aka Zend Engine) by making proactive changes for things like buffer overflows or format string vulnerabilities. It also tries to protect against badly written PHP applications, of which there are seemingly countless examples. Suhosin has two parts, a patch to the PHP core along with a PHP extension that implements additional hardening features.

The core patches are what try to protect against buffer overflows by adding canary values to internal data structures so that the overflows can be detected. In addition, the pointers to destructors (i.e. functions that are called when an element is freed) for internal hash tables and linked lists are protected as they can be a vector for code execution if a buffer overflow overwrites them. Format string vulnerability protection and a more robust implementation of realpath() round out the changes to the core.

The extension provides a whole host of other kinds of protections, largely against dodgy PHP programming practices. For example it protects against either remote or local code inclusion, which is one of the worst problems that has plagued PHP applications. It can disable the eval() call, prevent infinite recursion by putting a limit on call depth, stop HTTP response splitting attacks, filter uploaded files by a variety of conditions, and on and on. While it obviously can't prevent all badly written PHP from running amok, it's clear that the Suhosin developers have looked at a lot of common problems and tried to address them.

While most of the features are configurable, they are all going to impact performance in one way or another. That's a tradeoff that many seem to be willing to make, especially in shared hosting facilities where a vulnerability in a particular customer-installed application (or the version of PHP itself) might have serious repercussions for other customers. As the project's "Why?" page notes, it comes down to a question of trust:

But there is an additional reason for dropping Suhosin mentioned in Surý's posting: "Stefan's relationships with PHP upstream (and vice versa) isn't helping very much". He is referring to Stefan Esser, lead developer of Suhosin. Surý's statement is borne out in the thread, as there seems to be a fair amount of animosity between Esser and other posters in the php-devel list. But beyond the personalities involved, there is a more important question: if the hardening features in Suhosin are truly useful, why haven't they been pushed upstream?

A footnote in Surý's post refers to a section of the Suhosin FAQ that outlines the project's reasons for staying out of the mainline. It mentions the performance impact of the patches and that some PHP developers are not interested in adding code to protect against badly written applications. It also notes that by staying separate from the core, the project can make a statement about what it sees as deficiencies in security handling in PHP. But there seems to be more to it than that.

Various people have encouraged Esser to create RFCs for the features in the patch that he thinks should go into the mainline, but he largely dismisses those messages with statements like:

Esser clearly believes that all of the Suhosin changes should go into PHP without going through the RFC process. As Stas Malyshev pointed out, though, that's part of the collaboration process:

Malyshev also notes that it is hard for Esser to complain that the PHP developers aren't cooperating when he is unwilling to join with the project and follow its processes. But Esser is convinced that it would be a waste of his time:

Esser is also concerned that PHP developers are not paying enough attention to security overall. He pointed to a fix that recently went in to address problems in the HTTP response splitting protection, where, even though it is a security-related fix, there was inadequate review of two different attempts to fix the flaw. The first fix for the bug (bug #60227) went directly into PHP 5.3 back in November. Esser's complaint is strongly worded, but does point to a real problem:

We've heard those kinds of arguments before in a slightly different context. The grsecurity/PaX patches for the Linux kernel have been around for quite some time, but have always been maintained as out-of-tree patch sets. The pseudonymous "PaX Team", who maintains the PaX patches, made many of the same arguments about why those patches have not been submitted. It is certainly attractive to be able to go your own way, without having to coordinate or convince anyone outside of the project, but it does have its costs as well.

One of those costs is a reduction in the audience because distributions and others may shy away from non-mainline code. Even if the maintainer of an out-of-tree patch set does a perfect job (impossible, of course), there is a cost associated with using a non-standard tool, whether it's a programming language or kernel. That cost is borne by the distributions and some may be unwilling to start (or continue) bearing that cost.

One thing to note, however, is that Suhosin has been pretty effective at avoiding various PHP bugs along the way. The recent PHP remote code execution vulnerability found by Esser was thwarted by a suitably configured Suhosin. The HTTP response splitting problem was also solved in Suhosin long ago. On the other hand, certain (likely buggy) applications cannot run under Suhosin which also makes it difficult to adopt.

There is a fundamental problem, at times, connecting upstreams and security researchers. Free software encourages folks to scratch their own itch and that's just what Suhosin and grsecurity/PaX have done. But if the changes never make it to the mainline of the project, users may be suffering from bugs that could be avoided. The "all or nothing" approach is not likely to work with any project, but it is true that security issues are often not given the attention they deserve in those upstreams. It's a difficult problem to solve, but projects would be well-served by finding a way to cultivate more security-oriented developers into their communities.