Another look at response times [LWN.net]

Two weeks ago, this page compared the response times of several distributors to a small set of recent security issues. That article generated a number of comments and a fair amount of mail from people who felt that its conclusions were inaccurate. As before, the table shows the number of days required for each distributor to issue an update. For the purposes of this table, the clock starts when a vulnerability is disclosed, or when the first distributor alert is issued, whichever comes first. So, here is a new version of the response times table which takes those comments - and alerts issued after publication - into account:

Vulnerability Distributor

Debian Fedora Gentoo Red Hat SUSE Ubuntu

Apache mod_ssl 14 9 21 11 14 12

clamav 22 3 3 n/a 5 --

evolution -- 1 13 19 6 1

fetchmail 22 0 4 4 7 5

PCRE 13 4 14 18 16 3

PHP XML-RPC 9 4 5 6 7 4

PHP XML-RPC 2 18 10 9 4 15 5

ProFTPd 35 -- 4 n/a n/a n/a

vim modeline -- 16 n/a? 28 n/a? 1

Vulnerability	Distributor
Debian	Fedora	Gentoo	Red Hat	SUSE	Ubuntu
Apache mod_ssl	14	9	21	11	14	12
clamav	22	3	3	n/a	5	--
evolution	--	1	13	19	6	1
fetchmail	22	0	4	4	7	5
PCRE	13	4	14	18	16	3
PHP XML-RPC	9	4	5	6	7	4
PHP XML-RPC 2	18	10	9	4	15	5
ProFTPd	35	--	4	n/a	n/a	n/a
vim modeline	--	16	n/a?	28	n/a?	1

In the above table, numbers which are underlined reflect alerts issued after the previous version. Those which are, instead, bold are corrections for erroneous entries published two weeks ago.

As one can see, a number of corrections were required. One might conclude from this that your editor was being even more clueless than usual when compiling the previous version of the table. One would probably be right, but there is a little more to it than that. It turns out that putting together a table like this is a hard thing to do.

The previous version stated that Fedora had not issued an advisory for clamav. That is, in fact, true; no advisory ever came out. The clamav package in Fedora Extras was quietly replaced, however, shortly after the vulnerability was disclosed. In the presence of silent fixes, it is hard for users to know if they are vulnerable or not; this is doubly true in cases where security fixes are backported to previous releases of the affected package. Fedora Extras does not do backporting, but it still requires an alert administrator to know that, while clamav has been fixed, ProFTPd in Extras remains vulnerable.

Speaking of ProFTPd, your editor had seen that package in a SUSE distribution he had at hand, and assumed it was still distributed. That turns out not to be the case.

Both SUSE and Gentoo claim to not be affected by the vim modeline vulnerability because they ship versions with the modeline feature turned off by default. Turning off a possibly insecure feature is a good thing to do; it reflects a concern by the distributor for the security of its users. Some of those users, however, will certainly turn the feature back on. Others will be concerned by the fact that they are running software with a known, unpatched vulnerability, even if that vulnerability does not directly affect them. In such cases, it would make sense for the distributor to, at a minimum, issue an advisory explaining the situation. Putting out a fix would be better.

Other corrections above reflect simple screwups on your editor's part. Sorry.

The corrected table still shows some real patterns in the relative response times for security updates. There is value in this information. As time permits, LWN will be making changes to its security database to make the generation of this sort of table an easier and more accurate process. But a task which, in the presence of nice things like CVE numbers, should be relatively straightforward is likely to require a fair amount of time (and iterations) for the foreseeable future.

to post comments

Another look at response times

Posted Sep 22, 2005 4:06 UTC (Thu) by bangert (subscriber, #28342) [Link]

As time permits, LWN will be making changes to its security database to make the generation of this sort of table an easier and more accurate process. But a task which, in the presence of nice things like CVE numbers, should be relatively straightforward is likely to require a fair amount of time (and iterations) for the foreseeable future.

This is great news - I am looking forward to it. Thanks.

Another look at response times

Posted Sep 22, 2005 9:24 UTC (Thu) by james (guest, #1325) [Link]

... your editor had seen that package in a SUSE distribution he had at hand, and assumed it was still distributed. That turns out to be the case.

I assume that last sentence was supposed to contain a "not"...

Thank you for the article, and the concerns you raised. I've been noting the lack of Fedora Extras security advisories: it is too difficult to work out what needs upgrading. I understand that the article has led to a certain amount of discussion on the Fedora Extras list.

The original article was a splendid example of the LWN.net effect: a well-written, intelligent and thoughtful article followed up by a number of clarifying points, many from the people involved in the work in question. The result is a clear and authoritative view of the subject.

And it's why LWN.net is worth the money.

James.

Another look at response times

Posted Sep 22, 2005 9:36 UTC (Thu) by mjcox@redhat.com (guest, #31775) [Link]

For Red Hat we publish all our raw data so anyone that is interested can figure out the response times for any subset of issues. There's a perl script along with the data so you can look at say all the critical issues that affected RHEL3 in the last 3 months for example.

http://people.redhat.com/mjc/

We tend to update this page once or twice a month. (So if you see something take 28 days for example then it's probably because we rated it as having a lower severity than other issues at that time).

Another look at response times

Posted Sep 22, 2005 15:52 UTC (Thu) by zooko (guest, #2589) [Link] (2 responses)

Tables should come with a label telling what the values in the table are. In this case I'm guessing it is "days between (public announcement of) vulnerability and fix". Is that right?

Another look at response times

Posted Sep 23, 2005 14:22 UTC (Fri) by Duncan (guest, #6647) [Link] (1 responses)

Yes, it is "days", but IDR from the original article exactly how they are
measured. Of course, the original article is linked, but one shouldn't
have to look to it to find the missing info.

I agree with you, the table should be labeled. LWN usually does a pretty
good job at that sort of thing, but, perhaps due to the followup nature of
this article, usual thorough proofing may not have been applied.

I've seen Corbet respond in similar instances that while such comments are
appreciated, they'll often get faster action if mailed. IIRC, the address
suggested to mail them to is the general lwn at lwn dot net address, as
listed under the contact faq. The faq mentions that "somebody is always
watching" that address in any case, so it'd be the best for correction
type notes, I'd guess.

I'll mail a request for a label to it as soon as I post this.

Duncan

Another look at response times

Posted Sep 24, 2005 14:01 UTC (Sat) by Duncan (guest, #6647) [Link]

.. And they copied the necessary description from the previous article,
fixing the issue, as I had hoped.

=8^)

Duncan

Yeah, but...

Posted Sep 24, 2005 1:16 UTC (Sat) by roelofs (guest, #2599) [Link] (2 responses)

So, here is a new version of the response times table which takes those comments - and alerts issued after publication - into account:

Updating the incorrect entries is all well and good, but if you're going to issue a correction, why not go all the way? As someone else pointed out in the original comments, Mandriva has an excellent record (at least by this metric), and I included both that and Slackware in my own plain-text version of the table. It wouldn't have taken that much longer to verify and/or update those two columns for this article...

Greg

Yeah, but...

Posted Sep 24, 2005 2:25 UTC (Sat) by corbet (editor, #1) [Link] (1 responses)

Several people also thought that the table needed to include a wider set of vulnerabilities. Yes, I could have done all that, but... I was back on Monday with thousands of emails to wade through, and three kernel page articles, three front page articles, and a security lead to write by Wednesday. Plus a daughter who is desperately behind on her violin practice. So the security article didn't get as much time as it might have. Sorry.

Yeah, but...

Posted Sep 24, 2005 3:54 UTC (Sat) by roelofs (guest, #2599) [Link]

Argh, sorry--I completely forgot about your out-of-townness. Been there, done that, and it usually takes me three full weeks to get caught up... I hereby withdraw all of my whining. ;-)

Greg