|
|
Subscribe / Log in / New account

How long should security embargoes be?

By Jake Edge
February 8, 2012

Security embargoes are something of a double-edged sword. The idea is to coordinate a release date for a particular security fix between the affected parties (typically distributions in the Linux world). But, while the embargo is in place, no fixes are being issued which increases the length of time that users are vulnerable. That could lead to widespread—or targeted—attacks against vulnerable systems if the flaw is already known, is leaked, or is rediscovered during the embargo period. The chances of that may be relatively small, but they certainly increase as a function of the length of the embargo.

It is for those reasons that embargoes in the Linux world are typically short. While proprietary vendors will sometimes sit on a vulnerability for months, Linux embargoes are generally on the order of a week or two. The rules for the acceptable length of an embargo are set by the venue where the information is shared, which for Linux distributions is the closed linux-distros mailing list. There is also an associated distros mailing list which also adds in representatives from some of the BSDs. These two lists have taken the place of the vendor-sec list that was compromised in March 2011.

Up until recently, the guidelines for linux-distros specified that the maximum allowable embargo was 14 days. That means that anyone reporting a bug to the list should be willing to wait that long to publicly release any information; it also binds list participants to that deadline. The idea is that anyone who doesn't want to agree to embargoes of up to 14 days shouldn't be using linux-distros. As part of the discussion of the bug and fix on that list, a coordinated release date (CRD) would be chosen so that all distributions would release at the end of the embargo period.

List administrator Alexander Peslyak (aka Solar Designer) recently made a change to the guidelines to better reflect reality. There is an effort to avoid having a CRD that falls on a Monday or a Friday to try to not land on a holiday or other inconvenient day for administrators who may need to do lots of updates because of the security fix. That meant that the 14 day maximum sometimes stretched to 19 days, so Peslyak changed the page to reflect that. He also posted to the open oss-security list to highlight the change.

There were no complaints about the change, though there were some alternatives suggested (10 business days for example), until Peslyak followed up his message suggesting that perhaps the 14-19 days be reduced to 7-11 days. Even that length of time is longer than he would like, "but I am proposing what I think has a chance to be approved by others without making the list a lot less useful to them".

Peslyak also outlined his reasons for preferring shorter embargo windows. Distributions that have the fix ready more quickly can get it into the hands of users sooner, rather than waiting a week or more for the embargo to end. Also, it reduces the window of time in which the vulnerability could be rediscovered or leaked from the list somehow. There are also logistical concerns with longer embargoes including increased tracking of multiple overlapping embargoes.

Both Marc Deslauriers of the Ubuntu security team and Kurt Seifried of the Red Hat security response team were quick to disagree with a one-week (more or less) embargo period. Depending on the severity of the bug and the difficulty of the fix, it may be hard for some distributions to pull together, test, and release fixes in that time frame. In particular, Seifried is concerned about volunteer-run distributions that may lack the staff to ensure fixes in a shorter period. But Deslauriers makes another important point:

This means vendors will be keeping information about the vulnerability private until they are confident they are able to release within a week, at which point they will then share the information with other vendors who will scramble to get their updates ready.

As a distro, I now have two choices: I sit on vulnerabilities until our own QA and testing is done, at which point I send them to the list and hope that 7 days is enough for everyone else, or I simply stop using the list for anything that's more than trivial and contact other vendors directly.

That's the fine line that Peslyak is walking here. If the embargo requirements become too onerous (or seem that way), distributions may stop reporting to the list—or only report after they have made progress with a fix. But, other lists have other rules. The closed kernel security list says that it will do embargoes up to 7 days, but it seems to rarely happen that way, undoubtedly partially due to Linus Torvalds's distaste for embargoes. That policy may also result in distributions delaying reports to the kernel security list, of course.

It should be noted that these "closed" lists have been fairly leaky at times. In addition to the vendor-sec list compromise, the kernel security list may have been breached as part of the kernel.org compromise. The linux-distros list now uses PGP-encrypted emails, which should help unless the host doing the re-encryption to each member's key is breached.

There are also dangers from fixes that are rushed, of course. The recent PHP remote code execution vulnerability came about because of a rushed fix for a different security hole. There is certainly value in taking some time to get a security fix right, the only question here is how long that should be, and, for full disclosure types, whether users should be made aware of the problem while the fix is in progress.

In the end, the 14-19 day window stayed, though a preference for embargoes of less than 7 days was added. It's a difficult problem, partly because there are so many unknowns. Fixes can be difficult to apply (particularly if they need to be backported) and to test, especially for multiple distribution releases. But the longer the bug is embargoed, the longer users are at risk without any ability to mitigate the problem locally while awaiting a fix. That's why the full disclosure camp believes that information on security holes should be released without delay, so that users are empowered to make their own decisions. That approach is not very popular with vendors, of course. Embargoes and their length is an issue that will likely be debated again and again because there isn't an "obviously correct" solution.


Index entries for this article
SecurityBug reporting
SecurityDistribution security


to post comments

How long should security embargoes be?

Posted Feb 9, 2012 4:17 UTC (Thu) by arjan (subscriber, #36785) [Link] (2 responses)

I've been around distros small and big for a long time now, and personally I've come to the conclusion that I will not do embargoes for things I have any say or choice in, and I will not join lists that enforce embargoes on my behalf. [I've had people try the "but we put you on the CC with such a list on the CC as well so you're also under the embargo", but I just laughed at the guy... that was just so sad it was funny]

The tradeoff of leaving a wide range of users vulnerable during the embargo does not, in my mind, extend to 2 weeks... If you as a distro can't get a mitigation out in 24 hours, with maybe a more complete fix in 72 hours, then frankly, fix your internal processes. That's not a reason to keep people vulnerable for a very long time.

How long should security embargoes be?

Posted Feb 9, 2012 10:26 UTC (Thu) by Klavs (guest, #10563) [Link]

Not embargoing - might also help people ensure they have defence in depth - so one "currently released and unsolved" vulnerability doesn't hit them so hard.

How long should security embargoes be?

Posted Feb 10, 2012 23:39 UTC (Fri) by PaXTeam (guest, #24616) [Link]

users aren't vulnerable during the embargo only, they're vulnerable as long as they use the buggy code. the latter is usually much much longer than the former so a few days more or less for an embargo doesn't really change anything. actually i'm surprised you'd go public with such a statement considering your participation in one of the worst handled linux security bugs of all times. to refresh your memories, this is what was posted to vendor-sec on 2003.09.25:

<arjan> there's a security hole found by akpm
<arjan> that also hits your kernels
<arjan> Subject: [PATCH] do_brk() bounds checking
<arjan> that patch you want
<arjan> agreement is to put it in silently (eg no changelog)
<davej> ok
<arjan> it's not exactly public stuff either
<arjan> linus committed it with a non-security comment
<arjan> so should we
<davej> ok

and the result of this was the now infamous debian core infrastructure compromise a few weeks later. what did you want to prove again?

How long should security embargoes be?

Posted Feb 16, 2012 3:24 UTC (Thu) by slashdot (guest, #22014) [Link] (4 responses)

How about no embargo window at all?

There's no really no excuse for not being able to deliver a fix in 8-24 hours at most, assuming you have proper automated processes set up, and people distributed across timezones to allow 24/7 response.

And about holidays, well... if a security bug is discovered, one would hope people would work on holidays and at night too without interruption to fix it ASAP!

Honestly, someone who does this commercially and cannot push a security fix within 24 hours ought to be prosecuted for negligence.

How long should security embargoes be?

Posted Feb 16, 2012 8:10 UTC (Thu) by kleptog (subscriber, #1183) [Link] (3 responses)

I'm wondering what kind of testing you can seriously do in 24 hours. Probably just enough to prove that it doesn't crash your system or wipe any disks within 24 hours. In any case, just because someone has found a bug doesn't mean you have a patch to fix it.

There are classes of bugs, like missing bounds checking, which can be quickly deployed with little risk but the recent /dev/mem kernel bug and the PHP debacle are subtle enough that it can take >24 hours just to determine what the correct fix is. A hastily applied fix can make the problem worse.

How long should security embargoes be?

Posted Feb 16, 2012 9:01 UTC (Thu) by slashdot (guest, #22014) [Link] (2 responses)

For kernel issues, just run the testsuites for all packages in the distribution, plus any other generic testsuite/stress test you have, automatically in parallel on a cluster.

Plus, manually test as needed (for /proc/pid/mem, should take less than an hour to write a program to test it still works).

A competent programmer working an entire day should be able to fix pretty much any bug; if not, the feature is most likely totally broken and can thus be disabled in that time.

I strongly doubt Linus Torvalds needed more than a day to fix the /proc/pid/mem issue for instance.

How long should security embargoes be?

Posted Feb 27, 2012 4:22 UTC (Mon) by dirtyepic (guest, #30178) [Link]

Testsuites aren't going to find security issues opened up by rushed bug-fixes. Nor are they going to find kernel bugs that aren't blindingly obvious. Testsuites catch regressions in the package they're testing. Someone fixed a bug and added a test to make sure it didn't happen again. I'm not saying they won't catch bugs here and there, but relying on them for security is foolish.

And even if all distros had access to clusters of machines of every security-supported arch, you'd have to get upstreams to give a shit about unrelated testsuite failures before they'd become of any use whatsoever. In my experience, it's been largely "Well it passes on my machine, sucks to be you". Some upstreams are better than others (libtool always impressed me, even going as far as to offer to add a workaround for a broken version of a in-house tool of ours), but most regard test failures that don't happen to them to be someone else's problem. This is understandable of course; I'd rather work on something more interesting too.

I'm not going to reply to the rest of your message since I can't seem to come up with a response that would be appropriate in public.

How long should security embargoes be?

Posted Mar 1, 2012 12:42 UTC (Thu) by nix (subscriber, #2304) [Link]

A competent programmer working an entire day should be able to fix pretty much any bug
Er, no. Very much no. Bugfixes can take arbitrarily long periods of time, particularly if the bug turns out to be a design error. (My personal record time to fix a bug is six years -- holding off a design change until other events allowed the change -- and, like anyone, I have a number of bugs outstanding in my own code which will likely never be fixed because they are too minor or the required design change would be too large for the benefit.)


Copyright © 2012, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds