Stenberg: Pre-notification dilemmas [LWN.net]

Curl maintainer Daniel Stenberg expresses some frustrations with the vulnerability notification policies maintained by the distros mailing list.

The week before we were about to ship the curl 8.0.0 release, I emailed the distros mailing list again like I have done so many times before and told them about the upcoming six(!) vulnerabilities we were about to reveal to the world.
This time turned out to be different.
Because of our updated policy where the fixes were already committed in a public git repository, the distros mailing list’s policy says that if there is a public commit they consider the issue to be public and thus they refuse to accept any embargo.
What they call embargo I of course call heads-up time.

The kernel project has run into similar issues in the past.

to post comments

Get rid of the distro embargo period

Posted Mar 29, 2023 19:56 UTC (Wed) by geofft (subscriber, #59789) [Link] (4 responses)

Honestly, it seems like the right answer here is for distros to automate and streamline the process of releasing security updates so that an embargo period is unnecessary. The embargo can't be perfect, anyway - the distros need to publish their packages to a public repository, and the customers who actually run the code will take some time to pick up that update and install it everywhere. So the embargo period isn't the goal in and of itself; it's a tool to ensure that the time between public awareness of the vulnerability and fixes being applied to the actual systems is as small as possible.

As I understand it, the reason for the embargo is to provide time to validate the fix (ensure that it fixes an actual bug, it really fixes the bug, and it's not a duplicate of something already tracked), package the fix, minimally test it (it cannot be fully tested by the distro, because the fix is strictly embargoed from all customers), and prepare an announcement with relevant information including a CVE ID. Most of these steps can be automated - compared to when the linux-distros list was created, we have not just cheaper / faster computing resources but also fairly easy ability to get those resources in parallel instead of serial form. Even ten years ago you'd probably want to run builds on your own resources; now it's sensible to run at least tests and probably builds too on a large cloud provider's resources, and they will sell you 1000 computers for one minute for the same price as one computer for 1000 minutes. Especially if a patch against a stable release is provided, distros should be able to the computational work in much less than an hour.

(Of course, I'm sure I'm missing something practical in this "why don't they just..." analysis, and I'd love to know what it is. :) )

That leaves the work of backporting the fix, which really requires some human review if not action; coordinating and validating the fix; and assigning a CVE ID and writing up the vulnerability. For established projects like curl, it makes sense to entrust the upstream author with the coordination and validation work and perhaps to pre-assign them CVE IDs. (The CVE assignment process was a mess for many years, but I'm under the impression it's better now, and it's certainly a fixable mess either way, so it's not a good defense of a long embargo period.)

So that leaves just backporting the fix to the distro's choice of stable branch, which certainly requires human effort, but not multiple days of effort. For the commercial distros, they should be able to have a couple of engineers with an on-call schedule and commit to a response time in hours.

At that point, your embargo period is shorter than the likely time it will take customers to notice and apply the fixes, so you may as well get rid of it.

Note that this proposal doesn't impact embargoes for the purpose of researching the vulnerability and developing fixes - it allows the upstream project to continue to have its own internal embargo period for however long they like, and it also allows distros to accept confidential security vulnerability reports and relay them to the upstream project. That embargo period is the one at the heart of the "full disclosure"/"responsible disclosure"/etc. debate. But the distro embargo period, where sufficiently large redistributors who are neither the upstream maintainer nor the party running the code have some extended time to prepare updates, doesn't seem like it's inherently defensible under responsible-disclosure arguments; it's just an artifact of the fact that preparing updates takes time.

Get rid of the distro embargo period

Posted Mar 30, 2023 7:43 UTC (Thu) by epa (subscriber, #39769) [Link] (3 responses)

Backporting the fix… for critical software (and assuming curl is such) it seems like good practice to stick closely to an upstream version so if fixes are released you don’t need to backport. That does rely on the upstream’s willingness to maintain one or two ‘stable’ or ‘long term support’ releases.

Get rid of the distro embargo period

Posted Mar 30, 2023 9:29 UTC (Thu) by bagder (guest, #38414) [Link] (2 responses)

(curl author here)

The are countless distros and they all ship N number of parallel curl versions. We as an "upstream" project cannot possibly provide backports of fixes for an infinite number of vulnerable packages. In the curl project we actually don't even try - we provide patches for the latest version, anyone who wants to patch older releases are free to backport them.

In addition: looking at packages still provided by a specific distro a few years down the line, they have literally *hundreds* of custom patches applied to their code. Backports of security fixes, but also their own selection of bugfixes etc they deem important enough for their users.

That fact also makes it hard for a project to provide patches for such distros' old versions, as they are already shipping a frankenproject that over time deviate substantially from the upstream.

Get rid of the distro embargo period

Posted Mar 30, 2023 9:37 UTC (Thu) by epa (subscriber, #39769) [Link] (1 responses)

Thanks, I think we agree. The distros should make an effort to track the latest release, or if they are going to have an old one, they should all agree on the *same* old one and then there would be some hope of a co-ordinated effort to backport security fixes.

Get rid of the distro embargo period

Posted Mar 31, 2023 1:53 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

The problem is, they all want different old ones. Ubuntu wants one that's no more than six months old. Debian wants one that's a couple of years old. RHEL probably wants one from the mid 2000's or thereabouts (as well as several newer-but-still-old ones for more recent releases of the distro). You can't just have one version that makes all of those distros happy.

Stenberg: Pre-notification dilemmas

Posted Mar 30, 2023 9:41 UTC (Thu) by vegard (subscriber, #52330) [Link] (6 responses)

I find this blog post somewhat disingenuous.

At the same time as sending distros the notification about upcoming CVEs, Stenberg also disclosed on social media that they had 6 pending CVEs (https://mastodon.social/@bagder/110015733422233816) and if you went on the Curl GitHub page at that moment, you would see those patches sitting right there at the top of the git changelog.

The problem with this, as Solar Designer (who runs the distros list) pointed out, is that if Curl notifies the distros list, the distros now face a dilemma: do distros ship the patches (which are public and are easily discovered, and have been "somewhat disclosed" on social media already), or are they bound to secrecy (thus leaving users unpatched) until upstream make an official release?

Solar approached this whole thing in an open and friendly way, while Stenberg's responses were curt if not downright hostile. Stenberg's blog post is inflammatory -- many of the characterizations there are misleading (if not false) and again -- it's sad to see such hostility when both sides really want what's best for the users.

I think it's clear that we need a better protocol for shipping security fixes to end users, but it needs to work for both maintainers and distributors. If there are conflicting requirements, we should try to find middle ground. Maybe some concessions will need to be made.

Stenberg: Pre-notification dilemmas

Posted Mar 30, 2023 10:02 UTC (Thu) by Wol (subscriber, #4433) [Link]

> The problem with this, as Solar Designer (who runs the distros list) pointed out, is that if Curl notifies the distros list, the distros now face a dilemma: do distros ship the patches (which are public and are easily discovered, and have been "somewhat disclosed" on social media already), or are they bound to secrecy (thus leaving users unpatched) until upstream make an official release?

Regardless of good or bad personal interactions, I guess from reading all this, the distros should ship the patches.

I get the impression that what Stenberg wants is for the distros to be patched before the official announcement, which is sensible. So he's really saying "please act but don't talk". It won't hinder bad actors who are monitoring curl, but it will prevent a 0-day window for bad actors monitoring announcements.

"Deploy before you announce" just seems so sensible, but don't confuse "disclose" with "announce". Because, as Stenberg points out, you HAVE to disclose before you deploy, in order to make sure that the fix actually works.

Cheers,
Wol

Stenberg: Pre-notification dilemmas

Posted Mar 30, 2023 10:03 UTC (Thu) by vegard (subscriber, #52330) [Link]

Just a quick follow-up.

Case in point: the distros list did not force the curl project to properly disclose the issues in the end; the list was flexible in the face of a reporter who did not read the list rules before posting. Solar invited to a discussion and explained in friendly terms exactly why the list policy is the way it is. Why not have that discussion, on equal terms, in an open forum, instead of this one-sided blog post?

I'm sad because it's the users who lose in the end, every single time.

Stenberg: Pre-notification dilemmas

Posted Mar 30, 2023 20:14 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (2 responses)

I think this is a rather significant difference of opinion:

* Stenberg sees the list as a courtesy to distros. He posts there because he thinks they find it useful. When he's accused of "breaking the rules," his reaction is to take his ball and go home ("You don't want notifications on my terms? Fine, I won't send them.").
* The distros see the list as a collaborative project. They see Stenberg "breaking the rules" and ask him to please follow them.

I don't think either party is being unreasonable, they just have different expectations about what the list is and how it works. Perhaps there should be a list for people like Stenberg to drop courtesy notifications without any rules, aside from an intentionally underspecified "we will make a reasonable effort not to disclose anything that isn't already public, but no promises." If you don't provide that, then some upstreams may decide to stop giving such notifications altogether (or, they may have already made that decision long ago, and never communicated it to anyone).

Stenberg: Pre-notification dilemmas

Posted Apr 4, 2023 23:54 UTC (Tue) by geofft (subscriber, #59789) [Link] (1 responses)

It seems like it is clearly a courtesy to distros. Nothing in the rules prevents you from disclosing vulnerabilities to other people with their own embargo policy, which has to be the case - for infrastructure libraries (like cURL), you probably want to notify OSes not on the list like Android and Windows and macOS, and you might want to notify major software projects like browsers if they link their own copy of your code. And those projects will have their own disclosure policies. The rules make it clear that the distros list will disclose after no more than 14 days, so you may want to delay contacting them to maintain a global embargo. But if some other project had, say, a maximum 3-day embargo period, it would still make sense to contact both that project and distros list at the same time.

So, by extrapolation, why isn't it also fine to cross-post to the distros list and a virtual project with a zero-day embargo period - i.e., social media? The rules state that the list poster is taking on responsibility for certain things, but as I read them, all those things are about being sure to disclose at some point, not about preventing disclosure. The list recipients have the burden of maintaining the embargo, but the poster does not, as I read it.

Yes, this raises the question of what people on the list should do if they're nominally required to keep an embargo for information that is independently public or can be derived from public sources. But that could happen in many other circumstances too, like the 3-day-embargo-project scenario or even independent discovery, and it isn't clearly stated in the rules what happens them - a strict reading implies that the list members are bound to an embargo for up to 14 days even if 100% of the information just got independently posted to oss-security! It seems to me that if the information can clearly be derived from public sources, without using the embargoed post as a hint for where to look, the list members should no longer feel bound to the embargo. (So in this specific case, the distros should have been able to say "Thanks for the heads up, the bug is super obvious from the commit you just pushed publicly, we're going to cherry-pick them in public too.") But that should be clearly stated in the rules.

Perhaps the answer is simply that courtesy posts should just go to oss-security and forget about linux-distros? I am a little confused not to see mentions of that approach either in this blog post or the previous kernel discussions. Is the problem that oss-security rejects vague "This patch fixes CVE-xxxx whose description isn't public yet, please cherry-pick and I'll tell you what's going on next week" posts? (I don't see anything about this in oss-security's rules....)

Stenberg: Pre-notification dilemmas

Posted Apr 5, 2023 3:55 UTC (Wed) by vegard (subscriber, #52330) [Link]

Sending anything to distros is _always_ a courtesy. That part is very clear. Nobody is ever forced to send anything to the distros list. Some projects do, some don't; some security researchers do and others don't. The advance notification is extremely useful to the people working on a distribution, it allows for better testing, release planning, and prevents people from having to work unexpected extra hours. But let's be clear: it's both a courtesy to the people making a distribution as well as to the users of those distributions.

In a way, yes, it's perfectly fine to cross-post both to the distros list and to social media. What doesn't make sense is to publicly announce a security vulnerability and then at the same time expect the distros to hold off on shipping the fix.

The current distros list policy is there to set clear expectations for when an issue is considered public and can be disclosed (and when patches can be released); it establishes a protocol between reporter and receivers. In a way, the policy is there to prevent exactly this kind of issue where some parties are allowed to publish while others are not. (Sure, there are other concerns too, like ensuring that all issues will be disclosed within a given time frame -- this is a guarantee to non-members of the list and the community at large that distros will not keep things unfixed for too long.)

In the end, if you don't agree with the stated policy, then yes, don't send anything to the list. Just keep in mind that it's probably worse for everybody, and most notably the end users.

Stenberg: Pre-notification dilemmas

Posted Mar 30, 2023 21:53 UTC (Thu) by flussence (guest, #85566) [Link]

> Solar approached this whole thing in an open and friendly way, while Stenberg's responses were curt if not downright hostile.

I'll believe that when I see it. What definition of “open” is being used here?