LWN: Comments on "Fedora security response time"

Fedora security response time

zlynx — Sat, 09 May 2020 22:58:55 +0000

That makes sense. No developers, no bugs.

Fedora security response time

ghane — Sat, 09 May 2020 06:47:18 +0000

I am not sure why you smart developers insist on doing things the hard way.

Just develop Skynet. The T800 is designed to fix bugs *before* they occur.

:-)

--
Sanjeev

Fedora security response time

zlynx — Thu, 07 May 2020 00:25:32 +0000

Sure, why not?

Machine learning based unit, integration and fuzz testing to find the bugs, and a machine learning / AI process to fix the bugs.

Fedora security response time

Cyberax — Thu, 07 May 2020 00:20:15 +0000

Extrapolating... Subsecond response times by 2030?

Fedora security response time

KaiRo — Thu, 07 May 2020 00:05:46 +0000

Man, how the world has changed. In 2007, we were proud in the Mozilla project that we could ship security fixes in "10 f--ing days" (there is a story behind that phrase) after becoming aware of them. In 2020, we are talking about bringing security response times from a day or two down to hours in this case...

Fedora security response time

abo — Fri, 01 May 2020 07:31:15 +0000

Updates that break things also tend to cause people to disable them, which is a lot worse than a day's delay.

Fedora security response time

mcatanzaro — Thu, 30 Apr 2020 20:23:48 +0000

> I've yet to see a Linux exploit developed for a web engine vulnerability and deployed against users in the wild. Are you aware of any instance of this happening, ever?

I hope I don't come to regret this comment. :D It will happen eventually, but I haven't seen it yet.

Another consideration is that web engines are heavily sandboxed. We generally class everything more serious than a null pointer dereference as a remote code execution vulnerability, which is about as serious as it gets, but we also assume that you can't really do *too* much harm if you achieve code execution in the untrusted sandboxed process that has very limited filesystem access. Sandbox escapes are discovered on occasion, but they are rare and generally quite clever. Web engine CVEs are a never-ending stream, and as long as they're fixed in a reasonably timely manner, I think Linux distros will be all right.

For an example of not reasonably timely, look at QtWebKit.

Fedora security response time

highvoltage — Thu, 30 Apr 2020 16:27:30 +0000

It's been proven in multiple software projects over the years that rushing out updates lead to regular botched updates. This is an area of work that is indeed important to do fast, but above all to do properly.

Fedora security response time

mattdm — Thu, 30 Apr 2020 15:34:21 +0000

For some context on the original release engineering request: at that time, a number of us worked overnight to make sure we had a fix for Heartbleed (remember when that was the kind of security vulnerabilities our systems had? oh to simpler times), and then it turned out that it took another whole day just for that update to churn through the mechanical process, and something went wrong at just the wrong time making it even worse than expected. I'm quite confident that process improvements we've made since then make a repeat of that unlikely.

Fedora security response time

idealista — Thu, 30 Apr 2020 12:36:11 +0000

For that type of services, having a CDN nowadays it's close to a requirement rather than a feature.

Fedora security response time

mst@redhat.com — Thu, 30 Apr 2020 07:04:18 +0000

> the metadata hashes for the old packages need to invalidated quickly,
> throughout the entire update network (i.e. mirrors) "within an hour or less, preferably minutes";
> that would mean that the older, vulnerable packages would no longer be installable.
Looks like this will break my attempts to have a system cache and search/install packages quickly from there: any attempts to use cache will run into everything having been invalidated, unless I am prepared to hit network for updates every minute :(.

Fedora security response time

pabs — Thu, 30 Apr 2020 06:52:36 +0000

For the CDNs that Debian deals with (Fastly mostly for mirrors), they don't sync the entire mirror, they just dynamically populate their caches as clients fetch individual files. Debian gets gratis services from Fastly and the other external entities we use.

https://www.debian.org/partners/
https://wiki.debian.org/ExternalEntities

Fedora security response time

pabs — Thu, 30 Apr 2020 06:48:49 +0000

Debian has had to place a CDN frontend on that set of servers, as Linux kernel updates were overwhelming them.

Fedora security response time

AdamW — Thu, 30 Apr 2020 01:21:58 +0000

"Adam Williamson, who is part of the Fedora QA team, wondered about the "within hours" timeline."

Just to clarify this - I was being a bit wilfully gnomic, but my reply was intended to convey that the original mail simply stated "We need to ensure that security updates reach stable within hours of an upstream advisory", but never provided any justification for this assertion. It's not that I necessarily disagree, more a case of "please provide a rationale rather than simply asserting this".

Fedora security response time

Conan_Kudo — Thu, 30 Apr 2020 00:44:37 +0000

Push comes to shove, Fedora's tier-0 servers and tier-1 mirrors can serve content as close to immediately as possible. The MirrorManager system will allow total mirror network invalidation and force users to pull from those servers for that content.

However, it is not needed. An argument could be made that we haven't needed it even the couple of times we've used it last year and only needed it once at the beginning of the year before (Spectre/Meltdown was that year...).

Fedora security response time

ballombe — Wed, 29 Apr 2020 22:57:34 +0000

I note that Debian distribute security updates from a small set of servers separated from mirrors, so one never have to wait for the mirrors to pull.

Fedora security response time

smoogen — Wed, 29 Apr 2020 21:44:22 +0000

One of the problems not covered is that the Fedora mirroring system is a pull system where the downstream mirrors are in control of when they pull and what they pull. There are 4 tier-0 mirrors which we seed 12 tier-1 mirrors at various universities and public orgs like kernel.org. Those 12 feed all the level 2 mirrors. There is no CDN because those cost money in both pushing data into them and pulling data out of them. We have had a couple of offers for CDN data but use it for only limited uses as we found it took more than 24 hours to sync up the amount of data we normally change in a day to the CDN.

Finally Fedora does not push updates to clients. A client may check for updates regularly if they are running the standard desktop, or they may only do so when they can afford the cost on a metered line.

There are ways around this, but they all come with costs in time, release engineering changes, and tooling in client, server and user expectations. Those all need someone to drive the changes over the long term it will take to implement.