Rutkowska: Security challenges for the Qubes build process

[Posted May 31, 2016 by corbet]

Qubes founder Joanna Rutkowska writes about how Qubes works to avoid building compromised software into its distribution. "Ultimately, we would like to introduce a multiple-signature scheme, in which several developers (from different countries, social circles, etc.) can sign Qubes-produced binaries and ISOs. Then, an adversary would have to compromise all the build locations in order to get backdoored versions signed. For this to happen, we need to make the build process deterministic (i.e. reproducible). Yet, this task still seems to be years ahead of us."

Rutkowska: Security challenges for the Qubes build process

Posted May 31, 2016 15:26 UTC (Tue) by amacater (subscriber, #790) [Link] (16 responses)

Rule 1. Don't use an RPM based distribution :)

Rule 2. Talk to the Debian folk doing reproducible builds and the other distributions working with them

Rule 3. Minimise what you build - get a minimal proof of concept for a small target then build from there?

Rutkowska: Security challenges for the Qubes build process

Posted May 31, 2016 17:34 UTC (Tue) by kbrantley (guest, #70638) [Link] (15 responses)

What does RPM have to do with it?

Rutkowska: Security challenges for the Qubes build process

Posted May 31, 2016 19:57 UTC (Tue) by ms (subscriber, #41272) [Link] (4 responses)

I don't know what the OP meant, but I have tried to assble rpms deterministically and couldn't make it work. Same inputs, even using libfaketime and I couldn't get a deterministic output. With debs it does work - unpack the ar, reset the timestamps on the three inner archives and add them back in in the same order and the whole thing can be made deterministic. If anyone's managed it with rpms, I'd love to know how. I gave up. Couldn't work out what was going on in the rpm header and footer. Binary diff's only revealed so much...

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 0:10 UTC (Wed) by nevyn (guest, #33129) [Link]

This will give you a text repr.:

https://james.fedorapeople.org/yum/dump_rpm_info.py

...if you want to diff., in theory it should be just as easy as making debian completely reproducible ... except nobody really cares on the rpm side (well, they care about slightly different things).

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 21:43 UTC (Wed) by smoogen (subscriber, #97) [Link] (2 responses)

I think the problem with 'reproducible' is that we need to have a common definition of what reproducible means. For some places, if you have to reset the timestamps or other clock issues you are manipulating the data to make it reproducible in the same way that if you kept adding enough NULLS to the end of a file it will eventually have the same checksum. In other definitions it does not mean that. It also means you have to define what flags and controls are in place to be exactly the same. What was the clock of the box, what was the random number controller set to (so that anything that the software during compile time wants to be random isn't.) what were the file inodes used etc etc. Each of those may be listed by some group as being 'deterministic' because it is actually important for their use case.

However then you end up with arguments where people are 'arguing past each other' because they don't know that their definitions are not congruent. So are we all talking about the same 'deterministic'?

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 23:00 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

As far as the clock thing goes, I think that the usual solution is to strip dates out of things like Doxygen output, avoid __DATE__ (which should probably instead be provided via `git archive` through substitutions), etc.

There was also mention on the CMake list recently about sorting the output of file(GLOB) so that files are always in the same order (though file(GLOB) is an awful way to list source files anyways).

Rutkowska: Security challenges for the Qubes build process

Posted Jun 2, 2016 8:22 UTC (Thu) by ms (subscriber, #41272) [Link]

These are all excellent points.

For my purposes, what I would like would be for the build to be reproducible. Compilation is a pure referentially transparent operation - heck, *everything* is, provided you know all the inputs and what to set them to. For people compiling/packaging software, if they have the exact same inputs (sources etc) then they should get the bitwise identical output. So in fact most the challenge here is just figuring out what the inputs are to each operation.

When I package up my software for release, if I need to mess with timestamps then yes, I set them all to the timestamp of the commit that symbolises the release. I also use Nix for building my releases and publish the nix recipes so if anyone else wants to verify the binary result of building hasn't been manually fiddled, they can do so.

Rutkowska: Security challenges for the Qubes build process

Posted May 31, 2016 19:59 UTC (Tue) by amacater (subscriber, #790) [Link] (9 responses)

personal bias

RPM - slightly less obvious build process, slightly less sorted dependency management?

[Certainly true for Red Hat Enterprise Linux and CentOS in my experience - and reliance on third party repositories with no obvious developer traceability]

Binary database rather than human readable for installed packages, fork of RPM package manager itself (rpm5) ...

/personal bias

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 0:17 UTC (Wed) by nevyn (guest, #33129) [Link]

> fork of RPM package manager itself (rpm5) ...

So if someone says they are the new upstream and posts dpkg-2.0, and nobody uses it (or maybe one niche distro. uses it for a bit before realizing their mistake and reverting) ... then suddenly debian and everyone else has to say they are using a fork of dpkg?

Cool story.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 13:32 UTC (Wed) by johannbg (guest, #65743) [Link] (7 responses)

The fundamental problem here is fragmentation of package management in linux distributions.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 14:29 UTC (Wed) by Wol (subscriber, #4433) [Link] (6 responses)

Actually it's worse. It's the fragmentation of what is in a package. I know SuSE and RedHat are better now, but rpm's have NEVER been compatible across rpm-based distros, because SuSE is not a RedHat derivative. It was originally a Slackware derivative, then some other little-known distro, then it chose rpm as the package manager.

Cheers,
Wol

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 15:44 UTC (Wed) by johannbg (guest, #65743) [Link] (5 responses)

The fundamental problem here is that there is not a single unified standardized package management system used by *all* linux distributions not what's being shipped in those package which is determent either by application or application stacks providers or by the distribution themselves.

The above is *one* of the fundamental reason why applications or application stacks aren't being provided et all on linux or just only for single linux distribution or set of distribution using the same package management system.

If an unified standardization around containerized application or application stack can be achieved, then hat particular problem might be solved but time will tell if people, distribution and vendors can agree upon single unified specification surrounding that.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 3, 2016 1:57 UTC (Fri) by NightMonkey (subscriber, #23051) [Link] (4 responses)

Could the problem, at a basic level, be that, in many ways, the package manager is *exactly* what differentiates one distribution from another?

Rutkowska: Security challenges for the Qubes build process

Posted Jun 3, 2016 2:47 UTC (Fri) by johannbg (guest, #65743) [Link] (2 responses)

I would say what fundamentally deviates community distributions is the philosophy behind the community that surrounds them, the rest is just needles technology obscureness to fill their own need to be able to show the world they are different. Different packagemanagment system is one such obscurement but not the only one.

Unless there will be a consolidation atleast on the core/baseOS level within the next 5 years or so the linux ecosystem will start suffering irreversible damage which will ultimately leads to it's destruction afaikt since there is a bigger problem that will need to be starting to be resolved somewhere around that time.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 3, 2016 8:00 UTC (Fri) by pabs (subscriber, #43278) [Link] (1 responses)

The number of package managers has exploded over the last 15 years. First the distro package managers has increased over time. Then every single programming language reinvented the distro package managers (my guess is to cope with Windows/MacOS). This problem isn't going to go away, it is only going to get worse. People love to reinvent the wheel.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 11, 2016 15:34 UTC (Sat) by jospoortvliet (guest, #33164) [Link]

Yup and it is a pita for isv's. You are right that the languages and their packaging formats are part of the problem - and so is the inwards looking nature of most distributions, all mostly ignoring what the others do. Ask debian ppl what they think of the open build service and 99% doesn't even know what it is. I bet no suse person has had a good look at rh's copr and so on.

But maybe I am wrong and they looked at the tech from others and decided their goals, seemingly nearly identical from the pov of users, we're so different there was no need or room for reuse and/or collaboration.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 3, 2016 16:04 UTC (Fri) by misc (subscriber, #73730) [Link]

Not only. The lifecycle are usually different, the various choices made on technical level ( like apparmor in ubuntu, selinux in fedora/centos, and not much in debian, the choice of some distribution to use musl or refuse systemd ), the community and the philosophy itself.

Granted, it should matter that much, gnome on most distro is always gnome, firefox is always firefox even if it has another name. If the majority of your software usage is external proprietary SaaS, it doesn't really matter what you use, but that's not the case of everybody.

Rutkowska: Security challenges for the Qubes build process

Posted May 31, 2016 23:09 UTC (Tue) by deepfire (guest, #26138) [Link] (8 responses)

I'm surprised why Nix / Nixpkgs / NixOS wasn't mentioned yet. It goes a long way towards validation, build determinism, and declarative transparency.

Plus you get the warm and fuzzy feeling of its foundation having been designed by people people with academic background.

Nix only does checksum verification, no signatures, but adding that should be relatively straightforward.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 0:13 UTC (Wed) by nevyn (guest, #33129) [Link] (7 responses)

> I'm surprised why Nix / Nixpkgs / NixOS wasn't mentioned yet.

People barely care as is, they certainly don't care enough that they want to redownload their entire distro. when glibc has another DNS security bug.

Also Nix doesn't do anything to validate the source => binary part of the problem.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 1, 2016 20:02 UTC (Wed) by nix (subscriber, #2304) [Link] (2 responses)

I believe glibc has a special case in Nix (no relation); a glibc change does not trigger a rebuild of all dependent packages. (However, frankly, it should run an ABI check with the conformance checker and only trigger a rebuild if that says there are compatibility problems :) I mean, how often does libX11 break ABI? Oh, never... but an xlib update *does* trigger a massive rebuild of all dependent packages.)

Rutkowska: Security challenges for the Qubes build process

Posted Jun 2, 2016 12:16 UTC (Thu) by ms (subscriber, #41272) [Link] (1 responses)

Yeah, I'm surprised they've not gone further - you could imagine a way of figuring out the signature of the ABI: take all the public functions by name, with some representation of their types and hash all that lot together. Sure, it wouldn't stop semantic changes from breaking everything but nor does anything currently. You could even do it per function if you want to be really fancy to ensure that just adding functions wouldn't trigger a rebuild. I have no idea what the linker requirements are though, so I don't know if that's even feasible.

Rutkowska: Security challenges for the Qubes build process

Posted Jun 8, 2016 12:16 UTC (Wed) by nix (subscriber, #2304) [Link]

That's what the ABI compatibility checker does, more or less. (It does need debugging info to do it, though, and doesn't always work, just most of the time.)

Rutkowska: Security challenges for the Qubes build process

Posted Jun 4, 2016 22:26 UTC (Sat) by deepfire (guest, #26138) [Link] (1 responses)

> People barely care as is, they certainly don't care enough that they want to redownload their entire distro. when glibc has another DNS security bug.

Thankfully, a solution to this has been in the works for quite a while:

http://thread.gmane.org/gmane.linux.distributions.nixos/1...
https://github.com/NixOS/nixpkgs/pull/10851

> Also Nix doesn't do anything to validate the source => binary part of the problem.

Indeed, build signing is yet to be added. Or you have something other than signing in mind?

Rutkowska: Security challenges for the Qubes build process

Posted Jun 11, 2016 22:20 UTC (Sat) by nevyn (guest, #33129) [Link]

> Indeed, build signing is yet to be added

I'm not sure what that is, I hope that doesn't mean nix doesn't sign it's packages yet? The discussion was about reproducible builds (or something similar) which allows you to verify source => build.

Eg. https://wiki.debian.org/ReproducibleBuilds

Rutkowska: Security challenges for the Qubes build process

Posted Jun 9, 2016 12:30 UTC (Thu) by civodul (guest, #58311) [Link] (1 responses)

FWIW GNU Guix, which, like Nix, implements the functional package management paradigm, has a solution to overcome the problem of full rebuilds when a security update comes out:

https://www.gnu.org/software/guix/manual/html_node/Securi...
https://savannah.gnu.org/forum/forum.php?forum_id=8470

In addition, we provide infrastructure to "challenge" a binary provide, that is, to check whether the binaries it offers for download match those produced locally or provided by another server:

https://www.gnu.org/software/guix/manual/html_node/Invoki...
https://savannah.gnu.org/forum/forum.php?forum_id=8407

Rutkowska: Security challenges for the Qubes build process

Posted Jun 11, 2016 22:13 UTC (Sat) by nevyn (guest, #33129) [Link]

This seems like a sane workaround, hopefully guix will get more mindshare.