LWN: Comments on "Where does the RHEL 7 source code live?"

Where does the RHEL 7 source code live?

sitaram — Sat, 19 Jul 2014 15:25:53 +0000

I've started to say gitolite is more a tool-builder that just happens to come with a lot of tools pre-built, than a tool by itself.

In this case, however: https://github.com/sitaramc/gitolite/blob/master/src/VREF...

:-)

Where does the RHEL 7 source code live?

boklm — Thu, 10 Jul 2014 20:29:13 +0000

That is not the same. Securing a public facing server that host git repositories is difficult.

Securing a server on a private network that is only used to sign packages should be easier.

Where does the RHEL 7 source code live?

mathstuf — Sat, 05 Jul 2014 18:19:00 +0000

Meh, that pretty much disallows "git cherry-pick" from external patches though. Not to mention that "git commit -n" exists.

Server-side is the only place to enforce it *and* trust it. If there are hooks to drop, gitolite should support it just fine.

Where does the RHEL 7 source code live?

dag- — Fri, 04 Jul 2014 21:13:58 +0000

I would contend that for Red Hat the signing key is business critical, but the git.centos.org repository is not. So no doubt the key is well secured, and the process to sign packages is well protected as it would hurt their business and harm the trust customers have in them. git.centos.org not so much.

What's more, git.centos.org has (the same and) more attack vectors than the signing key/SRPMs used to have. So overall it is less secure as the previous way of working (which was the same for customers as it was for everyone else).

Where does the RHEL 7 source code live?

raven667 — Fri, 04 Jul 2014 15:40:11 +0000

> This leads to the next question, could this commit be published on git.centos.org? If you break in, yes you could publish this.

This seems to be the only actual argument that everything else derives from. If someone breaks into a developers laptop or the backend infrastructure then they can try and make commits to the repo. Other developers will be able to see these changes when they sync their own copies of the repos, so the changes cannot be hidden and they can be removed cleanly.

This kind of threat with this kind of impact exist in every project, there is nothing special about CentOS/RedHat that makes the impact greater than if someone tried to put bogus commits into the Linux kernel or some other major project. The strength of mitigation of risk is not in preventing this from happening, it's in the cleanliness of recovery if all the security controls which authenticates committers fail.

Where does the RHEL 7 source code live?

salimma — Fri, 04 Jul 2014 10:03:41 +0000

Pre-commit hooks that require commit authors to match their SSH key / HTTPS login would help. Gitblit allows for that; haven't looked deeply enough into Gitolite settings to see if there's a standard way of doing it there.

Where does the RHEL 7 source code live?

mjg59 — Fri, 04 Jul 2014 03:41:25 +0000

Why is it logically possible to steal the Red Hat key?

Where does the RHEL 7 source code live?

jcpunk — Fri, 04 Jul 2014 00:21:22 +0000

I'm afraid I disagree with your summary of my position.

Are you in agreement with the various steps within the argument? Do you take exception with any aspects?

To further press the issue: since it is logically possible to steal the Red Hat signing key, should we therefore sign nothing since the SHA sums are posted on RHN?

The currentgit.centos.org repos take roughly that approach.

Change hasn't affected SL much

dowdle — Thu, 03 Jul 2014 22:35:17 +0000

Just saw that Scientific Linux announced 7 Alpha this afternoon... so I don't think the change from SRPMs to git has slowed them down too much. SL has been working with the CentOS devs in advancing the tools and it is great to see the collaboration.

I remember when Red Hat went to the EL model and people were not happy with SRPMs as the way they released the sources. I recall an LWN article that mentioned some opinions that not releasing the source in a format that was easily consumable was potentially a violation of the GPL. So they have opened it up to the greater community in the most popular consumable format (git) and reserved SPRMs for paying customers. I'm not sure there is much to complain about there.

Where does the RHEL 7 source code live?

dowdle — Thu, 03 Jul 2014 22:27:52 +0000

Your contention is... what if someone bad gets root? That's always an issue with everything. What if someone gets root at Red Hat and could sign the packages?

Where does the RHEL 7 source code live?

jake — Thu, 03 Jul 2014 20:15:27 +0000

> For further reading I recommend Mike Gerwitz's “A Git Horror Story”

http://mikegerwitz.com/papers/git-horror-story to save others the googling ... well worth reading ...

You make a good point. Essentially rebuilds (and others) are trusting RH/CentOS to secure the infrastructure and to be vigilant about what gets pulled into the repositories. Bugs could certainly allow nefarious commits as well.

I meant to mention signed commits, but it slipped through the cracks. It might be worth engaging RH/CentOS about doing so for these repositories.

jake

Where does the RHEL 7 source code live?

jcpunk — Thu, 03 Jul 2014 19:01:46 +0000

This statement, to my mind, fundamentally misses the point:

“There are concerns expressed in the bug and mailing list threads about ensuring the authenticity of the code retrieved, but Git, coupled with SSL/TLS, should be enough to alleviate that concern. Using the tools to recreate the SRPMs from the Red Hat commits to the repository, all of which is done over HTTPS (i.e. SSL/TLS), should be safe.”

The use of Git and SSL/TLS sets a very high barrier for in-flight tampering. This is not the point of discussion.

I suggest you can safely download what has been published. The question I ask is, who wrote the published code?

The problem, as I see it, is once you've downloaded the git repository you've no assurance it hasn't been tampered with. This is different than the previous point. Allow me to explain why.

The show_possible_srpms.sh command uses the commit author to determine if this is a RHEL commit. Any user can set their git commit author to “CentOS Buildsys <bugs@centos.org>”. I strongly encourage each of you try this challenge: Can you create a commit that show_possible_srpms.sh reports as coming from Red Hat? I do mean this, please try this exercise.

The answer is, of course, yes. But it is in your local repository and not git.centos.org.

This leads to the next question, could this commit be published on git.centos.org? If you break in, yes you could publish this.

If you published your false commit on git.centos.org, would it be built and distributed? I would argue yes. How would the tools determine your false commit should not be built? If the SHA sums match, the author is correct, and the content was pulled over SSL/TLS is it safe?

Would the community eventually find the false commit? I continue to answer, yes. But, would it be found before being automatically built by the CentOS build system, pass the automated QA, Signed by the CentOS key, and published?

I'm less sure here.

If the answer is, “We don't sign/publish automatically”, then we must ask a few questions: (1) should we feel comfortable with the human involvement here? People make mistakes. (2) how do they determine if a package should not be signed/published? Keep in mind that CESA-2012:0451 was published before RHSA-2012:0451. How did they know this was safe to release before Red Hat said so? They used GPG checks, which don't apply to git.centos.org today. (3) does it seem like a good idea that a false build can get all the way through the build system without being caught? How close to published should a false commit get before a human stops it?

What if it gets published and someone installs it, do they get the fixed package? There are STILL hosts with heartbleed on the internet. There are STILL hosts with massively vulnerable versions of openjdk. This cuts both ways, if people didn't apply security patches they why would I expect they would get this bad package? I can't answer that, I can only provide my horror at the possibility of this scenario.

Is there a solution? Of course – PGP/GPG signing can still be done.

The source on ftp.redhat.com is signed. Anyone building that code can check that the source rpm has a valid signature before building it. If the commits from Red Hat were signed, then this entire exercise would be moot. But they are not, and with that the trust model breaks down.

Yes with Git and SSL/TLS all the files are transported from end to end in tact, but to make that SSL connection you had to trust the certificate's issuing authority and your browser automatically verified the remote host was authorized to use it. The same cannot be said of the content within the repository. The Transport Layer is Secure, the same cannot be said for the content being transported. Would we consider SSL/TLS safe if it did not verify the certificate?

For further reading I recommend Mike Gerwitz's “A Git Horror Story”