Where does the RHEL 7 source code live?
When the much-anticipated Red Hat Enterprise Linux (RHEL) 7 release was made on June 10, it started the process for various RHEL-rebuild distributions to get their new releases out. Previously, the starting point had been source RPM (SRPM) files that Red Hat used to release. Things have changed a bit this time, which caused some initial consternation among some rebuild developers, but it eventually turned out that it would not to be too difficult to get what is needed from the new Git-based repository.
The expected location for the SRPMS directory for RHEL 7 just has a README file that points to git.centos.org/project/rpms and the Red Hat Customer Portal as locations for the source code. The latter is, as its name would imply, only available to Red Hat customers. That means that all of the other rebuilds need to pick up the source code from a CentOS repository, which seems a little strange—at least at first.
One of the big changes between the release of RHEL 6 and that of RHEL 7 is that CentOS, arguably the leading RHEL-rebuild distribution, joined Red Hat. That would seem to put it in the pole position for creating a RHEL 7 rebuild, but CentOS won't have any real advantages over the others. It will be using exactly the same RHEL code as all of the other distributions. Beyond that, there is a firewall between RHEL and CentOS developers. In addition, CentOS is developing and building CentOS 7 fully in the open, so anyone interested can follow along in the mailing list and Git repositories.
The README led to queries on the centos-devel mailing list (from Morten Stevens and Connie Sieh) wondering about the change in format and location for the RHEL 7 source code. The biggest concern was how to get unmodified source code that corresponds to the RHEL 7 packages and how to verify that the code had not been modified along the way. Previously, the SRPMs that were signed with Red Hat's key would suffice, but those are only available to customers through the portal now—though they may eventually "leak" since most simply contain free software.
The terse message in the README file seemingly did not go far enough to alleviate concerns about exactly what source was stored in the Git repositories. In addition to the mailing list threads, a bug was filed over the lack of SRPMs, but it was quickly closed as NOTABUG. The comment on the closing did, however, seem to clear up any confusion about the origin of the code in the CentOS Git repository—Red Hat put the source code from RHEL 7 there as a starting point for CentOS and anyone else.
Getting Red-Hat-signed SRPMs is still possible for customers, but for non-customers looking for signed SRPMs, CentOS will be providing its SRPMs once it completes the CentOS 7 development process. Those will only minimally differ from RHEL 7, mostly just removing the Red Hat branding, so they make for a better starting point for a rebuild distribution. It is certainly different than what was available before, but not necessarily worse.
It is also possible to generate SRPMs that correspond to the original RHEL 7 code from the Git repository, though those SRPMs won't be signed. There are a number of scripts hosted in the centos-git-common repository that will help perform various tasks using the Git repositories. For example, pointing into_srpm.sh at a commit or branch of interest will use the .spec file found there to create an SRPM. One can also use show_possible_srpms.sh to list the SRPMs that can be built from a particular repository; the -r option will restrict the list to those that come from the Red Hat RHEL 7 commits. There is also information on the CentOS wiki on how to work with the myriad repositories at Git.CentOS.Org.
There are concerns expressed in the bug and mailing list threads about
ensuring the authenticity of the code retrieved, but Git, coupled with
SSL/TLS, should be enough to alleviate that concern.
Using the tools to recreate the SRPMs from the Red Hat commits to the
repository, all of which is done over HTTPS (i.e. SSL/TLS), should be safe.
Rebuilding
the SRPMs, rather than just downloading them, may make it a little more
difficult, but it does not fundamentally change what the RHEL rebuilds can
get. And having the code available in Git provides an API to do things
that would not
have been possible with the simple "directory full of SRPMs" approach used
previously. As Chris St. Pierre put it: "You no longer have to diff
the ftp 'ls' output -- there are actual machine-consumable feeds and APIs
to use.
"
There is clearly some trepidation among the non-CentOS consumers of RHEL source code, as evidenced by the thread and bug comments. But the biggest issue, in the end, seems to be more confusion than anything else. While Red Hat did mention this change in a FAQ about the CentOS merger, it probably could have done a lot more to publicize—and explain—the change. In the final analysis, it may seem a little odd to get RHEL source code from a CentOS repository, but it shouldn't pose any real barrier to anyone getting their work done.
Posted Jul 3, 2014 19:01 UTC (Thu)
by jcpunk (subscriber, #95796)
[Link] (10 responses)
“There are concerns expressed in the bug and mailing list threads about ensuring the authenticity of the code retrieved, but Git, coupled with SSL/TLS, should be enough to alleviate that concern. Using the tools to recreate the SRPMs from the Red Hat commits to the repository, all of which is done over HTTPS (i.e. SSL/TLS), should be safe.”
The use of Git and SSL/TLS sets a very high barrier for in-flight tampering. This is not the point of discussion.
I suggest you can safely download what has been published. The question I ask is, who wrote the published code?
The problem, as I see it, is once you've downloaded the git repository you've no assurance it hasn't been tampered with. This is different than the previous point. Allow me to explain why.
The show_possible_srpms.sh command uses the commit author to determine if this is a RHEL commit. Any user can set their git commit author to “CentOS Buildsys <bugs@centos.org>”. I strongly encourage each of you try this challenge: Can you create a commit that show_possible_srpms.sh reports as coming from Red Hat? I do mean this, please try this exercise.
The answer is, of course, yes. But it is in your local repository and not git.centos.org.
This leads to the next question, could this commit be published on git.centos.org? If you break in, yes you could publish this.
If you published your false commit on git.centos.org, would it be built and distributed? I would argue yes. How would the tools determine your false commit should not be built? If the SHA sums match, the author is correct, and the content was pulled over SSL/TLS is it safe?
Would the community eventually find the false commit? I continue to answer, yes. But, would it be found before being automatically built by the CentOS build system, pass the automated QA, Signed by the CentOS key, and published?
I'm less sure here.
If the answer is, “We don't sign/publish automatically”, then we must ask a few questions: (1) should we feel comfortable with the human involvement here? People make mistakes. (2) how do they determine if a package should not be signed/published? Keep in mind that CESA-2012:0451 was published before RHSA-2012:0451. How did they know this was safe to release before Red Hat said so? They used GPG checks, which don't apply to git.centos.org today. (3) does it seem like a good idea that a false build can get all the way through the build system without being caught? How close to published should a false commit get before a human stops it?
What if it gets published and someone installs it, do they get the fixed package? There are STILL hosts with heartbleed on the internet. There are STILL hosts with massively vulnerable versions of openjdk. This cuts both ways, if people didn't apply security patches they why would I expect they would get this bad package? I can't answer that, I can only provide my horror at the possibility of this scenario.
Is there a solution? Of course – PGP/GPG signing can still be done.
The source on ftp.redhat.com is signed. Anyone building that code can check that the source rpm has a valid signature before building it. If the commits from Red Hat were signed, then this entire exercise would be moot. But they are not, and with that the trust model breaks down.
Yes with Git and SSL/TLS all the files are transported from end to end in tact, but to make that SSL connection you had to trust the certificate's issuing authority and your browser automatically verified the remote host was authorized to use it. The same cannot be said of the content within the repository. The Transport Layer is Secure, the same cannot be said for the content being transported. Would we consider SSL/TLS safe if it did not verify the certificate?
For further reading I recommend Mike Gerwitz's “A Git Horror Story”
Posted Jul 3, 2014 20:15 UTC (Thu)
by jake (editor, #205)
[Link]
http://mikegerwitz.com/papers/git-horror-story to save others the googling ... well worth reading ...
You make a good point. Essentially rebuilds (and others) are trusting RH/CentOS to secure the infrastructure and to be vigilant about what gets pulled into the repositories. Bugs could certainly allow nefarious commits as well.
I meant to mention signed commits, but it slipped through the cracks. It might be worth engaging RH/CentOS about doing so for these repositories.
jake
Posted Jul 3, 2014 22:27 UTC (Thu)
by dowdle (subscriber, #659)
[Link] (4 responses)
Posted Jul 4, 2014 0:21 UTC (Fri)
by jcpunk (subscriber, #95796)
[Link] (2 responses)
Are you in agreement with the various steps within the argument? Do you take exception with any aspects?
To further press the issue: since it is logically possible to steal the Red Hat signing key, should we therefore sign nothing since the SHA sums are posted on RHN?
The currentgit.centos.org repos take roughly that approach.
Posted Jul 4, 2014 3:41 UTC (Fri)
by mjg59 (subscriber, #23239)
[Link] (1 responses)
Posted Jul 4, 2014 21:13 UTC (Fri)
by dag- (guest, #30207)
[Link]
What's more, git.centos.org has (the same and) more attack vectors than the signing key/SRPMs used to have. So overall it is less secure as the previous way of working (which was the same for customers as it was for everyone else).
Posted Jul 10, 2014 20:29 UTC (Thu)
by boklm (guest, #34568)
[Link]
Securing a server on a private network that is only used to sign packages should be easier.
Posted Jul 4, 2014 10:03 UTC (Fri)
by salimma (subscriber, #34460)
[Link] (2 responses)
Posted Jul 5, 2014 18:19 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link]
Server-side is the only place to enforce it *and* trust it. If there are hooks to drop, gitolite should support it just fine.
Posted Jul 19, 2014 15:25 UTC (Sat)
by sitaram (guest, #5959)
[Link]
In this case, however: https://github.com/sitaramc/gitolite/blob/master/src/VREF...
:-)
Posted Jul 4, 2014 15:40 UTC (Fri)
by raven667 (subscriber, #5198)
[Link]
This seems to be the only actual argument that everything else derives from. If someone breaks into a developers laptop or the backend infrastructure then they can try and make commits to the repo. Other developers will be able to see these changes when they sync their own copies of the repos, so the changes cannot be hidden and they can be removed cleanly.
This kind of threat with this kind of impact exist in every project, there is nothing special about CentOS/RedHat that makes the impact greater than if someone tried to put bogus commits into the Linux kernel or some other major project. The strength of mitigation of risk is not in preventing this from happening, it's in the cleanliness of recovery if all the security controls which authenticates committers fail.
Posted Jul 3, 2014 22:35 UTC (Thu)
by dowdle (subscriber, #659)
[Link]
I remember when Red Hat went to the EL model and people were not happy with SRPMs as the way they released the sources. I recall an LWN article that mentioned some opinions that not releasing the source in a format that was easily consumable was potentially a violation of the GPL. So they have opened it up to the greater community in the most popular consumable format (git) and reserved SPRMs for paying customers. I'm not sure there is much to complain about there.
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Where does the RHEL 7 source code live?
Change hasn't affected SL much