Fedora distributes new keys

The Fedora project is back on track after its recent "infrastructure issues" with new package signing keys as well as packages and updates signed with the new keys. Fedora users should be able to pick up the new key and update their systems now, with a minimum of hassle—just verifying and accepting the new key. But, no further information has been released about exactly what went wrong, leading to more speculation and some worry in the Fedora community.

When a user gets a package from their distribution—or, more likely, a mirror of their distribution repository—they need to have some way to determine that it is a valid package. Distributors sign packages using a private key; that signature can then be verified by using the distribution's public key. If the private key gets compromised somehow, malicious packages could be created that would be indistinguishable from the real versions. This is why private signing keys must be well guarded, usually by isolating them on separate machines and encrypting them with a password.

According to one of the announcements about the problem, there is no evidence that the passphrase used to guard the Fedora private signing key has been compromised, though the clear implication is that the encrypted key file may have been captured. Out of an abundance of caution—and perhaps the concern that the passphrase might be guessed or brute-forced—the project decided to generate new keys. Along with new keys come various headaches: re-signing all of the packages as well as getting the keys installed on user's machines.

Getting the keys to users is largely a matter of getting the new fedora-release package—along with PackageKit and friends for GUI-enabled updates—installed. That package contains the new key and repository name (updates-newkey). Of necessity, those updates are the last that will be signed with the old key, so they will install on existing Fedora systems. Once that package makes its way out to the mirrors, users can install it so that they can proceed with any needed updates using the new key.

A yum clean metadata was helpful at the time of this writing to accelerate the process; depending on which mirror is being used and when it gets updated, that may not be needed. After fedora-release is installed, yum list updates gives a long list of updates available, all signed with the new key. All a user needs to do is verify the key and add it to the RPM key database. Verifying the key is a manual step as a user must check its fingerprint against that published on the web site. The method described requires importing the key into gpg, then doing gpg --fingerprint fedora@fedoraproject.org to see the key fingerprint; this is clearly something that could be made easier.

As part of phase one of the re-signing, Fedora has re-signed all Fedora 8 and 9 package updates. Phase two is ongoing, re-signing each package that is distributed as part of the original release of Fedora 8 and 9. Fedora 10 already has a new signing key as well. From the perspective of a possible compromise of the signing keys, things are well on their way back to normal. But there is still the nagging issue of how this all came about to begin with.

Several different questions about the intrusion were directed at the Fedora board from community members in their IRC meeting on September 9. Unfortunately, there was no new information forthcoming, nor was there any indication of when that information might be available. According to the board member Tom "spot" Callaway, information will be released "when we're told that we can by the parties running the investigation, not a second before, and not a second later."

Red Hat is clearly holding all information about the intrusion as a closely guarded secret—whether that is at the behest of law enforcement or just lawyers is unclear. While there was no timeline given, the clear sense that one got from the meeting is that it might be weeks or months before clearance will be granted to even confirm that they know how the intrusion occurred. In addition, the Fedora board has not been officially briefed on the incident; some members have knowledge because of their Red Hat responsibilities, but the rest are in the dark. If one needed a reminder that Fedora is not an independent distribution, but instead is subject to the whims of Red Hat, this is a clear demonstration.

The justification for secrecy is that Red Hat is a publicly traded company so intrusions into its systems need to be treated differently. Some board members believe that had there not been an intrusion into the servers that handle packages for Red Hat Enterprise Linux—that is if it had only been Fedora servers that were affected—the incident would have been handled much more transparently. Overall, the board is clearly unhappy about the situation but, perhaps because they are almost all Red Hat employees, don't see that there is much that can be done about it. That too should serve as a reminder.

It should be noted that Debian has had several server compromises over the years (for example, 1 and 2), which is, perhaps, a poor record of server security, but it is an excellent example of transparency. Debian is rather well known for its independence, which is part of what allows it to be so open. Those incidents do serve as examples; perhaps they are not an exact fit for the current Fedora/RHEL intrusion but that remains to be seen.

It may very well be that Red Hat is between a rock and a hard place here. As a friend to free software, Red Hat is unparalleled, but once in a while it shows that it is foremost a corporation with responsibilities to its shareholders. When those responsibilities conflict with the transparency we have come to expect from free software projects—especially with regard to security issues—that transparency must be set aside. One can argue that Red Hat is being overly protective of the details—confirmation that they either know or do not know how the intrusion occurred for example—but that argument really can't be made until all the facts are known. For that we must wait for the process to run its course.