What happened with Fedora - and Red Hat too
From: | "Paul W. Frields" <stickster-AT-gmail.com> | |
To: | fedora-announce-list <fedora-announce-list-AT-redhat.com> | |
Subject: | Infrastructure report, 2008-08-22 UTC 1200 | |
Date: | Fri, 22 Aug 2008 08:00:02 -0400 | |
Message-ID: | <1219406402.21605.24.camel@victoria> |
Last week we discovered that some Fedora servers were illegally accessed. The intrusion into the servers was quickly discovered, and the servers were taken offline. Security specialists and administrators have been working since then to analyze the intrusion and the extent of the compromise as well as reinstall Fedora systems. We are using the requisite outages as an opportunity to do other upgrades for the sake of functionality as well as security. Work is ongoing, so please be patient. Anyone with pertinent information relating to this event is asked to contact fedora-legal@redhat.com. One of the compromised Fedora servers was a system used for signing Fedora packages. However, based on our efforts, we have high confidence that the intruder was not able to capture the passphrase used to secure the Fedora package signing key. Based on our review to date, the passphrase was not used during the time of the intrusion on the system and the passphrase is not stored on any of the Fedora servers. While there is no definitive evidence that the Fedora key has been compromised, because Fedora packages are distributed via multiple third-party mirrors and repositories, we have decided to convert to new Fedora signing keys. This may require affirmative steps from every Fedora system owner or administrator. We will widely and clearly communicate any such steps to help users when available. Among our other analyses, we have also done numerous checks of the Fedora package collection, and a significant amount of source verification as well, and have found no discrepancies that would indicate any loss of package integrity. These efforts have also not resulted in the discovery of additional security vulnerabilities in packages provided by Fedora. Our previous warnings against further package updates were based on an abundance of caution, out of respect for our users. This is also why we are proceeding with plans to change the Fedora package signing key. We have already started planning and implementing other additional safeguards for the future. At this time we are confident there is little risk to Fedora users who wish to install or upgrade signed Fedora packages. In connection with these events, Red Hat, Inc. detected an intrusion of certain of its computer systems and has issued a communication to Red Hat Enterprise Linux users which can be found at http://rhn.redhat.com/errata/RHSA-2008-0855.html. This communication states in part, "Last week Red Hat detected an intrusion on certain of its computer systems and took immediate action. While the investigation into the intrusion is on-going, our initial focus was to review and test the distribution channel we use with our customers, Red Hat Network (RHN) and its associated security measures. Based on these efforts, we remain highly confident that our systems and processes prevented the intrusion from compromising RHN or the content distributed via RHN and accordingly believe that customers who keep their systems updated using Red Hat Network are not at risk. We are issuing this alert primarily for those who may obtain Red Hat binary packages via channels other than those of official Red Hat subscribers." It is important to note that the effects of the intrusion on Fedora and Red Hat are *not* the same. Accordingly, the Fedora package signing key is not connected to, and is different from, the one used to sign Red Hat Enterprise Linux packages. Furthermore, the Fedora package signing key is also not connected to, and is different from, the one used to sign community Extra Packages for Enterprise Linux (EPEL) packages. We will continue to keep the Fedora community notified of any updates. Thank you again for your patience. -- Paul W. Frields gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://paul.frields.org/ - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-announce-list mailing list fedora-announce-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-announce-list
Posted Aug 22, 2008 13:15 UTC (Fri)
by kragil (guest, #34373)
[Link] (4 responses)
Posted Aug 22, 2008 13:20 UTC (Fri)
by motk (guest, #51120)
[Link]
Posted Aug 22, 2008 13:21 UTC (Fri)
by Felix_the_Mac (guest, #32242)
[Link]
Posted Aug 22, 2008 13:38 UTC (Fri)
by AlexHudson (guest, #41828)
[Link] (1 responses)
Posted Aug 22, 2008 13:46 UTC (Fri)
by AlexHudson (guest, #41828)
[Link]
Posted Aug 22, 2008 13:20 UTC (Fri)
by cdamian (subscriber, #1271)
[Link] (4 responses)
Posted Aug 22, 2008 13:29 UTC (Fri)
by spot (guest, #15640)
[Link]
Posted Aug 22, 2008 13:36 UTC (Fri)
by tialaramex (subscriber, #21167)
[Link] (2 responses)
Posted Aug 25, 2008 1:37 UTC (Mon)
by duffy (guest, #31787)
[Link] (1 responses)
Posted Aug 26, 2008 12:01 UTC (Tue)
by tialaramex (subscriber, #21167)
[Link]
You're correct that this doesn't entitle you, a Red Hat subscriber with say a single machine subscription, to install the packages on a dozen machines and then get support on them all. But it's the subscription agreement you're violating, ie Red Hat are entitled to withdraw all support until you pay for the extra eleven machines, but they are not entitled to force you to stop using or distributing the software since that would exceed their rights as distributors under the GNU GPL.
This is an important reason why RHEL subscription is a better option for corporate entities than Microsoft's equivalent. The Microsoft subscriptions terminate your license to use on expiry of the subscription, but Red Hat don't (for most of the components of RHEL they can't even if they wanted to) do that. All that expires is the support contract. If you find another organisation willing to support your systems you can switch, with no penalty.
Posted Aug 22, 2008 13:50 UTC (Fri)
by Trou.fr (subscriber, #26289)
[Link] (14 responses)
Posted Aug 22, 2008 14:18 UTC (Fri)
by smoogen (subscriber, #97)
[Link] (12 responses)
Posted Aug 22, 2008 14:29 UTC (Fri)
by NAR (subscriber, #1313)
[Link] (6 responses)
Posted Aug 22, 2008 14:36 UTC (Fri)
by smoogen (subscriber, #97)
[Link] (5 responses)
Posted Aug 23, 2008 11:59 UTC (Sat)
by darwish07 (guest, #49520)
[Link] (4 responses)
Being in the middle east, it's said in the Installation license agreement that you can not use Fedora if you're in one of the US prohibited countries (Syria and others).
This implies that Fedora is considered part of Redhat Inc. under the US law.
I have not used Fedora since about 3 years, but I remember very well this kind of restriction of fedora usage those days and I don't know if they still exist or not.
Posted Aug 23, 2008 15:03 UTC (Sat)
by smoogen (subscriber, #97)
[Link] (3 responses)
Posted Aug 23, 2008 15:41 UTC (Sat)
by darwish07 (guest, #49520)
[Link] (2 responses)
One has to wonder if this applies to the source code or to the pre-compiled binaries only.
Posted Aug 23, 2008 15:45 UTC (Sat)
by smoogen (subscriber, #97)
[Link] (1 responses)
Posted Aug 23, 2008 22:00 UTC (Sat)
by darwish07 (guest, #49520)
[Link]
Posted Aug 22, 2008 14:49 UTC (Fri)
by jreiser (subscriber, #11027)
[Link] (1 responses)
Posted Aug 22, 2008 19:00 UTC (Fri)
by rahvin (guest, #16953)
[Link]
Posted Aug 23, 2008 4:21 UTC (Sat)
by sbergman27 (guest, #10767)
[Link] (2 responses)
Posted Aug 23, 2008 22:48 UTC (Sat)
by smoogen (subscriber, #97)
[Link] (1 responses)
[You seem to have a tendency to make everything into the worst case for Red Hat. Why is that?]
Posted Aug 24, 2008 18:40 UTC (Sun)
by sbergman27 (guest, #10767)
[Link]
You are misjudging my position. Red Hat is an excellent example of a sane and pragmatic company which has not become so jaded, self-absorbed, and preoccupied with short term trivialities, not to see the big picture. But they are not going to blurt out information that could land them in court, damage them, their shareholders, their sizable market cap, their reputation, etc. as long as it can be avoided. It's the difference between being a smart and pragmatic company that does a lot of good, and being a saint.
I don't paint Red Hat as being saintly, and some people choose to interpret that as negativity. I use Red Hat derived distros almost exclusively in my consulting work, though I now use Ubuntu on my own desktop. (RHEL and CentOS just 'click' for business use.) I do tend to be critical of Fedora. But that is because, in my opinion, there is more to be critical about when it comes to recent Fedora releases. (I begin to see this on servers that have to handle more than about 16 simultaneous desktops. Not so much on the smaller ones. But the problems are embarrassing.) Other than that the software versions are not cutting edge, it is extremely hard for me to come up with anything critical to say about RHEL and CentOS. If you look at my overall posting history regarding Red Hat, rather than focusing upon those pertaining to this 'infrastructure issue', I think you will see that I am actually a big Red Hat fan. Just without being a *fanboy*.
Posted Aug 23, 2008 2:42 UTC (Sat)
by sbergman27 (guest, #10767)
[Link]
Posted Aug 22, 2008 15:09 UTC (Fri)
by kripkenstein (guest, #43281)
[Link] (6 responses)
Posted Aug 22, 2008 15:15 UTC (Fri)
by JoeBuck (subscriber, #2330)
[Link] (4 responses)
Posted Aug 22, 2008 15:19 UTC (Fri)
by kripkenstein (guest, #43281)
[Link] (3 responses)
Posted Aug 22, 2008 16:37 UTC (Fri)
by elanthis (guest, #6227)
[Link]
Posted Aug 22, 2008 16:39 UTC (Fri)
by Ed_L. (guest, #24287)
[Link] (1 responses)
Posted Aug 22, 2008 17:23 UTC (Fri)
by drag (guest, #31333)
[Link]
Posted Aug 25, 2008 0:56 UTC (Mon)
by jamesh (guest, #1159)
[Link]
It isn't uncommon for large organisations to have shared authentication systems distributed via LDAP or similar, so having the same password on multiple boxes is not that surprising.
Also, it isn't uncommon for people to use the same SSH key pair to log into multiple servers, so that is another possible explanation for both RH and Fedora infrastructure being breached in a single attack.
Posted Aug 22, 2008 17:11 UTC (Fri)
by drag (guest, #31333)
[Link] (4 responses)
Posted Aug 22, 2008 18:53 UTC (Fri)
by dmarti (subscriber, #11625)
[Link]
Posted Aug 24, 2008 11:01 UTC (Sun)
by tialaramex (subscriber, #21167)
[Link] (2 responses)
Create user SSH private keys only on machines you have physical control over, or where local policy prohibits such caution, on a few machines where you have the most confidence of their integrity.
Use a passphrase to protect the keys. Use SSH agent software. The agent software uses the passphrase to unlock a key and hold it in memory temporarily. If the key files (stored on disk) are obtained by malicious third parties (e.g. by accessing backups, old drives mistakenly sold without being wiped) they are useless without the passphrase to decrypt them.
You can then use the key to log into machines without presenting a password at all, if the target machine is compromised you reveal only the relatively worthless public key identity of your user. An attacker witnessing hundreds of logins of this sort with full root access on the target machine learns nothing except the identities of frequent users. Contrast this to the situation with passwords, where he learns hundreds of valuable passwords.
Where convenience (or e.g. firewall policy) dictates that you SSH into one security pivot machine A and from there into others (B ... Z), you have two choices which reflect different security beliefs about the machines.
1. SSH to machine A, separate key on that machine for SSH to B ... Z. This keeps security of the set A ... Z separate from security for any other machines you have access to. An attacker in full control of A can authenticate (using the key stored on A) to other alphabetic machines at will but not elsewhere.
2. SSH agent tunnel. Local machine SSH agent is tunnelled to A for authenticating connections from A to other alphabetic machines. This limits power of attacker in full control of A to hijacking your authentication capability when you are logged into A. They can authenticate to any system for which your key validates, not just A ... Z, but the key and its passphrase remain inviolate on your local system.
In either case attackers who somehow have privileges on other alphabetic machines (B ... Z) learn nothing helpful and cannot leverage their privileges to attack other systems via SSH.
Posted Aug 26, 2008 8:00 UTC (Tue)
by drag (guest, #31333)
[Link] (1 responses)
Command line? Bah. Seahorse with Gnome-keyring and ssh-agent provide a more-or-less complete gui-based ssh solution for Linux desktops. Secure remote GUI, secure file transfer, secure terminal access.
One happy accident was that Seahorse used Gnutls so that it was not vulnerable to the OpenSSL issues with Debian. Dumb luck that one, at least on my part.
Certainly you can use command line, I do, but that's not even necessary anymore.
> Create user SSH private keys only on machines you have physical control over, or where local policy prohibits such caution, on a few machines where you have the most confidence of their integrity.
When I can help it I only have ssh keys on one machine. My local workstation. No point in anything else, usually.
I look at it like a spider. My reach goes out to my machines from my laptops. Without the key there is no way for anybody to get access to any of my systems. Only reach out and pull, never push from outside in.
From a personal standpoint ssh does everything.
Which, of course, is why the attacker was very interested in OpenSSH.
------------------------------------------------
As far as firewalls go...
As long as I can get out into the internet over HTTPS then the entire internet is available to me. Such is the magic of OpenVPN... It's virtually impossible to block if you want to allow even proxy-filtered http access. Stick a remote server up to run OpenVPN on port 443, tell it to use TCP, and a couple other options and your done. I use Shorewall for the firewall/router rules. Then I have my own private NAT firewall sitting out there on the internet, hundreds of miles from any network I use.
-----------------------------------
For private networks in private networks with filtering firewalls at the router, as long as they have remote access to ssh on a machine on the inside then it's possible to access any port on any server on that network using Ssh port-forwarding. No special rules required.
I've even done port reversal and opened up holes in networks so I can get remote access to a network that otherwise would of been completely blocked off. (Multiple levels of NAT be damned. If you can get out, they can get in. No question about it. Just need a man (aka Windows XP and Outlook) on the inside.)
So if you have a couple hops to get to a internal server then it's possible to tunnel ssh over ssh via forwarding and keep one session separate from the other, even if it's inside the other. Of course you may run into latency issues that cause such a setup to be to irritating to use in that manner.
That is:
(private network X)|firewall||internet||firewall|(private network Y)
Private Network X has Host A on it.
Now.. Lets assume that:
So you ssh from Host A to Host B. Then you setup port forwarding so port 8020 on Host A connects over ssh to port 22 on Private Server C.
Now you can connect directly from the Host A to Private Server C via port 8020 on A's localhost. That is "ssh -P8020 127.0.0.1 " connects directly to "privateServerC:22"
So you see, tunnel ssh over ssh to piggyback while preserving isolation. :) Even if Host B got hacked, they still would not have any access to Server C.
Works very well for any port on any system.
Posted Aug 26, 2008 12:24 UTC (Tue)
by tialaramex (subscriber, #21167)
[Link]
* Of course you can conjure up solutions involving an untrusted connection directly from B to C carrying just the file contents, then verifying a checksum via the SSH tunnel and so on. But it'd take a lot of paranoia to justify actually writing scripts for that rather than just agreeing it would work in principle.
Posted Aug 25, 2008 9:21 UTC (Mon)
by russell (guest, #10458)
[Link]
So why were they connected?
What happened with Fedora - and Red Hat too
??
So the ssh packages are/were vulnerable?
What happened with Fedora - and Red Hat too
Read for content, and Don't Panic.
SSH Packages Vulnerable?
The Red Hat statements say that the packages are being replaced as a precaution.
They do not state that the package contents have been altered in any fashion.
What happened with Fedora - and Red Hat too
I think they're saying that someone built some bad ssh packages and managed to get the system
to sign them before they got shut out. I don't think they're saying those packages got
distributed via Red Hat.
So, unless you're getting your RPMs from some dodgy place, it's not a problem. I guess the
main worry would be people cracking a system and installing those RPMs - they'd be difficult
to tell apart from the real thing without those check scripts Red Hat put up.
It sounds like the Fedora systems stood up to the attack pretty well, though.
What happened with Fedora - and Red Hat too
Heh, scratch that - they didn't actually say that the ssh rpms were bad, just that the
attacker had (re?)signed them.
What happened with Fedora - and Red Hat too
> We are issuing this alert primarily for
> those who may obtain Red Hat binary packages via channels other than
> those of official Red Hat subscribers.
What channels are those ?
I manage a few RHEL servers and they all are updated via RHN.
What happened with Fedora - and Red Hat too
"Channels" may have been a poor choice of terminology, given its usage in an RHN context.
Essentially, what this means is:
"We are issuing this alert primarily for those who obtain Red Hat binary packages from sources
other than Red Hat Network."
What happened with Fedora - and Red Hat too
I assume that some people run RHEL but don't want to pay Red Hat. Red Hat don't provide
binaries to such people, but nothing stops you (a Red Hat subscriber) from taking most RHEL
packages and uploading them to a public FTP server, this isn't even illegal, at least for the
vast majority of packages. People who download those packages have only the RPM signature to
rely on to prove that they're authentic rather than being trojan code you uploaded to exploit
them. Whereas you, as a subscriber have the additional protection of Red Hat's site security.
There's no good reason to do this sort of thing when CentOS exists (and when RHEL
subscriptions are comparatively cheap), but then there's no good reason to climb Everest and
plenty of people do that too.
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
While the security implications are potentially huge, I find the announcement is missing
important informations : how were they compromised ? how could it happen in the first place ?
Also, they didn't announce the compromise in one week, why ? They could have said they were
compromised and that they were invastigating it.
All in all I think there is too little information disclosed and I hope they will disclose
more details soon.
What happened with Fedora - and Red Hat too
It will probably take a bit of time. A breach of a publically traded company will be
investigated by various US Federal law enforcement and possibly other agencies(not sure if the
stock people have to look).. those agencies will put a legal clamp on releasing any
information until their investigation is done and they are sure that releasing the information
will not damage their investigation or possible future court case.
That basically means that Red Hat/Fedora can't say anything without getting an ok from their
legal who has to get an ok from the various investigating groups who will have to co-ordinate
with each other to give an ok.
[phew...]
I guess Fedora is not a publicly traded company, so information about the Fedora server compromise should not be under any legal clamps of this type...
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
Fedora is funded (systems, hardware, bandwidth, etc) by Red Hat, and is considered under
United States and most International law to be a part and parcle of Red Hat, Inc.
Yes, AFAIK Fedora is legally under Redhat Inc.
Yes, AFAIK Fedora is legally under Redhat Inc.
State of projects controlled by US-based organizations
State of projects controlled by US-based organizations
Very nice information indeed. Thank you!
State of projects controlled by US-based organizations
What happened with Fedora - and Red Hat too
"A breach of a publically traded company will be investigated ..." That opinion is
optimistic. Every year there are quite a few breaches of publicly traded companies that are
not detected or not reported. Not reporting a breach that has been detected is a violation of
applicable regulations, but there are dummies or crooks at that stage, too.
What happened with Fedora - and Red Hat too
It's assumed that given the information that RedHat picked up the phone and called the FBI
(who handles computer crime cases). Once that phone call is made Redhat will be "asked" to not
release any information that would interfere with the investigation.
Any breach where the Fed's don't get involved is a breech that wasn't reported to the Fed's.
Personally I wouldn't trust any company that doesn't call the cops when their business is
broken into, whether physical or informational.
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
[You seem to have a tendency to make everything into the worst case for Red Hat. Why is that?]
"""
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
Good to finally have some details. Those details, it turns out, are worrying.
My prime concern upon reading this summary is that there were two *separate* breaches, one for
the Fedora servers, and one for the Red Hat servers. It seems reasonable to presume that this
isn't a coincidence, for two such successful attacks to occur in close proximity. So, this is
a targeted effort against both Red Hat and Fedora, which by itself worries me.
In addition, this appears to be a serious breach in the sense that it isn't incidental to one
particular machine. That is, it seems most reasonable that the same attack method worked on
both targets. To me this implies that it wasn't a socially-engineered stealing of a password
to an account on one of the machines, as this wouldn't work for the other - unless a single
user has accounts on both, with the *same* password (not a good idea in general, but many of
us fall prey to this sort of thing...). If not, then the attackers have a method that works
against both the Fedora and Red Hat servers; it is possible that they have in their hands
details of a security vulnerability shared between these systems, which appears to me to imply
that it might be present in lots of systems around the world.
All of that said, it does appear Red Hat/Fedora are taking these intrusions seriously, and
that little actual damage has been done, that is the good news here as I see things.
Probably only one breach, since Red Hat provides Fedora's infrastructure.
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
Sure, could be - I really don't know. But I still tend to doubt that the same server is used
for both Fedora and Red Hat in their infrastructure for the functionality that was breached.
And given how much space was spent in the official summary about how the two breaches are
separate, I tend to presume we are talking about separate servers. But I hope that's
incorrect, and I presume we'll find out soon enough.
What happened with Fedora - and Red Hat too
It could have been a single Red Hat employee who let his key or passphrase get stolen. If he
had access to both Red Hat and Fedora systems, well... there you go.
I presume we will find out "soon enough" as well. However, given that in all likely hood RedHat/Fedora have the best security in the business, I would not presume we will find out before Red Hat quietly shares at least some details with its erstwhile competitors and sometime collaborators e.g. Debian, Gentoo, Mandriva, Canonical, Freespire, SuSE...
What happened with Fedora - and Red Hat too
What happened with Fedora - and Red Hat too
Usually the human is the weakest link in any security system.
It's very likely that the attacker gained access to a account using social engineering or by
bad ssh habits on the part of a Fedora developer.
What happened with Fedora - and Red Hat too
> a password to an account on one of the machines, as this wouldn't
> work for the other - unless a single user has accounts on both, with
> the *same* password (not a good idea in general, but many of us fall
> prey to this sort of thing...).
What happened with Fedora - and Red Hat too
A similar thing happened with Debian a few years ago.
If I remember correctly...
How it was detected was that the attacker tried to modify a file or two that was monitored by
Aide. Apparently it wasn't a very good attacker as he didn't notice they were running a HIDS
and didn't take steps to modify the Aide database or otherwise subvert it so he was detected
pretty early on.
How the security breach happened was that a developer had let his ssh password get exposed on
a insecure system.
-------------------------------------------------
As is a unfortunate habit among many Linux users and admin is that they ssh from one machine
to another willy nilly.
So you ssh from A to B and then from B to C and then from C to D and then use scp from D back
to A. Just over a day's work you often forget what machine your actually logged into and
functionally it doesn't matter a whole lot.
But from a security standpoint if B is compromised then any attacker has a good chance of
getting your password for C, D, and A.
(the solution is always ssh from a single, trusted machine, usually your desktop. Always go A
to B, A to C, A to D. Things like ssh keypairs help by making doing the 'right thing' more
convenient.)
I would not be surprised if a similar thing happened here. It's a very common bad habit.
Another useful piece of configuration: if your server accepts incoming ssh, block outgoing ssh with a local firewall rule. If you use your laptop for outgoing ssh, block incoming ssh with a local firewall rule.
ssh hygeine
Don't use passwords for remote authentication
Don't use passwords for remote authentication
Private Network Y has Host B and Private Server C.
* Your on 'Host A'.
* The NAT Firewall for 'Network Y' has a pinhole stuck in it so port 22 on host B is exposed to the world.
* You want to get to port 22 on 'Private Server C'
Don't use passwords for remote authentication
What happened with Fedora - and Red Hat too