What happened with Fedora - and Red Hat too [LWN.net]

What happened with Fedora - and Red Hat too

Posted Aug 22, 2008 17:11 UTC (Fri) by drag (subscriber, #31333)
Parent article: What happened with Fedora - and Red Hat too

A similar thing happened with Debian a few years ago.

If I remember correctly...

How it was detected was that the attacker tried to modify a file or two that was monitored by
Aide. Apparently it wasn't a very good attacker as he didn't notice they were running a HIDS
and didn't take steps to modify the Aide database or otherwise subvert it so he was detected
pretty early on.

How the security breach happened was that a developer had let his ssh password get exposed on
a insecure system.

-------------------------------------------------

As is a unfortunate habit among many Linux users and admin is that they ssh from one machine
to another willy nilly. 

So you ssh from A to B and then from B to C and then from C to D and then use scp from D back
to A.  Just over a day's work you often forget what machine your actually logged into and
functionally it doesn't matter a whole lot. 

But from a security standpoint if B is compromised then any attacker has a good chance of
getting your password for C, D, and A. 

(the solution is always ssh from a single, trusted machine, usually your desktop. Always go A
to B, A to C, A to D. Things like ssh keypairs help by making doing the 'right thing' more
convenient.)

I would not be surprised if a similar thing happened here. It's a very common bad habit.

(Log in to post comments)

Don't use passwords for remote authentication

Posted Aug 24, 2008 11:01 UTC (Sun) by tialaramex (subscriber, #21167) [Link]

Don't use passwords, especially for privileged access. SSH provides a two factor system which is adequate for anyone who doesn't find the command line intimidating.

Create user SSH private keys only on machines you have physical control over, or where local policy prohibits such caution, on a few machines where you have the most confidence of their integrity.

Use a passphrase to protect the keys. Use SSH agent software. The agent software uses the passphrase to unlock a key and hold it in memory temporarily. If the key files (stored on disk) are obtained by malicious third parties (e.g. by accessing backups, old drives mistakenly sold without being wiped) they are useless without the passphrase to decrypt them.

You can then use the key to log into machines without presenting a password at all, if the target machine is compromised you reveal only the relatively worthless public key identity of your user. An attacker witnessing hundreds of logins of this sort with full root access on the target machine learns nothing except the identities of frequent users. Contrast this to the situation with passwords, where he learns hundreds of valuable passwords.

Where convenience (or e.g. firewall policy) dictates that you SSH into one security pivot machine A and from there into others (B ... Z), you have two choices which reflect different security beliefs about the machines.

1. SSH to machine A, separate key on that machine for SSH to B ... Z. This keeps security of the set A ... Z separate from security for any other machines you have access to. An attacker in full control of A can authenticate (using the key stored on A) to other alphabetic machines at will but not elsewhere.

2. SSH agent tunnel. Local machine SSH agent is tunnelled to A for authenticating connections from A to other alphabetic machines. This limits power of attacker in full control of A to hijacking your authentication capability when you are logged into A. They can authenticate to any system for which your key validates, not just A ... Z, but the key and its passphrase remain inviolate on your local system.

In either case attackers who somehow have privileges on other alphabetic machines (B ... Z) learn nothing helpful and cannot leverage their privileges to attack other systems via SSH.

Don't use passwords for remote authentication

Posted Aug 26, 2008 8:00 UTC (Tue) by drag (subscriber, #31333) [Link]

> Don't use passwords, especially for privileged access. SSH provides a two factor system which is adequate for anyone who doesn't find the command line intimidating.

Command line? Bah. Seahorse with Gnome-keyring and ssh-agent provide a more-or-less complete gui-based ssh solution for Linux desktops. Secure remote GUI, secure file transfer, secure terminal access.

One happy accident was that Seahorse used Gnutls so that it was not vulnerable to the OpenSSL issues with Debian. Dumb luck that one, at least on my part.

Certainly you can use command line, I do, but that's not even necessary anymore.

> Create user SSH private keys only on machines you have physical control over, or where local policy prohibits such caution, on a few machines where you have the most confidence of their integrity.

When I can help it I only have ssh keys on one machine. My local workstation. No point in anything else, usually.

I look at it like a spider. My reach goes out to my machines from my laptops. Without the key there is no way for anybody to get access to any of my systems. Only reach out and pull, never push from outside in.

From a personal standpoint ssh does everything.

Which, of course, is why the attacker was very interested in OpenSSH.

------------------------------------------------

As far as firewalls go...

As long as I can get out into the internet over HTTPS then the entire internet is available to me. Such is the magic of OpenVPN... It's virtually impossible to block if you want to allow even proxy-filtered http access. Stick a remote server up to run OpenVPN on port 443, tell it to use TCP, and a couple other options and your done. I use Shorewall for the firewall/router rules. Then I have my own private NAT firewall sitting out there on the internet, hundreds of miles from any network I use.

-----------------------------------

For private networks in private networks with filtering firewalls at the router, as long as they have remote access to ssh on a machine on the inside then it's possible to access any port on any server on that network using Ssh port-forwarding. No special rules required.

I've even done port reversal and opened up holes in networks so I can get remote access to a network that otherwise would of been completely blocked off. (Multiple levels of NAT be damned. If you can get out, they can get in. No question about it. Just need a man (aka Windows XP and Outlook) on the inside.)

So if you have a couple hops to get to a internal server then it's possible to tunnel ssh over ssh via forwarding and keep one session separate from the other, even if it's inside the other. Of course you may run into latency issues that cause such a setup to be to irritating to use in that manner.

That is:

(private network X)|firewall||internet||firewall|(private network Y)

Private Network X has Host A on it.
Private Network Y has Host B and Private Server C.

Now.. Lets assume that:
* Your on 'Host A'.
* The NAT Firewall for 'Network Y' has a pinhole stuck in it so port 22 on host B is exposed to the world.
* You want to get to port 22 on 'Private Server C'

So you ssh from Host A to Host B. Then you setup port forwarding so port 8020 on Host A connects over ssh to port 22 on Private Server C.

Now you can connect directly from the Host A to Private Server C via port 8020 on A's localhost. That is "ssh -P8020 127.0.0.1 " connects directly to "privateServerC:22"

So you see, tunnel ssh over ssh to piggyback while preserving isolation. :) Even if Host B got hacked, they still would not have any access to Server C.

Works very well for any port on any system.

Don't use passwords for remote authentication

Posted Aug 26, 2008 12:24 UTC (Tue) by tialaramex (subscriber, #21167) [Link]

The ssh-in-ssh tunnel is clever (and worth recommending) but doesn't really solve my day-to-day need for the scenario I outlined. The trouble is that if you connect from A to B via commercial leased line at 2Mbit/s and want to move a large file from B to C (which is next to it in a rack and via GigE) then it will take days to move via your ssh-in-ssh trick, compared to a few minutes with some trust invested in B*. Tunnels are transparent... right up until bandwidth and latency matters.

* Of course you can conjure up solutions involving an untrusted connection directly from B to C carrying just the file contents, then verifying a checksum via the SSH tunnel and so on. But it'd take a lot of paranoia to justify actually writing scripts for that rather than just agreeing it would work in principle.