LWN: Comments on "Linux botnets"

Single Packet Authentication is a far better solution.

shapr — Wed, 14 May 2008 20:56:44 +0000

I prefer Single Packet Authentication. The great advantage of SPA is that brute force scanners never know there's a service running.

The general case is, don't show headers when a user connects, just accept a connection when there's a correct login, and silently drop packets for illegal logins. That approach would dramatically reduce the attack surface for servers.

ssh public keys

muwlgr — Thu, 01 Mar 2007 22:14:03 +0000

Not everyone knows about ssh-copy-id utility.
May be that's what you need.
This operation is very popular, so openssh developers have written the tool for us all...

challenge-response on ssh

nix — Tue, 20 Feb 2007 20:47:56 +0000

Well, ChallengeResponseAuthentication == public-key authentication and/or
use of OPIE, RSA SecurID, or some other one-time authentication system
(some of which OpenSSH has native support for).

remote root

hein.zelle — Mon, 19 Feb 2007 15:54:12 +0000

> Where do you see the problem? I agree that logging of access is not as it
> should be but it is still available and come one, having root access does
> on most systems mean you have all the power to manipulate the logs. So
> why care.

One reason I care is that it's easy to accidently turn password authentication back on. On many debian systems I've seen, the option UsePAM (on by default) effectively allows password authentication, even when PasswordAuthentication is off. This is not the case on the latest ubuntu, but dangerous nevertheless. I'd rather have an ssh login as a regular user, and then become root using su.

What is the reasoning behind not using su to become root? I understand the password will go over the line, but it's encrypted. Is this advised against for fear of keyloggers or so?

Linux botnets

smoogen — Fri, 16 Feb 2007 02:34:34 +0000

The standard way I have seen it done is that the application uses 'local' delivery and that is checked for 'correctness' at the machine and at the border SMTP router. This stops the majority of spambots in a large environment.

Linux botnets

smoogen — Fri, 16 Feb 2007 02:32:11 +0000

I will say that Selinux has saved a customers bacon before.. the person had a bad PHP app installed and it got abused. The attacker was not able to execute anything however because the Selinux policy wouldnt let the attacks do anything from network connections to data viewing.

[While I do not use it.. I am betting that grsecurity might be able to do the same thing.]

SSH scanning -- solutions

smoogen — Fri, 16 Feb 2007 02:28:26 +0000

Thanks for everyone putting up various solutions.. they should make interesting grumpy old security admin articles some day.

They will also be handy for the admin who at 2am has to fix this problem and does a google search.

Linux botnets

smoogen — Fri, 16 Feb 2007 02:27:07 +0000

Ok the security gained by using two layers is via tracking down who logged in... which becomes very important in large teams. If you are administrating a couple hundred linux servers you may have a team of 5-12 people who need root access. Knowing who executed a root level command and when is important and more secure in that if you lock down sudo you can see what they ran versus having a black hole that root logged in at 02:00 and logged out at 02:30 and you have no idea what they ran.

In the case of small teams.. you may not feel that you need this, but it comes in handy if the business grows... you find yourself with 12-20 people with the root password.

remote root

dd9jn — Fri, 16 Feb 2007 00:25:08 +0000

"Allowing direct root access means that root access is not revokable
per-admin; if the password is somehow compromised"

FWIW, I was talking about public key authentication for root access. This also means that revoking access is as simple as deleting one line from authorized_keys.

Where do you see the problem? I agree that logging of access is not as it should be but it is still available and come one, having root access does on most systems mean you have all the power to manipulate the logs. So why care.

challenge-response on ssh

ccyoung — Thu, 15 Feb 2007 23:52:31 +0000

how? is there a package? or does it require real work?

remote root

rfunk — Thu, 15 Feb 2007 21:21:08 +0000

This is an old debate. But you'll be hard-pressed to find an experienced professional
sysadmin who will allow remote root logins.

Allowing direct root access means that root access is not revokable per-admin; if the
password is somehow compromised (e.g. an admin is fired or is careless with the
password) you have to change the root password and communicate that to all admins
(with the associated insecurity of that communication). If admins are getting root from
their own accounts, then it's sufficient to disable or re-password a single admin's account
without affecting other admins.

So in the sudo case, if that one password is guessed, it's easier to recover than in the
single remote-root case.

Just running a root shell is dangerous. It's much better to be root only for what needs to
be done as root, to avoid accidents or possibly tripping over sabotage (e.g. someone
having gotten in and messing with your ls command).

This slashdot comment is one place that covers the issue well:
http://it.slashdot.org/comments.pl?sid=180864&cid=149...

Linux botnets

tetromino — Thu, 15 Feb 2007 20:22:16 +0000

How is sudo any more secure than root ssh logins with a password? In either case, if you can guess ONE password, you get remote root...

SSH scanning

kh — Thu, 15 Feb 2007 16:30:37 +0000

I have been happy with denyhosts

SSH scanning

stevan — Thu, 15 Feb 2007 16:29:56 +0000

The blacklist.py python script
(http://blinkeye.ch/mediawiki/index.php/SSH_Blocking) works extremely well
for manging ssh scans, in our experience. The answer, though, is, of
course, keyed-only ssh access.

SSH scanning

nowster — Thu, 15 Feb 2007 16:19:35 +0000

Order is important in these iptables commands. The commands in the parent appear to match on any traffic. Use instead:

# Accept trusted hosts
iptables -A INPUT -s 192.168.0.0/24 -p tcp -m tcp --dport ssh -j ACCEPT

# For outsiders, rate-limit and enjoy
iptables -A INPUT -p tcp -m tcp --dport ssh \
        -m state --state NEW \
        -m recent --hitcount 3 --seconds 180 --update -j DROP

iptables -A INPUT -p tcp -m tcp --dport ssh \
        -m state --state NEW \
        -m recent --set -j ACCEPT

Linux botnets

rfunk — Thu, 15 Feb 2007 15:20:40 +0000

Use sudo, with user's password. Make the basket of users who have access to sudo be
very small, and watch it closely.

Being able to get direct access to a root shell from the internet is just crazy.

PHP sending mail

rfunk — Thu, 15 Feb 2007 15:17:59 +0000

The local SMTP server can do its own checking to detect and block outgoing spam.

It can also queue the real messages when the recipient's mail server is temporarily
unavailable; sending directly from PHP can't do that.

SSH scanning

nix — Thu, 15 Feb 2007 15:02:00 +0000

Why not just turn off password-authentication on your Internet-facing SSHen? Stick to challenge-response and you'll be safe from all these scanners (modulo major holes in sshd itself, which are rare.)

Linux botnets

tialaramex — Thu, 15 Feb 2007 14:56:49 +0000

I don't see any reason to permit remote root login on Internet-facing servers (or desktop machines for that matter). A genuine administrator should have a user account, unique to them as a person, which can be audited.

The first attack scenario I'm considering is an outsider who is able to connect to the SSH server and perhaps has some limited (unprivileged) access to the target machine (e.g forum user access on a web server), plus they can snoop some of your traffic. This scenario would be typical for a black hat setting up a zombie network, who already has subverted some nearby machines in the network but not yours.

With root SSH logins, they can target the root account (yes, it could be renamed in theory, but unlike the Windows "Administrator" account that's not a routine precaution) and the only thing keeping you safe is that SSH server's authentication. Any flaws in that single line of defense whether security mistakes (you left your laptop unattended and they stole the private key file?) or system problems (OpenSSH is revealed to allow in 1-in-4 billion connections without authenticating) are a total loss.

In my alternative, they must actively target administrative users, without knowing in advance what names are used. Even if they get SSH access as a user, they still need to escalate to root, which is another layer of security, albeit one that we know is much weaker. On the other hand if they obtain (for example by social engineering) a root password or sudo root password equivalent, they still need remote access to use it. So in either case you've got two layer security.

I use and recommend private user logins, via ssh public keys PLUS an audited authentication step for escalating to root privileges. Also people should pay attention to the security of their SSH private keys. On every machine where you keep such a key, consider how a black hat might get access to the key and what they can access with it once they have it. If you use a passphrase (which you should) how are you sure the entry method is itself secure (IIRC trojans asking for the SSH passphrase have been seen in the wild) ?

Linux botnets

dd9jn — Thu, 15 Feb 2007 13:03:37 +0000

So and how do you get root access? Using su requires a password again and sudo without password will do nothing else but alias that user account to root. There is an old crypto rule which states: Put all your secret into one basket and watch that basket very well.

Public key authentication is far better than any password scheme. If you worry about a private key compromise, use a smart card.

Linux botnets

minichaz — Thu, 15 Feb 2007 12:43:26 +0000

I agree with using keys (ideally with passphrases too) but there's no need to allow root logins through SSH, particularly on internet facing servers. Set "PermitRootLogin no" and use "AllowGroups" or "AllowUsers" to prevent attacks against other accounts which should never connect over SSH.

Charlie

SSH scanning - fail2ban

DG — Thu, 15 Feb 2007 12:10:08 +0000

alternatively try fail2ban (on ubuntu/debian)

Linux botnets

NAR — Thu, 15 Feb 2007 12:09:06 +0000

PHP applications should not be making outgoing network requests.

Then how should a PHP-based forum software send a "registration succeeded" e-mail to the user? Could connect to the local SMTP server, but spam could be sent out this way too.

Bye,NAR

ssh public keys

dion — Thu, 15 Feb 2007 11:05:27 +0000

I think the reason that some (a lot of) people use passwords with ssh is that they see ssh as "telnet, only secure".

They haven't looked at authorized_keys or they think that distributing their keys is too much trouble.

I've recently moved from doing copy+paste of id_dsa.pub when needing access to using a handy little shell script (don't run it on your machine, unless you want me to access it): http://dion.swamp.dk/ssh.sh

Before doing that I some added only one of my desktop machines or maybe I didn't bother with the key setup at all and just used password login, because I rarely needed access to that particular system.

With this solution all I need to get access to a new box is to tell the administrator (well, myself most times) to run that script and I'll never need to log in with a password.

Putting your public key on your website really ought to be in every "root account for dummies"-type book.

SSH scanning

bkoz — Thu, 15 Feb 2007 10:51:44 +0000

Thanks for the iptables hackery. This is the #1 issue I see in my logs.

Linux botnets

wingo — Thu, 15 Feb 2007 09:47:52 +0000

Informative comment, thanks.

SSH scanning

ahoogerhuis — Thu, 15 Feb 2007 09:44:57 +0000

# Accept trusted hosts
iptables -A INPUT -s 192.168.0.0/24 -p tcp -m tcp --dport ssh -j ACCEPT

# For outsiders, rate-limit and enjoy
iptables -A INPUT -m recent -m state -p tcp -m tcp --dport ssh --state NEW --hitcount 3 --seconds 180 --update -j DROP
iptables -A INPUT -m recent -m state -p tcp -m tcp --dport ssh --set --state NEW -j ACCEPT

i.e. don't meddle in SSH from places we trust, for outsiders that DO need access, give them three attempts, otherwise it's the doghouse for a few minutes. Simple, very effective.

-A

Linux botnets

dd9jn — Thu, 15 Feb 2007 09:34:06 +0000

Don't even use a password at all to login in to a server. sshd_config should have an entry "PermitRootLogin without-password" and user accounts should all have a disabled password. Use ~/.ssh/authorized_keys. If you have a need to login from more than one client machine, use a smart card to access the server. I know that this is a trivial suggestion but when I occasionally see people login to their servers, most are entering a password.

SSH scanning

ldo — Thu, 15 Feb 2007 09:11:42 +0000

I wrote a script which continually scanned /var/log/messages for "invalid user" entries logged by sshd, and did a

iptables --append INPUT --source srcaddr -j DROP

which was removed after 10 minutes. Most of the scanners never came back after the 10 minutes.

Linux botnets

zlynx — Thu, 15 Feb 2007 02:47:09 +0000

SELinux can help here. PHP applications should not be making outgoing network requests.

If SELinux is too difficult, iptables can filter away outgoing traffic as well. Not enough people put outgoing blocks on their firewalls.

A server farm / rack provider might also run IDS like Snort. See if you can get them to copy you on IDS alerts related to your IPs.

And for crying out loud, don't use your login password for your application's SQL account, helpfully listed in a plain text PHP include.

Linux botnets

smoogen — Thu, 15 Feb 2007 02:42:15 +0000

Having cleaned up a share of Linux systems.. the standard infection methods are:

1) ssh scanning. The botnet uses a dictionary attack against accounts to see what people have let open. In most cases, the crack-masters have gone through many broken systems and worked out what the most common account names are, and then used large numbers of broken into systems as a very large john-the-ripper cluster to figure out what passwords they could get. They then use those passwords as most likely because people choose passwords similarly. They then use large herds of bots to scan every open 22 port. Some botnets seem to also scan other 'common' ssh ports (2222 and 23). [I have seen bot 'clusters' scan a network and then whatever port you stuck SSH on other boxes would then start going after.]

2) PHP scanning. Looking at my logs I get about 40 scans a week for every PHP application that has had a vulnerability since 2000.

3) Webmin scanning. This is where popular webmin ports are scanned for and a similar set of tools as the ssh scanning are used. A lot of application vendors like to use this to help troubleshoot their applications from afar.. many of these vendors don't update the software or are aware that the webmin they installed was bad. They also like to choose password like the application vendor name spelt backwards.

4) Xvnc scanning. Same thing as above... the Oracle application Xvnc is rather old.

The big thing that comes up with several of these is that most botnet people are quite happy if they dont get root access. The ability to create a .<space> directory in the person home directory, /tmp or /var/tmp is fine with them. They can still execute their EnergyMech bot to get to some undernet IRC channel and get commands on what spam to send through the world. This doesn't mean that they wont' try to get root access on the system.. but for 99% of what they want to do.. they do not need to be root on a system.. just a normal user.