Dealing with automated SSH password-guessing

Just about everyone who runs a Unix server on the internet uses SSH for remote access, and almost everyone who does that will be familiar with the log footprints of automated password-guessing bots. Although decently-secure passwords do much to harden a server against such attacks, the costs of dealing with the continual stream of failed logins can be considerable. There are ways to mitigate these costs.

We've probably all seen thousands of lines like this in our system logs:

    Oct 21 22:49:13 wibbly sshd[20474]: Failed password for user from 192.168.1.42 port 52498 ssh2

Generally, these are the signatures of self-propagating password-guessing bots that try to crack passwords by brute-force; if they succeed, they copy themselves onto the destination system and start scanning for remote systems to password-guess from there. They also often signal home to some control organization that an account has been cracked. There are a lot of bots out there: for this article, I spun up a completely fresh VPS with one of my providers, and the first such log entries appeared 71 minutes after the machine first booted.

Let's take it for granted that all LWN readers have sensible policies regarding password complexity that are rigorously and automatically enforced on all our users, and so we're unlikely to fall victim to these bots. Nevertheless both SSH connection setup and teardown, and the business of validating a password, are computationally expensive. If the internet has free rein to ask you to do this, the CPU costs in particular can be significant. I found that until I took measures in mitigation, my sshd was constantly using about 20% of a CPU to evaluate and reject login attempts.

Change the port number

It's a really simple fix, but changing the port number away from the default of 22 can be surprisingly effective. My wife and I both run VPSes with the same provider, both of which have been up and running for several years, identical in most respects save that she runs her sshd on a non-standard port number in the mid-hundreds. In the last four complete weeks of logs, my VPS saw a total of slightly over 23,000 rejections; hers saw three.

It's not an infallible technique on its own; recently, someone found her alternate port, and tried several hundred different user/password combinations before getting bored and moving on. It can also work badly with some clients that can't easily be configured to use a non-standard port number.

But changing the port number is probably the lightest-weight mitigation technique in this article; it is a simple matter of changing a line in /etc/ssh/sshd_config (or your distribution's configuration file) from Port 22 to Port 567, for example. You will need to pick a port without a current listener to avoid clashes; netstat can be helpful for this. Moreover, you will need to add permission to your firewall rules for the new port number before you restart the service, then take out the old port 22 permission after the restart and confirmation that everything works. I advise against just changing the existing firewall permission from port 22 to the new port; it's easy to get the sequencing wrong, and lock yourself out of your remote server that way.

Fail2ban

Fail2ban is an elegant piece of software that watches log files for evidence of abuse of network services, then adjusts firewall rules in realtime to ban access from the offending IP addresses to the service in question. It's pretty easy to find; CentOS users will find it in the EPEL repository, Ubuntu seems to have it in the standard repositories, and it shouldn't be hard to find for most other major distributions.

The project has a good page of HOWTOs, which include an excellent guide to using it on SSH. Although I disagree with banning IP addresses for a whole day — I think it's much too easy to lock oneself out of one's own systems — I found it otherwise easy to follow, and it works. On my CentOS system, I configured in the EPEL repository, did a "yum install fail2ban", and put the following into /etc/fail2ban/jail.local:

    [ssh-iptables]
    bantime = 600
    enabled = true
    filter = sshd
    action = iptables[name=SSH, port=ssh, protocol=tcp]
    maxretry = 5
    logpath=/var/log/secure

After that, doing a "service fail2ban start" was enough to lock out a test client system after five failed SSH attempts.

iptables rate limiting

One of the shortcomings of Fail2ban is a strong IPv4 focus. The huge size of the IPv6 address space makes scanning for SSH servers much more expensive, but my servers are already seeing non-service-specific IPv6 port scans most days, and the problem will only grow. As the Internet of Things comes along, and more houses have routed IPv6, there will be a lot more SSH servers to find. It's one thing having my colocated server being password-guessed by random servers in North Korea; it is quite another having all my kitchen appliances being password-guessed by half a million badly-configured light bulbs. So I do my rate-limiting with iptables (and ip6tables). The following lines of iptables commands log and reject the fourth and subsequent new connections in a rolling 60-second window:

    iptables -A INPUT -p tcp -m state --state NEW --dport 22 -m recent \
             --name sshrate --set
    iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent \
             --name sshrate --rcheck --seconds 60 --hitcount 4 -j LOG \
	     --log-prefix "SSH REJECT: "
    iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent \
             --name sshrate --rcheck --seconds 60 --hitcount 4 -j REJECT \
	     --reject-with tcp-reset

These lines need to come near the beginning of your iptables rules; specifically, before any blanket ACCEPT for any ESTABLISHED connections, and before any general permit for port 22 (or your sshd port of choice). You can use identical rules for ip6tables, and if your router is running Linux, you can add them to the FORWARD chain as well, protecting all the devices inside your network.

This is an extremely inexpensive way of rejecting over-frequent attempts to authenticate via SSH. Looking at the last complete week of logs on my colocated server, iptables refused over 153,000 connection attempts. Only 9,800 connection attempts were permitted to talk to sshd, and refused for having invalid passwords. 94% of malicious traffic was rejected quite cheaply, by iptables.

Although the ban period is only 60 seconds, if the banned address keeps trying to connect during the period of the ban — as bots tend to do — the ban continually renews itself. Only when the connection attempts stop can the 60-second window clear out and allow new connection attempts to succeed. Thus, this approach tends to impact bots more than humans, which is good. On the odd occasions I'm using a tool that needs to make large numbers of frequent connections, I can always make a manual SSH connection in SSH master mode, then tell the tool to use SSH slave mode and feed it the relevant ControlPath.

Public-key authentication

Another simple but powerful approach is to disallow password-based authentication in favor of public-key-based authentication. Changing the PasswordAuthentication parameter from yes to no in the sshd configuration file, will cause sshd to refuse attempts to authenticate via password. Your system still pays the setup and teardown cost of each SSH connection, but it doesn't have the expense of validating a password each time.

I can't quantify the utility of this approach, because my users wouldn't put up with it, but with Amazon EC2, nearly all new instances come out-of-the-box configured to refuse password-based authentication. I suspect this has done a great deal to decrease the number of systems on the internet that are vulnerable to automated password-guessing.

Two-factor authentication

As mentioned above, my users needed the ability to authenticate via password from time to time; for example, sometimes they needed to log in from a work site, where they were unwilling to leave a trusted key pair installed for a long time.

Two-factor authentication (2FA) is an excellent way to handle this; the recent rise in popularity of Google Authenticator has done much to introduce people to the idea, and to demystify it. Since Google Authenticator is simply a slightly idiosyncratic implementation of OATH Open Authentication, much can be done with this infrastructure without coupling oneself permanently to Google. Note that, as described below, this does not affect public-key-based SSH access; that can continue without the need for a 2FA exchange. Only password-based access requires 2FA, thus completely frustrating the bots.

On my CentOS 6 system, I had to do:

    # yum install pam_oath
    # ln -s /usr/lib64/security/pam_oath.so /lib64/security/  

Then I created a the /etc/users.oath file with mode 600 and owner root:root, and populated it with lines like:

    #type   username        pin     start seed
    HOTP   yatest             -        123a567890123b567890123c567890123d567890

where that start seed is a 40-hex-digit string of great randomness; I generated it with:

    # dd if=/dev/random bs=1k count=1|sha1sum

This seed should also be fed to an OATH HOTP software token. Google Authenticator, which is one such, suffers from the idiosyncrasy that it wants its seed in base-32 encoding, rather than hex. You can convert it yourself, or use a friendlier software OATH token generator, such as Android Token. Once this is done, the token will produce a stream of seemingly-random six-digit numbers that will be exactly what your server is expecting.

Then, in the file /etc/pam.d/sshd, immediately before the first line containing password-auth I added the line:

    auth required pam_oath.so usersfile=/etc/users.oath window=10

and in /etc/ssh/sshd_config ensured I had the lines:

    UsePAM yes
    PasswordAuthentication no
    ChallengeResponseAuthentication yes 

Once I had restarted sshd, I found that when SSHing in, I was asked first for the OATH six-digit number from the token, secondly for my password, and then granted access.

As an aside, do not succumb to the temptation to produce more than one or two token numbers right away. The function of HOTP OATH is to produce a sequence of numbers that are completely predictable if you know the secret seed (but not predictable if you only know previous numbers in the sequence). The window=10 parameter in the /etc/pam.d/sshd entry means that the server should accept not only the next expected number in the sequence, but any of the nine numbers after that as well. If you keep pressing the button on your software token, and run through (say) the next 20 numbers, you will get further through the sequence than the server expects you to be. Once server and token get out of sync, best practice is to reset both server and token, and to change the shared secret.

Note that you should keep a live SSH session, authenticated as root, throughout this process, so you don't accidentally lock yourself out of your own system. Better still, be within six feet of the system console when trying it for the first time.

In conclusion, if you don't like this incessant password-guessing stressing your system, you have options. Work out which one(s) best fit your needs, and give them a go.

Index entries for this article
Security	Tools/Password guessing prevention
GuestArticles	Yates, Tom

---

to post comments