Standards, the kernel, and Postfix

Standards like POSIX are meant to make life easier for application developers by providing rules on the semantics of system calls for multiple different platforms. Sometimes, though, operating system developers decide to change the behavior of their platform—with full knowledge that it breaks compatibility—for various reasons. This requires application developers to notice the change and take appropriate action; not doing so can lead to a security hole like the one found in the Postfix mail transfer agent (MTA) recently.

The behavior of links, created using the link() system call—on Linux, Solaris, and IRIX—is what tripped up Postfix. In particular, what happens when a hard link is made to a symbolic link. Many long-time UNIX hackers don't realize that you can even do that, nor what to expect if you do. Postfix relied on a particular, standard-specified behavior that many operating systems, including early versions of Linux, follow.

Links can be a somewhat confusing, or possibly unknown, part of UNIX-like filesystems, so a bit of explanation is in order. A link created with link()—also known as a hard link—is an alias for a particular file. It simply gives an additional name by which a particular chunk of bytes on the disk can be referenced. For example:

    link("/path/to/foo", "/link/to/foo");

creates a second entry in the filesystem (called /link/to/foo) which points to the same file as /path/to/foo. The file being linked to must exist and reside on the same filesystem as the link.

Symbolic links, on the other hand, are aliases of a different sort. A symbolic link creates a new entry (e.g. inode) in the filesystem which contains the path of the linked-to file as a string. There is no requirement that the file exist or be on the same filesystem—the only real requirement is that the path conform to standard pathname rules. The symlink() system call is used to create them:

    symlink("/path/to/foo", "/symlink/to/foo");

Both symbolic links and hard links can also be created from the command line using the ln command (adding a -s option for symbolic links).

So, when making a hard link to a symbolic link, there are two choices: either follow the symbolic link to its, possibly nonexistent, target and link to that or link to the symbolic link inode itself. POSIX requires that the symbolic link be fully resolved to an actual existing file, which is the behavior that Postfix relies upon.

The exact sequence of events is lost in the mists of time, but Linux changed to non-standard behavior—at least partially for compatibility with Solaris—in kernel version 1.3.56 (which was released in January 1996). Some discussion prior to that change adds an additional reason for it: user space has no way to make a link to a symbolic link without it. Some saw that as a flaw in the interface and proposed the change. An application developer that wanted the original behavior would be able to implement it by resolving any symbolic links before making the hard link.

To further complicate things, it appears that the POSIX behavior was restored in the 2.1 development series, only to be changed back in late 1998. This change led to the comments currently in fs/namei.c for the function implementing link():

    /*
     * Hardlinks are often used in delicate situations.  We avoid
     * security-related surprises by not following symlinks on the
     * newname.  --KAB
     *
     * We don't follow them on the oldname either to be compatible
     * with linux 2.0, and to avoid hard-linking to directories
     * and other special files.  --ADM
     */

Where oldname is the file being linked to and newname is the name being created. For the curious, KAB is Kevin Buhr and ADM is Alan Modra.

Unfortunately, according to Postfix author Wietse Venema, the link(2) man page didn't change until sometime in 2006. This makes it fairly difficult for application developers to learn about the change, especially because they may not follow kernel development closely.

Postfix allows root-owned symbolic links to be used as the target for local mail delivery, specifically to handle things like /dev/null on Solaris, which is a symbolic link. Because an attacker can make a link to a root-owned symbolic link on vulnerable systems, Postfix can get confused and deliver mail to files that it shouldn't. This can lead to privilege escalation (via executing code as root) by making a hard link to a symbolic link of an init script (CVE-2008-2936).

As Venema outlines in the Postfix security advisory, the problem can be resolved by requiring that symbolic links used for local delivery reside in a directory that is only writeable by root. It is not a perfect solution, though: "This change will break legitimate configurations that deliver mail to a symbolic link in a directory with less restrictive permissions." There are other workarounds for people who don't want to use the provided patch to Postfix. Protecting the mail spool directory is one solution; Venema provides a script to use to do that. Some systems can be configured to disallow links to files owned by others, which is another way to avoid the problem.

This issue has given Postfix a bit of a black eye, but that is rather unfair. The problem was found by a SUSE code inspection, but it has existed in certain kinds of Linux installations of Postfix for a long time. It could be argued that testing should have found it—there is a simple test for vulnerable systems—but relying on documented behavior that is part of an important standard that Linux is said to support is not completely unreasonable either. It is likely that the full implications of not supporting the standard were not completely understood until recently.

Linux was still in its infancy when the original change went in. One would like to think that a change of that type today would be nearly impossible because it breaks the kernel's user-space interface. If it were to happen, somehow, the resulting hue and cry would be loud enough that application developers would hear. But that's for intentional changes; a bug introduced into a dark corner of the kernel's API might go unnoticed for quite some time. Hopefully, none of those lingers for ten years before being discovered.

Update: The original article referred to CVE-2008-2937 as also being a consequence of the link issue, which it is not. It is an unrelated issue that was found during the same code review.

---

(Log in to post comments)

POSIX describes C library functions, not the actual system calls, right? So isn't this more of
a glibc issue, since its role is to provide library functions which invoke the proper system
calls?

Pedant point: POSIX incorporated all of SuS some years back; the result is 
called POSIX. (Confusing, I know.)

That manpage, and the posixoptions manpage it links to, are excellently done. (It is a tad
unclear, though, whether posixoptions describes the meaning of the options or whether it
instead describes what values those options have got on a Linux (glibc?) system...)

I understand how linux's behavior differs from the standard, but I don't see how this can fool
postfix into into doing something it wouldn't otherwise do.

If you have a root owned symlink to an init script (as in the example), can't you just mv that
symlink into /var/mail and get the same effect?

I'm sure I'm missing something here but as the article points out, this is a subtle area.

>The attacker can make a hard link, though, and that's where the hole is.

And grsec has had a kernel patch that addresses exactly this problem for a long time — denying
hardlinks to files in directories not owned or writable (one or the other, don't remember) by
the actor.

Hmmm...  Seems like "not writable by" might be more likely than "not owned by."  While I can't
think of a specific instance, I imagine that disallowing hard links in /tmp might confuse a
couple things.  

(But think of all the race conditions we might close!  Oh, wait, that's mostly symlinks.)

Yes it does confuse a few things. Not in /tmp, but surprisingly, in /var/spool/atjobs! I
solved it by augmenting the batch to allow an exception for system UIDs (uid < 1000, or
whatever you specified as an exception).

Ah. Because you can't unlink the original name of the root-owned symlink (in the case of an
important file in /etc at leats), you can't get the same effect.

And I can't think of a root-owned file that would live in a dir a regular user can write to...

I don't think it's possible for hard links to span filesystems. The point of a hard link is
that the two names have the same inode, and a dentry pointing to an inode on a different
filesystem either wouldn't work at all (since the dentry doesn't include enough information to
specify the inode completely) or wouldn't be really a separate filesystem.

I suppose, however, you could have a single filesystem that you'd carved up and arranged with
bind mounts to look like multiple filesystems, and the OS wouldn't necessarily block creating
the hard links (although, IIRC, Linux presently does).

That's my understanding as well.  I was pretty sure traditional *nix filesystems worked this
way.  What I wasn't sure of is whether union mounts, or one of the newer filesystems supported
something fancier.  

I know if I state unequivocally that it can't be done, the odds are against me someone will
demonstrate I'm wrong.  ;-)

I find it a bit odd that a normal user can do things like this:
ln /etc/shadow ~/myfile
and control where system files show up in the file system. 

Would something break if hardlinking to files not owned by yourself would be forbidden?

its even more odd if you create the link in a directory which has the sticky bit set:

ln /etc/shadow /tmp/myfile

you can create that link, but you can't delete it afterwards (only root can do that)

> ln /etc/shadow /tmp/myfile
> you can create that link, but you can't delete it afterwards (only root can do that)

Wow, that's insane!

I'd never thought about this issue of hardlinks...but it kinda seems bad that
you can create a file  owned by another user.

By golly, you're right!

% su
# cp /bin/true /test
# chmod u+s /test
# exit
% ln /test ~/suid_program
% su
# rm /test
# exit
% ls -l ~/suid_program
-rwsr-xr-x 1 root root 26468 2008-08-21 13:06 suid_program

Yikes!

Perhaps this indicates the need for a 'delete' system call which makes sure a file is really
gone, rather than merely 'unlink'ing one of the names which points to it.

I guess it would also be a (small) improvement for package managers to overwrite existing
files, instead of removing the old one first.

Exactly.  The relatively smooth upgrading of software on Unix-like systems is because you can
unlink the old file and create a new file with the same name.  The old one will continue to
exist, with its old data, as long as a process has it open.  If you try to fix the suid binary
problem by forcibly removing a file, or overwriting its contents, then you risk breaking
running code.

(Since writing bytes to a file is not atomic, you could theoretically introduce new security
holes by overwriting the contents of a suid binary.  What if someone runs it halfway through
the update, when it contains a mixture of old and new code?)

One answer would be to move the suid flag out of the inode and put it in the directory entry.
Then you could make as many links to a file as you wanted, but you wouldn't be able to get the
suid magic (unless, of course, you have permission to set the suid bit anyway).  But given the
conservatism of Unix-like systems and the large amount of filesystem code that would have to
change, I can't see this happening soon.

There seem to be three issues:

1. Atomic replacement of an executable (by name)
2. Removing permissions from potentially hard linked old copies (by inode)
3. Reporting possible nefarious activities by hard linkers.

I think this sequence deals with them all:

1. Hard link canonical name of old executable to first temporary name.
2. Create new executable with second temporary name and appropriate privs.
3. rename() second temporary name to canonical name (this is atomic).
4. Chown/chmod first temporary name to remove any special privileges, execute bits, etc.
5. stat() first temporary name and report if there is more then one link
(Should be just one from first temporary name at this point).
6. Unlink first temporary name.

At the end of the sequence, you will have atomically replaced the executable, removed any
special privileges from any links people may have hidden away, and reported (to the extent
possible without a full search of the filesystem) that somebody has been linking to
executables.

Of course, the sendmail example is a bad one as typically there is more then one valid link.
'sendmail' and 'newaliases were traditionally the same program.  I know of no way to atomically
make a bunch of links to one
program so the best I can think of would be to make a bunch of temporary names for the new
execuable and then rename() them all into place sequentially in steps 2 and 3.  Steps 4-6
could remain the same and would still be able to accurately report improper links while at the
same time making them useless.

Here is a handy tool to make a collection of hard links to suid or sgid binaries and keep the
collection up to date.  Use as

% mkdir suid_collection
% suid_harvester /sbin /bin /usr/sbin /usr/bin /usr/local/bin suid_collection

#!/usr/bin/perl
use warnings;
use strict;
use 5.010;
use File::Find;
use File::stat;
use Fcntl qw(:mode);
use File::Spec;

sub interesting {
    my $sb = shift;
    my $mode = $sb->mode;
    my $suid = $mode & S_ISUID;
    my $sgid = $mode & S_ISGID;
    my $uid = $sb->uid;
    my $gid = $sb->gid;
    my $xusr = $mode & S_IXUSR;
    my $xgrp = $mode & S_IXGRP;
    my $xoth = $mode & S_IXOTH;

    if ($suid) {
	if ($xoth) {
	    # suid, executable by all.
	    return 1;
	}
	elsif ($xgrp) {
	    # suid, executable by some group.
	    return 1;
	}
    }
    elsif ($sgid) {
	if ($xoth) {
	    # sgid, executable by all.
	    return 1;
	}
	elsif ($uid != 0 and $xusr) {
	    # suid, executable by someone non-root.
	    return 1;
	}
    }
    return 0;
}

die "usage: $0 source... dest\n"
  if @ARGV < 2;
my $dest = pop @ARGV;
my @source = @ARGV;
my $dest_abs = File::Spec->rel2abs($dest);
chdir $dest or die "cannot chdir to $dest: $!";
my %already;
foreach (glob '*') {
    if (/\A([0-9]+)[.].+\z/) {
	my $ino = $1;
	$already{$ino}++ && warn "two files in $dest beginning $ino.\n";
	my $sb = lstat $_;
	warn("cannot lstat $dest/$_, skipping\n"), next if not $sb;
	if (not interesting $sb) {
	    warn "sadly, $_ no longer has useful permissions\n";
	}
    }
}
sub for_each_file {
    my $sb = lstat $_;
    warn("cannot lstat $File::Find::name, skipping\n"), return if not $sb;
    my $ino = $sb->ino;
    $already{$ino}++ && return;
    return if not interesting $sb;
    my $link_to = "$ino.$_";
    say "$File::Find::name -> $link_to";
    link $File::Find::name, "$dest_abs/$link_to"
      or warn "cannot link to $link_to: $!\n";
}
find \&for_each_file, @source;

 Actually the really savvy admin, or patch/fix script author, would perform  a sanity check on
files to be removed by the patch, opening it, fstat()ing  it, chmod() it to remove SUID/SGID
bits, unlink()ing it, and checking to ensure that the next fstat() returns a  link count of
zero.  If that fails (someone else has a hard link to it) then  over-write the file contents
with an program which does the following:

  * syslog()s the user and the command argument
  * optionally warns the user that they are attempting to use an out-dated version of the
program (and advising them against using hard links to system binaries in general?)
  * optionally wraps the new binary (execve()'s it) or exits

>>I find it a bit odd that a normal user can do things like this: ln /etc/shadow ~/myfile and
control where system files show up in the file system.
>
> Not just odd, but a security problem too. 

This is why the package manager tool, that replaces the application, should first remove all
read/write/exec permissions from the file before unlinking it. This way the hardlink won't be
usable by the attacker anymore, as he can't execute it anymore.
A call to revoke() might be needed, too, to close all currently open mmap(). I'm not sure on
that for regular files...

I don't know whether apt/rpm actually do this.

But udev, for example, uses this to avoid attacks on /dev files.

So how do other mail servers (qmail, for example) handle this situation? It seems like an
issue that is very difficult to pinpoint, but perhaps could have been avoided altogether by
more cautious design.

Interesting. I always though the ln command had the argument in the wrong order, and it turns
out that the link function switches the order...

> The examples for link(2) and symlink(2) currently both have the arguments
> the wrong way round.

Gah!  Fixed now, thanks!

jake