LWN.net Logo

The Savannah Compromise - what really happened?

January 1, 2004

This article was contributed by Joe 'Zonker' Brockmeier.

2003 hasn't been a banner year for computer security, and that includes Linux. The CVS repository for the Linux kernel was attacked (if clumsily), several servers related to the Debian project were compromised, and the GNU Project's Savannah server was also broken into recently. Since there has been little information published about the nature of the Savannah compromise, we contacted Bradley Kuhn, executive director of the Free Software Foundation for more information.

Kuhn described the Savannah compromise as "almost identical to what happened to Debian." (A detailed account of the Debian compromise can be found here.) Kuhn said that he believes that the Savannah compromise and the Debian attacks were related, and happened at about the same time. However, he said that the project has not put a great deal of time and effort into analyzing the attacks because it was more important to put Savannah back online and to try to harden the system to see to it that a similar compromise doesn't happen again. The hard drives from Savannah have been saved for future reference, but the project is not putting its efforts into thoroughly analyzing the attacks.

For the most part, Savannah has been restored and changes have been made to try to ensure a similar attack will not be possible. However, there are still some features that remain unavailable, including Web CVS access and new projects are not being approved for the time being. According to the Savannah website, new projects will probably be accepted sometime before the end of January, 2004.

Has there been an attempt to insert a trojan into any of the code residing on Savannah? Kuhn says that they've asked the owners of projects on Savannah to go through and verify the code that is on Savannah to be sure that it hasn't been trojaned. So far, there have been no reports of tainted code. However, not all of the projects have reported their status. Kuhn also noted that projects on the Savannah website will soon have an indicator to report whether or not the developers have verified that they have checked the integrity of their software.

We also asked if there was any sensitive information on Savannah that may have been compromised. Kuhn said that the useful information on Savannah mostly consists of the code for the various projects, and that the only other information of interest would be developers' passwords. The passwords on Savannah have been reset, of course, and the developers have been encouraged to "investigate their own personal security."

For now, the GNU Project is not actively pursuing criminal prosecution of the attacker or attackers. Kuhn says that the project is not "ethically opposed" to prosecuting the intruder, but that with limited resources he'd rather divert time and energy to restoring the services and trying to harden systems to make future attacks more difficult and easier to contain.

To that end, the compromise may actually be a good thing in the long run. Kuhn said that they have contacted the CVS maintainers and have offered to pay for development of features that would allow GPG signing of commits through CVS -- making it much more difficult for changes to be inserted unnoticed into code held in a CVS repository. He said that they have also contacted the GNU Arch maintainer about adding GPG signing. Though it may take some time to develop, the addition of GPG signing to commits would be a welcome feature.

Kuhn said that he expects that the future will bring more attacks on the community, as free and open source software become more prevalent. Opponents of the open development model will no doubt be using these events as an illustration of the "dangers" of open source. Though the recent intrusions have mostly been an inconvenience, it's important that the community learn from these attacks, and redouble efforts to prevent them in the future.


(Log in to post comments)

The Savannah Compromise - what really happened?

Posted Jan 1, 2004 18:47 UTC (Thu) by miah (subscriber, #639) [Link]

Gnu arch now has gpg signature support. I read this in tomlords advogato post yesterday.

The Savannah Compromise - what really happened?

Posted Jan 1, 2004 19:03 UTC (Thu) by miah (subscriber, #639) [Link]

Let me correct this. It was robertc's advogato entry.

"On the GNU Arch front, my gpg signing code is now in mainline, along with enhancements from Tom - grab tla 1.2 if you want it. We support signed changesets, re-signed mirrors, and mirrors that copy the source signatures. AFAIK only monotone has anything equivalent, (although bitkeeper has apparently good integrity mechanisms - itls not signed-by-the-author changesets)."

The Savannah Compromise - what really happened?

Posted Jan 1, 2004 19:55 UTC (Thu) by rao (subscriber, #78) [Link]

The Savannah codebase and infrastructure was audited after the compromise to find potential security holes that the cracker could have used. CVS 1.12.5 and 1.11.11 were released on 2003-12-18 as a direct result of that work. Futher details on CVS will be released in the coming days. Services are being brought back up on Savannah as they are secured. For instance, under the new Savannah setup, each software project's CVS repository resides in its own chroot, and other essential system services also reside in their own chroots. The FSF and Savannah volunteers have taken this compromise very seriously, and we've taken steps to limit the damage from any future compromises.

Paul Fisher
Free Software Foundation

The Savannah Compromise - what really happened?

Posted Jan 1, 2004 20:17 UTC (Thu) by miah (subscriber, #639) [Link]

What has been done to ensure its not possible for future attackers from breaking out of the chroot? Have you looked into propolice(ssp), grsecurity, some sort of mac/acl system?

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 0:16 UTC (Fri) by JoeBuck (subscriber, #2330) [Link]

chroot jails are useless against an attacker who manages to get root from within the jail, so if a local root exploit exists, chroot is no help.

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 4:53 UTC (Fri) by jonabbey (subscriber, #2736) [Link]

Can you explain that? How does one get out of a chroot jail, even with root?

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 5:20 UTC (Fri) by spotter (subscriber, #12199) [Link]

what does root mean? it means that when you do a ".." it goes to "." instead of the parent directory.

now imagine you are able to call chroot, you can change your "root" to a directory below you. now, any directory you are in is not the root, so ".." will go to the parent of that directory instead of going to "." and therefore you have broken out of the chroot.

I've looked at 2 simple ways around this.

1) every process should have a list of chroot/root points (instead of just one) and whene ever you hit one o those points ".."->"." I broached this idea to the l-k list a year or 2 ago, but people weren't really interested. probably have my code for it lying around somewhere.

or

2) have a filesystem that is aware of chroots, and doesn't let a process walk past any chroot point. since file system's don't know about chroot(), would also need to wrap the chroot() syscall in code that set up the appropriate data structures for the fs. this works because even though ".." links to the parent directory, if the filesystem's permission() function prevents any process (even roots) from walking past a directory, the process is effectively chained in. somewhat of a hack, but it works fine, have code that implements this too.

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 8:07 UTC (Fri) by eru (subscriber, #2753) [Link]

How does FreeBSD:s jail(2) do it? I suppose it fixed the chroot escaping problem.

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 16:03 UTC (Fri) by eru (subscriber, #2753) [Link]

Suppose you just changed the kernel to disable the chroot call for every process (even ones with uid 0) whose current root directory differs from the real root? Would that cause problems to any legitimate applications?

The Savannah Compromise - what really happened?

Posted Jan 4, 2004 6:43 UTC (Sun) by Ross (guest, #4065) [Link]

What I wished for was that chroot() would be considered a capability and
that it could be disabled. In fact, capabilities in Linux aren't very
useful because they can only restrict actions that are already reserved
for the superuser. So I can't, for example, say that this process and its
children can't call ptrace(), chroot(), etc.

The Savannah Compromise - what really happened?

Posted Jan 15, 2004 11:51 UTC (Thu) by edmundo (guest, #616) [Link]

Would it help for chroot to take no argument and only chroot to the current directory?

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 5:24 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

if you are root inside a chroot jail and the chroot has access to /proc, or anything in the chroot has access to file handles pointing outside the jail, or the system will honor raw access to a device from within that jail then the attacker has a way out of the jail.

the biggest problem is that even if you don't put any software in the chroot the attacker can install their own so they can then issue the mount command (along with the correct device info) to the kernel and the kernel will allow the access becouse you are root.

useing chroot can't prevent an attacker from getting into a system, but it is one more thing that they need to deal with to really get control of the system (and the more you strip down the chroot sandbox the more work it takes to break out and the less vunerable you are to automated attacks)

The Savannah Compromise - what really happened?

Posted Jan 4, 2004 6:45 UTC (Sun) by Ross (guest, #4065) [Link]

You are assuming the chroot() area is writable and that the compromized
process is running as root (non-root can not call mount(2)).

Breaking out of a chroot jail

Posted Jan 2, 2004 15:36 UTC (Fri) by LogicG8 (guest, #11076) [Link]

I just happened to be reading about this
so I have a really good url handy.
http://www.bpfh.net/simes/computing/chroot-break.html

The Savannah Compromise - what really happened?

Posted Jan 3, 2004 2:13 UTC (Sat) by iabervon (subscriber, #722) [Link]

On the other hand, it'd be very difficult to get root in the jail if
there's nothing setuid root or running as root in the jail. Anything
kernel-level that will give you root in this situation would probably let
you do arbitrary other things anyway, and anything userspace can't give
you root. Tasks requiring root access can be done from outside the jail,
so in-jail root doesn't actually need to be possible at all, which makes
security auditting much simpler, because you can be sure that permissions
will be followed by everything in the jail.

The Savannah Compromise - what really happened?

Posted Jan 4, 2004 6:47 UTC (Sun) by Ross (guest, #4065) [Link]

Yes, exactly. But one must be careful that anything running outside the
chroot()ed area treats anything that is writable in the chroot()ed area as
untrusted. This means being very careful opening files and validating
inputs (it's a lot like handling /tmp correctly... i.e. almost impossible).

The Savannah Compromise - what really happened?

Posted Jan 6, 2004 0:46 UTC (Tue) by JoeBuck (subscriber, #2330) [Link]

But we are talking about actions taken in response to the Debian and Savannah compromises. Given a kernel bug that allows a cleverly written program to get root, if you can execute a program from within the jail and such a flaw exists, you get a get-out-of-jail-free card. Another such bug was just discovered.

So, I repeat: chroot jails are useless if a kernel bug provides a root exploit.

The Savannah Compromise - what really happened?

Posted Jan 8, 2004 23:06 UTC (Thu) by Ross (guest, #4065) [Link]

Oh, yes, I completely agree in that case. But people weren't careful to
limit their arguments to that case. In fact, with a root-exploitable
kernel bug any attacker that can run arbitrary code can access the whole
system, no matter what user they run the code as or what type of jail the
code is put into. In short, having a secure kernel is a requirement for
any other security on the system so long as untrusted users have access to
run code.

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 10:32 UTC (Fri) by lacostej (guest, #2760) [Link]

how can one be sure that the Savannah and Debian attacks were "almost identical" if the Savannah disks were not thoroughly analyzed?

The Savannah Compromise - what really happened?

Posted Jan 2, 2004 12:32 UTC (Fri) by leonb (guest, #3054) [Link]

And what about sourceforge?

Shouldn't we assume that the debian and savannah attackers also have tried to compromise it? Do we have any positive or negative information about this?

- L.

Developers passwords?

Posted Jan 15, 2004 23:31 UTC (Thu) by barrygould (guest, #4774) [Link]

Were the developers passwords stored in a database or something, without using MD5 or equivalent?

Copyright © 2004, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds