Debian.org compromise update

[Posted November 30, 2003 by corbet]

From:		James Troup <james-AT-nocrew.org>
To:		debian-devel-announce-AT-lists.debian.org
Subject:		more details on the recent compromise of debian.org machines
Date:		Fri, 28 Nov 2003 01:04:00 +0000

Hi,

*NB* bear in mind that:
  a) the information on the break-in in comes from compromised machines
     and thus has to be taken with appropriate skepticism.
  b) the investigation is still ongoing - as I was writing this draft
     further information came to light which may invalidate a lot of
     it.  [Or not - as it turns out].

                              Detection
                              ---------

On November 20 it was noticed that master was kernel oops-ing
lots. While investigating this it was discovered that murphy was
showing the exact same oops, which was an overly suspicious
coincidence.  Also klecker, murphy and gluck have aide installed to
monitor filesystem changes and at around the same time it started
warning that /sbin/init had been replaced and that the mtime and ctime
timestamps for /usr/lib/locale/en_US had changed.

Investigation revealed the cause for both these things to be the
suckit root kit (see the "Suckit" appendix for more info).


                            What happened?
                            --------------

On Wednesday 19th November (2003), at approximately 5pm GMT, a sniffed
password was used to access an (unprivileged) account on
klecker.debian.org.  Somehow they got root on klecker and installed
suckit.  The same account was then used to log into master and gain
root (and install suckit) there too.  They then tried to get to murphy
with the same account.  This failed because murphy is a restricted box
that only a small subset of developers can log into.  They then used
their root access on master to access to an administrative account
used for backup purposes and used that to gain access to Murphy.  They
got root on murphy and installed Suckit there too.  The next day they
used a password sniffed on master to login into gluck, got root there
and installed suckit.

See the "Time-line" appendix for more details on times.

                               Response
                               --------

Gluck was powered down and an image has been made of it's disks for
forensic analysis.

Since we didn't have direct physical access to klecker it's Internet
connection was shut down and disk images were made via serial console
to a local machine on a firewalled net connection.

master and murphy were kept running for a short while in order to make
an announcement of the compromise, after which they were also taken
off-line and imaged.

                               Cleanup
                               -------

After a thorough cleanup and reinstall of modified files the non-US and
security archives were verified by looking at mirror logs for changes and
comparing MD5 checksums of the files on Klecker and those on three
different trusted mirrors.

Gluck, Master and Murphy were wiped and reinstalled from CD.  data and
services are in the process of being restored.

All machines and data were checked for devices outside of /dev, suid
executables, writable files, etc. and all suspicious files were
removed.  Services (and their scripts/programs) are being compared to
known-good sources and sanity checked before being re-enabled.

Since we now knew we had compromised accounts and sniffers on our
hands we had to assume that that an unknown number of accounts were
now compromised, so all accounts were locked, passwords invalidated
and ssh authorised keys removed.

                        How could this happen?
                        ----------------------

All the compromised machines were running recent kernels[1] and were
up-to-date with almost all security updates[2].

However there was two problems.

(1) The kernels running on the machines in question didn't all get a
    ptrace fixed kernel as fast one might have liked.  Master, Klecker
    and Murphy got new kernels in May but Gluck for various reasons
    didn't get upgraded till August (although I believe it had
    /proc/sys/kernel/modprobe fixed to at least block the most common
    exploit before that).

(2) Master had a copy it's old harddrive still lying around by
    accident.  Unfortunately it had a lot of old, unpatched suid
    binaries on it.

Although these could have been the attack vector, I don't believe they
were.  (2) seems unlikely simply because master wasn't, AFAWK, the
first host compromised.  Although it's possible an attacker with local
access to gluck got root through (1), it seems unlikely they'd sit on
that for <n> months and then use it on several machines only to
comeback and rootkit several debian.org machines and at least one
(that we know of) other unrelated system at the same time (and which
didn't have an extended ptrace vulnerability exposure.)

Based on that and the forensics on the unrelated system mentioned
above, I believe that there was an as of yet unknown local root
exploit used to go from having local unprivileged access to having
root.

                        Where do we go from here?
                        -------------------------

Unfortunately due to the fact there is (I believe) an unknown local
root exploit in the wild, we can't yet unlock the Debian accounts.
Obviously we can't continue without LDAP accounts for very long
either.  At the moment I'd ask for a little more patience both a)
while the painful and painstaking task of restoring machines one by
one is completed and b) while we try and exhaust all reasonable
avenues of investigation to determine how the attacker went from
unprivileged to root.

Obviously we're looking at hardening our boxes and tightening up our
procedures to try and stop this from happening again.  I'll send more
details on that later.

			       Finally
			       -------

Developers worried about their own machines might like to have a look
at:

  http://www.wiggy.net/debian/developer-securing/

================================================================================

                              Appendices
                              ~~~~~~~~~~

Thanks
------

 o Adam Heath and Brian Wolfe for their work on master & murphy.
 o Wichert Akkerman for his work on klecker.
 o Dann Frazier and Matt Taggart for their work on gluck.
 o Michael Stone and Robert van der Meulen for their forensics work.
 o Jaakko Niemi for his work on checking and re-enabling lists.debian.org.
 o Colin Watson for his work on checking and re-enabling bugs.debian.org.
 o Josip Rodin for his work on checking and re-enabling the lists web archives.

[This text is based on a draft by Wichert Akkerman.]

========================================

Time-line
---------

All times in GMT.

o Klecker init timestamp: Nov 19 17:08
o Master sk timestamp: Nov 19 17:47
o Murphy sk timestamp: Nov 19 18:35
o Oopses on Murphy start: Nov 19 19:25
o Oopses on Master start: Nov 20 05:38
o Gluck init timestamp: Nov 20 20:54

========================================

Suckit
------

Suckit is a rootkit which installs a sniffer, a process hider, a file
hider and a backdoor login in a running kernel. Apparently there was a
flaw in its kernel code which caused the kernel to oops on master and
murphy. This also explained why /sbin/init was replaced: the new init
loads suckit into the kernel and then proceeds to start the real init,
making sure that it is still active after a reboot.

========================================

Footnotes:

[1] Klecker: 2.4.22, Master & Murphy: 2.4.21-rc2, Gluck: 2.4.22rc2

[2] klecker was missing the latest postgresql updated.  ssh on all
    machines was a DSA-customized version which was missing only the
    3rd and final round (i.e. Solar Designer's patches) of ssh
    updates.

========================================

P.S. As always, I speak only for myself.

-- 
James


-- 
To UNSUBSCRIBE, email to debian-devel-announce-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

breakin

Posted Nov 30, 2003 17:21 UTC (Sun) by ccyoung (guest, #16340) [Link] (1 responses)

could the compromise have occurred earlier and only exploited recently? eg, someone could have been using a root password mined six months ago.

breakin??, in the minds...

Posted Dec 1, 2003 17:43 UTC (Mon) by mmarq (guest, #2332) [Link]

Was it really a breakin ??

After all, do to the extensive distributed nature of those machines, inherent to the development model they contain, the simple fact that someone could had telled a friend, selled, have being spyed on or had used its own password...

So, unless prove otherwise, i'm convinced that there wasent no unknowned exploit involved...

F??k the theorys, Linux/FLOSS is changing the world, and you wouldnt expect that the "powers that are", will sit comfortably and watch, while their most important points of world information, control and command, the IT software infrastructure, gets in a state of public control(where they should always had been)...

(If someone thinks the "powers" will sit tight, that person deserves the "Stupidiest Theory" award of the millenium)

This is not a big deal, unless the community chooses so. It only reminds "us" how important THURST is... (3 points for M$!!... isnt that odd!!??)...

As in case of hardware drivers, if there is a "technical" solution
for all the important problems, then Linux/FLOSS will win for sure.

local root ~= remote root

Posted Nov 30, 2003 19:22 UTC (Sun) by ncm (guest, #165) [Link] (5 responses)

This is a good demonstration that the distinction always made between local privilege-elevation bugs and remote exploits is academic hair-splitting. It's rarely difficult to get unprivileged access through a buggy non-privileged service. (Web-server plug-ins are a reliable source of entry points.) Once you're in, privilege elevation takes you the rest of the way.

Certainly the distinction is useful to security students and analysts, but it's misleading for everybody else. "Oh, that one's just a local exploit; not so bad." The OpenBSD advocates promote the fallacy: "only one remote exploit in this millennium!" (or something like that), encouraging us to ignore almost equally damaging exploits in non-core services that provide access to local accounts and more damaging attacks.

There's a similar fallacy in distinguishing security holes from other bugs. Without a depth of analysis that hardly anybody can ever afford, almost any bug might actually be a security hole, too. The OpenBSD people get this one right -- to them, any bug is a security hole until proven otherwise, and they encourage running latest versions -- but almost everybody else gets it wrong. When I fixed a double-free segfault in lib[mumble], nobody posted security warnings about every program that relies on it. despite that double-free bugs can often be exploited.

Debian gets this wrong, and very selectively backports only proven security holes, ignoring the myriad bugfixes that might just as easily be security holes as well. To find holes in stable-branch services, just look for bug fixes in later versions, particularly in libraries used by those services. Failing that, look at new features added shortly before the library-version used. Chances are the last new feature added has bugs that haven't been noted yet, and that might be exploitable.

This might be a good place to mention that the CVS codebase is almost irreparably insecure. The practical implications are: (1) A remotely-accessible CVS server should never be run on a host that does anything else that matters, or that has access to anything else; (2) An anonymous CVS server should never be the same CVS server that is used for checkins, or even run on the same machine. The pserver should be a slave that only gets read access to a copy of the archive. (3) Checkins on remotely-accessible servers should result in patches logged to another archive kept on another, not-remotely-accessible machine. Patches from that server should be posted to the mailing list.

local root ~= remote root

Posted Nov 30, 2003 20:29 UTC (Sun) by mikesalib (guest, #17162) [Link] (1 responses)

While I agree with a lot of what you say, I think you may be oversimplifying the situation.

For most machines, the distinction between local and remote root exploits is far from academic. That is because most machines do not offer meaningful unprivelaged access to the wide area net. That includes most small group servers and almost all desktops. For that class of machines, a remote root vulnerability is a very real hazard while a local root hole is much less serious because only authorized users can get into the system anyway.

My desktop is a classic example; the only people who have access to this machine are myself and one or two trusted friends. A remote root hole can take my machine down or turn it into a worm or spam spreading zombie so I pay attention to those. But I rarely worry about local exploits because I don't plan on exploiting my own box anytime soon.

The truth is that most computers in the world are not used for classic Unix work where they have 200 relatively untrusted people logged in remotely doing all sorts of work. For such machines, there are lots of unprivelaged access paths and the distinction between local and remote root exploits is pretty silly. The debian servers definitely fall into that category. But it is important to understand that most computers on the net don't fall into that category.

As for Debian's refusal to port non-security bug fixes into their stable distribution, I think they've made the right choice. Every change carries with it the possibility of introducing new bugs, bugs that can themselves present new vulnrabilities. While some projects maintain bug fix only branches for stable releases, for other projects, the only way to get at some bugfixes is to update to the development version.

Bugfixing verification and testing takes a lot of time and effort, especially when you have to support 12+ architectures. Because Debian only pushes security critical updates, administrators can feel a lot more confident updating production machines without having to worry about an unecessary change breaking a system that does real work. Yes, if your machine is compromised you can't work, but if you're constantly breaking a working production system applying unneeded bug fixes that change functionality, you're not much better off. Either way you have unplanned outages.

Debian made a judegement call on where they want to direct their limited (read: volunteer) resources and it looks like the right call to me.

local root ~= remote root

Posted Dec 1, 2003 7:39 UTC (Mon) by ken (subscriber, #625) [Link]

---
My desktop is a classic example; the only people who have access to this machine are myself and one or two trusted friends. A remote root hole can take my machine down or turn it into a worm or spam spreading zombie so I pay attention to those. But I rarely worry about local exploits because I don't plan on exploiting my own box anytime soon.
---

You are very much mistaken here every bug is a potetial problem even for you.

Take a bug in some media codec for xmms. Now someone could infect your computer by making you play some mp3,ogg(whatever) file.

You must realise that any program that takes external data could be used to gain access to you computer it dose not have to be suid or a program litening to the network and once they got your account they hardly even need a local ecploit to gain root. A simple redirect on sudo or su would do it for most home computers.

Still a remote hole is a big problem as the rate of infection is so great something that a local exploit is not.

local root ~= remote root

Posted Nov 30, 2003 21:36 UTC (Sun) by iabervon (subscriber, #722) [Link]

In some cases, the security issue is more remote user-level exploits;
it's reasonably likely that on a desktop, most of the significant
information could be read without priviledge escalation, and the box is
secured to avoid remote access to the machine at all. In fact, remote
access in general may be a narrower window to secure.

Of course, local root exploits are still significant, because more damage
may be caused by an attacker if they are available. Furthermore, it's
very hard to ensure the security of a remotely accessible user account,
so local root exploits only don't matter if there is no possibility of a
non-local user of any sort.

local root ~= remote root (it doesn't have to be)

Posted Nov 30, 2003 22:49 UTC (Sun) by Ross (guest, #4065) [Link]

I agree with what you are saying in that people generally discount
local exploits as unimportant when they really are. However I disagree
that they all but guarantee remote root access.

Personally I run all services as non-root users chroot()ed to a directory
with no executables or libraries and which is mounted with nosuid and
nodev and no proc mounted. (I wish that the kernel provided a way to
remove process capabilities, but it only allows to you give portions of
root access to non-root users. I know I could use LSM but that's getting
overly complicated.)

(That's not to say I don't sometimes take additional precautions. For
example my firewall has no suid or sgid executables.)

Except for kernel bugs I find it unlikely that a remote user could
exploit a buggy service running as a non-root user in order to obtain
root access -- even with buggy userspace binaries installed.

local root ~= remote root

Posted Dec 1, 2003 19:06 UTC (Mon) by job (guest, #670) [Link]

I disagree. If you properly ACL your system with SELinux or its likes, the
privilege escalation part becomes extremely difficult.

Debian.org compromise update

Posted Nov 30, 2003 22:44 UTC (Sun) by maceto (guest, #16498) [Link] (2 responses)

I think maby Debian should be even more secure ( it is more secure out of the box than most linux distros), what I mean is xinetd as default and with no services running, make shure everyone know what tripwire is, tara (http://www-arc.com/tara/) might also be something to look into ( should be a must read for admins. Lids (lids.org) should aslo be an option, sudo can be used as well instead of lids.

What I am saying here is that Debian, is secure, but if these servers were locked down properly and with some more security addons, it would not matter if one had a password etc, cause if one used SECURE REMOTE PASSWORDS (SRP- http:srp.stanford.edu) and some of the stuff over one would have: a box were almost every dir/file is locked, a reboot have to be done to get access to these files, and with a lilo/grub pass that would be a little harder to overcome. And if one get the root pass, there is still the lids to overcome (encrypted that one too)

1. SRP- well sniffing/spofing: a little harder

2.Tripwire: file integrity can be run from anoter server weekly

3. Tara checks that stuff are setup right

4. Lids makes files no write even for root, it can detect scans/port scanners and inform via e-mail

Ok makes the server somewhat harder to setup/ get working- but hell lot more secure.

Debian.org compromise update

Posted Dec 1, 2003 9:13 UTC (Mon) by jfs (guest, #7140) [Link] (1 responses)

Almost everything you describe here is already documented in the Securing Debian Manual. Debian does not only provide Tripwire, but a myriad other file integrity checkers so that people can chose whatever they see fit: aide, samhain, integrit...

As for Tara, Debian provides Tiger 3.2.1, which includes all the checks and features that Tara provides and many more. Notice that Tiger is no longer orphaned and is actively maintained in a Savannah project. The lead developer (me) is also the Debian package maintainer.

Probably the only thing missing yet in order for admins to setup a secure server is an easier way to provide Linux kernels already patched with security in mind, currently we provide a set of kernel patches for security and even if it's easy for most admins to patch the kernel (if using make-kpkg which is provided by kernel-package) it's not as direct as it should be. We're working on it, both in the user-awareness level and in providing improved ways to do this.

Debian.org compromise update

Posted Dec 1, 2003 12:23 UTC (Mon) by maceto (guest, #16498) [Link]

HI, did not know that Tiger was back :-) It was not meant to flame debian in any way- best around!

What I cannot understand is why these servers were cracked!

Cause if setup with most of these kernel patches for security it would be hard if not even impossible- there is a SE-linux debian server running, and the person who have this invites people to try to crack it, and to my knowlegde it not done yet.

I donm know how these servers were setup, but my guess the did not include alot of these kernel patches. And a solution as you mention is maby to include this in a better way.

Would it not be possible to make a "harden Debian" out of the box, with a prebuilt kernel and kernel sources, this harden would be perfect for servers ( can`t be to hard -stuff gotta run on it to), but a set of tools wich would include the most to get started. And some docs wich are just for that harden version- wich explain in detail what is done, how stuff work,how to turn off.

Which remote services please? [Debian.org compromise ...]

Posted Dec 1, 2003 14:15 UTC (Mon) by Wills (guest, #1813) [Link] (4 responses)

It seems likely, as Debian implies, the initial compromise was via an unspecified remote service but what's interesting about Debian's response is that they didn't list in detail which remote services they were originally running on their compromised machines. Instead they speculated about various possible local exploits to root. Yet the number of possibly exploitable remote services on any given machine is relatively small compared to the thousands of possibly exploitable local services, by which I mean all interfaces including the kernel, glibc, Xfree86, etc.

The remote services are the supposedly locked front-door to a machine. Once someone has managed to break through one of the remote services onto a machine, even as a non-root user, there are likely hundreds of unknown local exploits from user to root in any Linux distribution including Debian. It should be easier to find and fix exploits in a small number of remote services than in that typically huge number of local services. I wish Debian would publish the list of remote services they were running so the community can study those services such as sshd and openssl again very closely for possible remote exploits.

Which remote services please? [Debian.org compromise ...]

Posted Dec 1, 2003 15:12 UTC (Mon) by mh (guest, #7058) [Link] (3 responses)

Read the article again:

On Wednesday 19th November (2003), at approximately 5pm GMT, a sniffed
password was used to access an (unprivileged) account on
klecker.debian.org.

This probably means ssh access.

Sniffed password

Posted Dec 1, 2003 18:50 UTC (Mon) by ncm (guest, #165) [Link] (1 responses)

In other words, somebody broke into some developer's machine and recorded a user there logging into the Debian server. Since we don't know whose computer was broken into before the attack on the Debian server, there's no reason to expect the attacker won't sniff the password and break in again. Since the attacker can sniff keystrokes, he can also alter source code, sign packages, or any number of much more subtle things than just installing rootkits.

There's no compelling reason to believe it hasn't happened already.

I find it odd that the announcement didn't include a request to scour every host that developers have logged in from. Maybe that indicates they know whose it was.

Sniffed password

Posted Dec 1, 2003 21:12 UTC (Mon) by ballombe (subscriber, #9523) [Link]

I find it odd that the announcement didn't include a request to scour every host that developers have logged in from. Maybe that indicates they know whose it was.

They have, please read http://www.wiggy.net/debian/developer-securing/ .

Also I believe they know who he was since they mentioned a failed login attempt to murphy.

Which remote services please? [Debian.org compromise ...]

Posted Dec 5, 2003 13:33 UTC (Fri) by Wills (guest, #1813) [Link]

Yes, I read the mention of password sniffing but I must admit I thought it was too incredible to be the real reason (not being disclosed by Debian) because Debian is supposed to take security seriously and they would neither allow any developer to login to important Debian servers from insecure/untrusted PCs including public internet terminals nor allow untrusted people to have physical access to their key servers. If a Debian server password was really sniffed, one of Debian's tusted developers must have used an insecure, untrusted PC, which had a sniffer installed, to login to a trusted Debian server -- very poor security practice. If you value the security of a remote server, never login to it except from a PC which is kept as secure in every way as the server.