|
|
Subscribe / Log in / New account

Numerous Debian Project systems compromised

From:  Martin Schulze <joey-AT-infodrom.org>
To:  Debian Announcements <debian-announce-AT-lists.debian.org>
Subject:  Some Debian Project machines have been compromised
Date:  Fri, 21 Nov 2003 11:46:19 +0100

------------------------------------------------------------------------
The Debian Project                                http://www.debian.org/
Some Debian Project machines compromised                press@debian.org
November 21st, 2003
------------------------------------------------------------------------

Some Debian Project machines have been compromised

This is a very unfortunate incident to report about.  Some Debian
servers were found to have been compromised in the last 24 hours.

The archive is not affected by this compromise!

In particular the following machines have been affected:

  . master (Bug Tracking System)
  . murphy (mailing lists)
  . gluck (web, cvs)
  . klecker (security, non-us, web search, www-master)

Some of these services are currently not available as the machines
undergo close inspection.  Some services have been moved to other
machines (www.debian.org for example).

The security archive will be verified from trusted sources before it
will become available again.

Please note that we have recently prepared a new point release for
Debian GNU/Linux 3.0 (woody), release 3.0r2.  While it has not been
announced yet, it has been pushed to our mirrors already.  The
announcement was scheduled for this morning but had to be postponed.
This update has now been checked and it is not affected by the
compromise.

We apologise for the disruptions of some services over the next few
days.  We are working on restoring the services and verifying the
content of our archives.


Contact Information
-------------------

For further information, please visit the Debian web pages at
<http://www.debian.org/> or contact <press@debian.org>.


-- 
To UNSUBSCRIBE, email to debian-announce-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org



to post comments

GPG signed announce

Posted Nov 21, 2003 15:16 UTC (Fri) by teo (guest, #14575) [Link]

Here is the GPG signed announce from Joey.

It's a sad day for us. The important thing now is to check what happened and to advice our users if it was security exploit.

And if it wasn't, that's what I hope, we've to find out who/what compromised the trust keyring.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 15:30 UTC (Fri) by pointwood (guest, #2814) [Link] (5 responses)

Very sad indeed :(

I can see the reason some people thinks it's cool to hack government and military sites and various companies sites, but hacking/cracking servers like debian servers is something I simply don't understand what the motive is.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 15:42 UTC (Fri) by allesfresser (guest, #216) [Link] (1 responses)

There's lots of people that would love to be able to claim that free software is insecure...

Numerous Debian Project systems compromised

Posted Nov 21, 2003 15:56 UTC (Fri) by XERC (guest, #14626) [Link]

A lot of Micro$oftians....

Numerous Debian Project systems compromised

Posted Nov 21, 2003 15:54 UTC (Fri) by XERC (guest, #14626) [Link] (2 responses)

I agree with you on that one. It's like cuting down trees in the park. I undesrstand, that if the trees were so rotten, that they would be a risk for park visitors and the park maintainer desn't take the trees down by itself, then it would be OK to take them down, silently, violently, in the dark, but did anyone inform the sight administrator before the attack or atleast leave a note?

It's funny, that it's the Debian's server and not some commercial distribution's server. Debian seems to relate to freedom more, that other distros, and has a long and rich history.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 16:20 UTC (Fri) by ccchips (subscriber, #3222) [Link]

Yeah, but what if the people funding you (as a cracker) thought they owned the designs for the nucleic acids that made up those trees, or that their own forests were more immune to the cutting of trees?

When Microsoft first announced, all over the place, in big letters, on their sycophant press web sites, that Linux security wasn't any better than Windows security, I immediately figured some nitwit would try (or be paid to try) some idiot thing like this.

And, by the way, if any of you readers happen to be such nitwits, consider this:

(1) Linux is free and open software. The developers know exactly how to *publicly* fix what caused this, and/or come up with ways to prevent it again.

(2) If you get caught, you can be made to go away for a very long time. And *none* of us will have *any* sympathy for you.

(3) If you were, in fact, paid or funded or helped in any way by a third party, you've become a patsy. How do you feel about that?

Numerous Debian Project systems compromised

Posted Nov 21, 2003 16:24 UTC (Fri) by ccchips (subscriber, #3222) [Link]

Another thought:

There are cases on the books where people committed arson in public forests because they didn't like the idea of publicly-owned land, or didn't like the way it was managed, etc.

Who Is Investigating?

Posted Nov 21, 2003 15:59 UTC (Fri) by rriggs (guest, #11598) [Link] (1 responses)

This is a national and international infrastructure problem. At least some of these servers appear to be hosted in the US. The FBI should be investigating this. Crimes against servers that are located elsewhere in the world and were compromised should be ingestigated by the appropriate local authorities and Interpol.

Who Is Investigating?

Posted Nov 22, 2003 10:59 UTC (Sat) by nix (subscriber, #2304) [Link]

That's a good way of making sure those servers get locked into the limbo of `legal evidence' and not released for years, while not actually finding any of the perpetrators.

Actually fixing the hole (whatever it was) and getting the servers back online is surely more important.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 16:07 UTC (Fri) by elanthis (guest, #6227) [Link] (1 responses)

So, how many people really think Debian is more secure than Fedora now, eh? ;-)

(for the humour impaired - that was a joke)

Numerous Debian Project systems compromised

Posted Nov 21, 2003 16:35 UTC (Fri) by newren (subscriber, #5160) [Link]

Okay, perhaps I'll look like I'm humor impaired, but I'll bite.

I'm reminded of the saying that "Our true intentions often come out in jest." It's nice to see you claim it's a joke, but I don't see how it could be taken as anything other than a slight to Debian. Fedora happens to be my distribution of choice, but I don't see the need to jump on the weaknesses or failings of others. And I'm not at all convinced that this couldn't happen to Fedora.

Personally, I think someone was out to make FLOSS (Free/Libre/Open Source Software) look bad; they want to make it look insecure. This isn't just bad for Debian. This is bad for all of us.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 16:21 UTC (Fri) by maceto (guest, #16498) [Link] (10 responses)

I think it is more a problem with who should have access/ who not than a security exploit. Remember that Debian have 1200+ people working on the project, "it`s open for the public". Suse etc everything is closed, same goes for alot of the other distros.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 17:02 UTC (Fri) by Wummel (guest, #7591) [Link] (6 responses)

Maceto wrote:
> Remember that Debian have 1200+ people working on the project...
Yes, and every Debian developer (including me ;) is at least in theory capable to insert a backdoor in his/her packages. And packages get installed with root permissions.

Due to this big responsibility/power, a new person who wants to become a Debian developer needs not only identity and skill checks, you need now also an existing Debian developer as advocate.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 17:29 UTC (Fri) by eludias (guest, #4058) [Link] (2 responses)

I would advocate a number of enhancements to the Debian process (and other distro's):

- It has become time to have packages which you can install without having root privileges while running the install and deinstall scripts. Only a select number of packages would need them anyway. The only thing you need root priviliges for in a 'normal' package is for extracting your files, which is something which is checkable and containable.
- A network of 'package checkers' should exist, which compile Debian packages on their own and compare the compiled version with compiled versions on the servers. This should be distributed, so everyone who wanted to donate some CPU cycles and bandwith would be able to do so. Some kind of background task like SETI@home which would check a random package per day per computer or so. This would not help against source-code comprises, but it helps against trojaned binaries without knowing how it was trojaned.

Numerous Debian Project systems compromised

Posted Nov 22, 2003 4:28 UTC (Sat) by IkeTo (subscriber, #2122) [Link] (1 responses)

- It has become time to have packages which you can install without having root privileges while running the install and deinstall scripts. Only a select number of packages would need them anyway. The only thing you need root priviliges for in a 'normal' package is for extracting your files, which is something which is checkable and containable.

You have no idea of what a package actually does. Most packages must be installed as root. E.g., libraries must install to one of the standard places or they won't be used at all and programs using them will vomit with a message saying library not found. Many packages needs to be able to write to /etc and /usr/share so that they can install the configuration files and data files. Many needs to run daemon processes. If an individual user needs to install something, he can get the package, unpack it himself, put the binary into somewhere he wants, and hope it work. But most likely, it won't. It is more likely than not that at the end one needs a re-compile to change the locations where configuration and data files are to be found. Of course, binary packages are meant to be compiled, and so it means that it is not something that is supposed to be done by the package anyway.

- A network of 'package checkers' should exist, which compile Debian packages on their own and compare the compiled version with compiled versions on the servers. This should be distributed, so everyone who wanted to donate some CPU cycles and bandwith would be able to do so. Some kind of background task like SETI@home which would check a random package per day per computer or so. This would not help against source-code comprises, but it helps against trojaned binaries without knowing how it was trojaned.

Did you even try to compile two programs in two individually configured computers and compare them? If you did so, you should know that they won't be the same: anything from compiler version to compiler configuration to library version to date of compile will make the actual binary file different. Trojaned binaries can already be checked by using the MD5 checksum now. Your approach would actually help, in the case that the compilation process in the Debian Developer is compromised so that sane source file produces binary files with a trajon horse. But then, if the Debian developer's computer is compromised, I think nothing can be guaranteed anyway.

Numerous Debian Project systems compromised

Posted Nov 22, 2003 9:44 UTC (Sat) by NAR (subscriber, #1313) [Link]

You have no idea of what a package actually does. Most packages must be installed as root.

I do have an a idea what a package actually does, altough my experience is with Solaris packages - however, the ideas are the same. In Solaris, the installation of a package goes through 5 distinct phases:

  1. request: the package asks the user various questions (e.g. the port where the installed daemon should listen).
  2. checkinstall: checks if the system is ready to install the package. In our project a "security daemon" supplies the username/password (used to login to the database, etc.) for the other applications. We use this during install so we have to check whether this daemon runs or not.
  3. preinstall: make some modifications to the system before the files of the package are copied to the system.
  4. the actual copying of the files (this also sets the rights and ownership of the files).
  5. postinstall: make some modification on the installed package (e.g. write the config variables got during request into the config file).

On Solaris, the checkinstall phase runs as nobody user. It sounds a good idea that the other phases should run also as nobody, however, it caused us some problems - the above mentioned "security daemon" accepts connections only from specific users, so there are a lots of "su" commands in our install code and we couldn't place all of our checks into the checkinstall phase :-( I think that in this case one size does not fit all - for some packages it might be feasible that only the actual copying runs as root (e.g. if the package installs under /usr). For some packages even this might not bee needed (I saw a system where a "program installer" user had write rights on /usr/local - he only installed programs from source, but this technique might be applied to binary packages). But there are packages where more phases should run as root.

Bye,NAR

Numerous Debian Project systems compromised

Posted Nov 21, 2003 19:00 UTC (Fri) by JoeBuck (subscriber, #2330) [Link] (2 responses)

Unfortunately, based on similar problems at the FSF, it is quite possible that this is a "inside job" (that one of the 1200+ people with developer access might have done the crack). Having people vouch for each other doesn't prevent this. There was a guy I knew once that I would have trusted with my life, and then I found out that he stole checks from another friend who trusted him to house-sit, forged the signature and stole a significant amount of money.

Having a bad seed in the Debian Project is a scary thought: all these folks can upload packages, and each package is installed as root and can therefore do pretty much whatever it wants. The Debian policies concerning digital signatures probably deter most temptations in this area, because the bad guy's signature would be attached to the bad code. Still, it's worrying.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 23:19 UTC (Fri) by Ross (guest, #4065) [Link]

This is interesting. I would like to know what mechanism was used to
either gain unauthorized remote access to the system or to escalate from
an authorized level of access to an unauthorized one.

Is it something that can prevented by better hardening of the servers?
Was it due to unapplied patches or misconfiguration? Bad passwords?
GPG bugs? Compromized systems which were trusted by the server?

I also wonder if there is any connection with the recent modification of
the Linux CVS gateway.

I think we can better protect ourselves in the future if we understand
how these attacks are being perpetrated.

Numerous Debian Project systems compromised

Posted Nov 22, 2003 16:49 UTC (Sat) by ccchips (subscriber, #3222) [Link]

This is interesting. I have posted comments before on this site in which I mention the word "betrayal."

It doesn't hurt the cause if people around the world, especially those who don't know the technical ins and outs of computers, could understand the nature of this beast in our field.

People who advocate more liberal drug laws are familiar with it since day one. There was a time, in the 1970's, when it actually became quite fashionable to spurn informants and people who betrayed one's trust. In our industry, however, we are currently at a point where we are still quite able to identify, and *expose*, people who are willing to sell out to the likes of Microsoft and SCO by engaging in clearly criminal behavior.

It was said earlier that bringing in law-enforcement could mean that the servers would be impounded, but there must be some better way to do it. People who violate the law by breaking into the property of others should be punished, and that's that.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 18:46 UTC (Fri) by piman (guest, #8957) [Link]

> Debian have 1200+ people working on the project, "it`s open for the public".

Indeed, and every member of that "public" requires photo identification, often a personal meeting with an existing developer, a GPG signature, and an official advocate who is already a developer.

If a developer him or herself does indeed trojan the system (unlikely -- most crackers don't have that kind of patience, especially for real-life interactions), there will almost certainly be a trail back to that person.

Numerous Debian Project systems compromised

Posted Nov 21, 2003 21:01 UTC (Fri) by NAR (subscriber, #1313) [Link] (1 responses)

I don't know the system the Debian project uses to authenticate, etc. but the fact that there is 1200+ contributors doesn't mean that it's enough to break into one computer used by one of 1200+ contributors to wreak havoc in Debian?

Bye,NAR

Numerous Debian Project systems compromised

Posted Nov 21, 2003 21:10 UTC (Fri) by piman (guest, #8957) [Link]

(Long, random) LDAP passwords and SSH, and package uploads must be signed by GPG keys. Passwords are only emailed encrypted. Even if you have a developer's compromised home system, it's pretty hard to get things uploaded. And most developers don't have root or extra-privileged access to the machines.

Note also that auric, the main archive server, was not compromised at all.

Numerous Debian Project systems compromised

Posted Nov 23, 2003 12:20 UTC (Sun) by XERC (guest, #14626) [Link]

Well, but as with any cracking or hacking, it's clear, that IT HAD A SECURITY HOLE, and that fact doesn't depend on the party or person, who states that, even, if that party were Micro$oft.

Actually, what conserns to the technical aspects, then I suggest, that the first thing, that could be done in Linux in general, is to start using some of the OpenBSD packages and libraries. "Search/replace_all" the dangerous and historic C functions(like strcpy, etc.) with newer and safer ones, start using randmomized process ID's(may be Debian already uses those??), "chroot" the webserver's process to it's
own directory, stop using /tmp as a place for temporary files(use /tmp/usrname with chmod 0660), make as many operations as monolitic as possible,
check the operations return value, when creating files, make sure, that, when something is writeable, then it's not executeable, and vice versa, don't use easily predictable filenames for temporary files(date, proccessID, userID, etc), filter the strings against string-attacks(a string, that is written by overflow and can be intepretad as a machine code), pay attantion to double-intepretation(for instance,
PHP forbidden, CGI writes PHP code which get's then executed as PHP), etc.

All of the previous is doable withought manualy touching much of the existing code. OK, after that, well take a look at social engineering and may be Linux developers should more closely cooperate with OpenBSD developers when looking at the code from security's point of view. And again, "copy/paste" from OpenBSD as much as possible, specially those parts, that are at relatively mature states.

Hey, but why couldn't Debian be hosted on the
OpneBSD server? It's just a plain server, it doesn't have to be at such a
bleeding age? I think that it would be pretty OK, if OpenBSD were the final, robust, endproduct and various Linux distros were the development plathforms, where all new and exciting is tried and developed.


Numerous Debian Project systems compromised

Posted Nov 24, 2003 15:27 UTC (Mon) by otro_mas (guest, #6820) [Link]

It's interesting to see the kernel attempt, the Debian attempt, and this paper[1] from Microsoft, all three in a quarter.

[1] http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/msit/security/mssecbp.asp


Copyright © 2003, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds