Numerous Debian Project systems compromised
| From: | Martin Schulze <joey-AT-infodrom.org> | |
| To: | Debian Announcements <debian-announce-AT-lists.debian.org> | |
| Subject: | Some Debian Project machines have been compromised | |
| Date: | Fri, 21 Nov 2003 11:46:19 +0100 |
------------------------------------------------------------------------ The Debian Project http://www.debian.org/ Some Debian Project machines compromised press@debian.org November 21st, 2003 ------------------------------------------------------------------------ Some Debian Project machines have been compromised This is a very unfortunate incident to report about. Some Debian servers were found to have been compromised in the last 24 hours. The archive is not affected by this compromise! In particular the following machines have been affected: . master (Bug Tracking System) . murphy (mailing lists) . gluck (web, cvs) . klecker (security, non-us, web search, www-master) Some of these services are currently not available as the machines undergo close inspection. Some services have been moved to other machines (www.debian.org for example). The security archive will be verified from trusted sources before it will become available again. Please note that we have recently prepared a new point release for Debian GNU/Linux 3.0 (woody), release 3.0r2. While it has not been announced yet, it has been pushed to our mirrors already. The announcement was scheduled for this morning but had to be postponed. This update has now been checked and it is not affected by the compromise. We apologise for the disruptions of some services over the next few days. We are working on restoring the services and verifying the content of our archives. Contact Information ------------------- For further information, please visit the Debian web pages at <http://www.debian.org/> or contact <press@debian.org>. -- To UNSUBSCRIBE, email to debian-announce-request@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Posted Nov 21, 2003 15:16 UTC (Fri)
by teo (guest, #14575)
[Link]
Here is the GPG signed announce from Joey.
It's a sad day for us. The important thing now is to check what happened and to advice our users if it was security exploit.
And if it wasn't, that's what I hope, we've to find out who/what compromised the trust keyring.
Posted Nov 21, 2003 15:30 UTC (Fri)
by pointwood (guest, #2814)
[Link] (5 responses)
I can see the reason some people thinks it's cool to hack government and military sites and various companies sites, but hacking/cracking servers like debian servers is something I simply don't understand what the motive is.
Posted Nov 21, 2003 15:42 UTC (Fri)
by allesfresser (guest, #216)
[Link] (1 responses)
Posted Nov 21, 2003 15:56 UTC (Fri)
by XERC (guest, #14626)
[Link]
Posted Nov 21, 2003 15:54 UTC (Fri)
by XERC (guest, #14626)
[Link] (2 responses)
It's funny, that it's the Debian's server and not some commercial distribution's server. Debian seems to relate to freedom more, that other distros, and has a long and rich history.
Posted Nov 21, 2003 16:20 UTC (Fri)
by ccchips (subscriber, #3222)
[Link]
When Microsoft first announced, all over the place, in big letters, on their sycophant press web sites, that Linux security wasn't any better than Windows security, I immediately figured some nitwit would try (or be paid to try) some idiot thing like this. And, by the way, if any of you readers happen to be such nitwits, consider this: (1) Linux is free and open software. The developers know exactly how to *publicly* fix what caused this, and/or come up with ways to prevent it again. (2) If you get caught, you can be made to go away for a very long time. And *none* of us will have *any* sympathy for you. (3) If you were, in fact, paid or funded or helped in any way by a third party, you've become a patsy. How do you feel about that?
Posted Nov 21, 2003 16:24 UTC (Fri)
by ccchips (subscriber, #3222)
[Link]
There are cases on the books where people committed arson in public forests because they didn't like the idea of publicly-owned land, or didn't like the way it was managed, etc.
Posted Nov 21, 2003 15:59 UTC (Fri)
by rriggs (guest, #11598)
[Link] (1 responses)
Posted Nov 22, 2003 10:59 UTC (Sat)
by nix (subscriber, #2304)
[Link]
Actually fixing the hole (whatever it was) and getting the servers back online is surely more important.
Posted Nov 21, 2003 16:07 UTC (Fri)
by elanthis (guest, #6227)
[Link] (1 responses)
(for the humour impaired - that was a joke)
Posted Nov 21, 2003 16:35 UTC (Fri)
by newren (subscriber, #5160)
[Link]
I'm reminded of the saying that "Our true intentions often come out in jest." It's nice to see you claim it's a joke, but I don't see how it could be taken as anything other than a slight to Debian. Fedora happens to be my distribution of choice, but I don't see the need to jump on the weaknesses or failings of others. And I'm not at all convinced that this couldn't happen to Fedora. Personally, I think someone was out to make FLOSS (Free/Libre/Open Source Software) look bad; they want to make it look insecure. This isn't just bad for Debian. This is bad for all of us.
Posted Nov 21, 2003 16:21 UTC (Fri)
by maceto (guest, #16498)
[Link] (10 responses)
Posted Nov 21, 2003 17:02 UTC (Fri)
by Wummel (guest, #7591)
[Link] (6 responses)
Due to this big responsibility/power, a new person who wants to become a Debian developer needs not only identity and skill checks, you need now also an existing Debian developer as advocate.
Posted Nov 21, 2003 17:29 UTC (Fri)
by eludias (guest, #4058)
[Link] (2 responses)
- It has become time to have packages which you can install without having root privileges while running the install and deinstall scripts. Only a select number of packages would need them anyway. The only thing you need root priviliges for in a 'normal' package is for extracting your files, which is something which is checkable and containable.
Posted Nov 22, 2003 4:28 UTC (Sat)
by IkeTo (subscriber, #2122)
[Link] (1 responses)
You have no idea of what a package actually does. Most packages must be installed as root. E.g., libraries must install to one of the standard places or they won't be used at all and programs using them will vomit with a message saying library not found. Many packages needs to be able to write to /etc and /usr/share so that they can install the configuration files and data files. Many needs to run daemon processes. If an individual user needs to install something, he can get the package, unpack it himself, put the binary into somewhere he wants, and hope it work. But most likely, it won't. It is more likely than not that at the end one needs a re-compile to change the locations where configuration and data files are to be found. Of course, binary packages are meant to be compiled, and so it means that it is not something that is supposed to be done by the package anyway. Did you even try to compile two programs in two individually configured computers and compare them? If you did so, you should know that they won't be the same: anything from compiler version to compiler configuration to library version to date of compile will make the actual binary file different. Trojaned binaries can already be checked by using the MD5 checksum now. Your approach would actually help, in the case that the compilation process in the Debian Developer is compromised so that sane source file produces binary files with a trajon horse. But then, if the Debian developer's computer is compromised, I think nothing can be guaranteed anyway.
Posted Nov 22, 2003 9:44 UTC (Sat)
by NAR (subscriber, #1313)
[Link]
I do have an a idea what a package actually does, altough my experience is with Solaris packages - however, the ideas are the same. In Solaris, the installation of a package goes through 5 distinct phases:
On Solaris, the checkinstall phase runs as nobody user. It sounds a good idea that the other phases should run also as nobody, however, it caused us some problems - the above mentioned "security daemon" accepts connections only from specific users, so there are a lots of "su" commands in our install code and we couldn't place all of our checks into the checkinstall phase :-( I think that in this case one size does not fit all - for some packages it might be feasible that only the actual copying runs as root (e.g. if the package installs under /usr). For some packages even this might not bee needed (I saw a system where a "program installer" user had write rights on /usr/local - he only installed programs from source, but this technique might be applied to binary packages). But there are packages where more phases should run as root.
Posted Nov 21, 2003 19:00 UTC (Fri)
by JoeBuck (subscriber, #2330)
[Link] (2 responses)
Unfortunately, based on similar problems at the FSF, it is quite possible that this is a "inside job" (that one of the 1200+ people with developer access might have done the crack). Having people vouch for each other doesn't prevent this. There was a guy I knew once that I would have trusted with my life, and then I found out that he stole checks from another friend who trusted him to house-sit, forged the signature and stole a significant amount of money.
Having a bad seed in the Debian Project is a scary thought: all these folks can upload packages, and each package is installed as root and can therefore do pretty much whatever it wants. The Debian policies concerning digital signatures probably deter most temptations in this area, because the bad guy's signature would be attached to the bad code. Still, it's worrying.
Posted Nov 21, 2003 23:19 UTC (Fri)
by Ross (guest, #4065)
[Link]
Is it something that can prevented by better hardening of the servers? I also wonder if there is any connection with the recent modification of I think we can better protect ourselves in the future if we understand
Posted Nov 22, 2003 16:49 UTC (Sat)
by ccchips (subscriber, #3222)
[Link]
It doesn't hurt the cause if people around the world, especially those who don't know the technical ins and outs of computers, could understand the nature of this beast in our field. People who advocate more liberal drug laws are familiar with it since day one. There was a time, in the 1970's, when it actually became quite fashionable to spurn informants and people who betrayed one's trust. In our industry, however, we are currently at a point where we are still quite able to identify, and *expose*, people who are willing to sell out to the likes of Microsoft and SCO by engaging in clearly criminal behavior. It was said earlier that bringing in law-enforcement could mean that the servers would be impounded, but there must be some better way to do it. People who violate the law by breaking into the property of others should be punished, and that's that.
Posted Nov 21, 2003 18:46 UTC (Fri)
by piman (guest, #8957)
[Link]
Indeed, and every member of that "public" requires photo identification, often a personal meeting with an existing developer, a GPG signature, and an official advocate who is already a developer. If a developer him or herself does indeed trojan the system (unlikely -- most crackers don't have that kind of patience, especially for real-life interactions), there will almost certainly be a trail back to that person.
Posted Nov 21, 2003 21:01 UTC (Fri)
by NAR (subscriber, #1313)
[Link] (1 responses)
Posted Nov 21, 2003 21:10 UTC (Fri)
by piman (guest, #8957)
[Link]
Note also that auric, the main archive server, was not compromised at all.
Posted Nov 23, 2003 12:20 UTC (Sun)
by XERC (guest, #14626)
[Link]
Actually, what conserns to the technical aspects, then I suggest, that the first thing, that could be done in Linux in general, is to start using some of the OpenBSD packages and libraries. "Search/replace_all" the dangerous and historic C functions(like strcpy, etc.) with newer and safer ones, start using randmomized process ID's(may be Debian already uses those??), "chroot" the webserver's process to it's All of the previous is doable withought manualy touching much of the existing code. OK, after that, well take a look at social engineering and may be Linux developers should more closely cooperate with OpenBSD developers when looking at the code from security's point of view. And again, "copy/paste" from OpenBSD as much as possible, specially those parts, that are at relatively mature states. Hey, but why couldn't Debian be hosted on the
Posted Nov 24, 2003 15:27 UTC (Mon)
by otro_mas (guest, #6820)
[Link]
[1] http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/msit/security/mssecbp.asp
GPG signed announce
Very sad indeed :(Numerous Debian Project systems compromised
There's lots of people that would love to be able to claim that free software is insecure...
Numerous Debian Project systems compromised
A lot of Micro$oftians....
Numerous Debian Project systems compromised
I agree with you on that one. It's like cuting down trees in the park. I undesrstand, that if the trees were so rotten, that they would be a risk for park visitors and the park maintainer desn't take the trees down by itself, then it would be OK to take them down, silently, violently, in the dark, but did anyone inform the sight administrator before the attack or atleast leave a note?Numerous Debian Project systems compromised
Yeah, but what if the people funding you (as a cracker) thought they owned the designs for the nucleic acids that made up those trees, or that their own forests were more immune to the cutting of trees?Numerous Debian Project systems compromised
Another thought:Numerous Debian Project systems compromised
This is a national and international infrastructure problem. At least some of these servers appear to be hosted in the US. The FBI should be investigating this. Crimes against servers that are located elsewhere in the world and were compromised should be ingestigated by the appropriate local authorities and Interpol.
Who Is Investigating?
That's a good way of making sure those servers get locked into the limbo of `legal evidence' and not released for years, while not actually finding any of the perpetrators.Who Is Investigating?
So, how many people really think Debian is more secure than Fedora now, eh? ;-)Numerous Debian Project systems compromised
Okay, perhaps I'll look like I'm humor impaired, but I'll bite.Numerous Debian Project systems compromised
I think it is more a problem with who should have access/ who not than a security exploit. Remember that Debian have 1200+ people working on the project, "it`s open for the public". Suse etc everything is closed, same goes for alot of the other distros.
Numerous Debian Project systems compromised
Maceto wrote:Numerous Debian Project systems compromised
> Remember that Debian have 1200+ people working on the project...
Yes, and every Debian developer (including me ;) is at least in theory capable to insert a backdoor in his/her packages. And packages get installed with root permissions.
I would advocate a number of enhancements to the Debian process (and other distro's):Numerous Debian Project systems compromised
- A network of 'package checkers' should exist, which compile Debian packages on their own and compare the compiled version with compiled versions on the servers. This should be distributed, so everyone who wanted to donate some CPU cycles and bandwith would be able to do so. Some kind of background task like SETI@home which would check a random package per day per computer or so. This would not help against source-code comprises, but it helps against trojaned binaries without knowing how it was trojaned.
Numerous Debian Project systems compromised
- It has become time to have packages which you can install without having root privileges while running the install and deinstall scripts. Only a select number of packages would need them anyway. The only thing you need root priviliges for in a 'normal' package is for extracting your files, which is something which is checkable and containable.
- A network of 'package checkers' should exist, which compile Debian packages on their own and compare the compiled version with compiled versions on the servers. This should be distributed, so everyone who wanted to donate some CPU cycles and bandwith would be able to do so. Some kind of background task like SETI@home which would check a random package per day per computer or so. This would not help against source-code comprises, but it helps against trojaned binaries without knowing how it was trojaned.
You have no idea of what a package actually does. Most packages must be installed as root.
Numerous Debian Project systems compromised
Numerous Debian Project systems compromised
This is interesting. I would like to know what mechanism was used toNumerous Debian Project systems compromised
either gain unauthorized remote access to the system or to escalate from
an authorized level of access to an unauthorized one.
Was it due to unapplied patches or misconfiguration? Bad passwords?
GPG bugs? Compromized systems which were trusted by the server?
the Linux CVS gateway.
how these attacks are being perpetrated.
This is interesting. I have posted comments before on this site in which I mention the word "betrayal."Numerous Debian Project systems compromised
> Debian have 1200+ people working on the project, "it`s open for the public".Numerous Debian Project systems compromised
I don't know the system the Debian project uses to authenticate, etc. but the fact that there is 1200+ contributors doesn't mean that it's enough to break into one computer used by one of 1200+ contributors to wreak havoc in Debian?
Numerous Debian Project systems compromised
(Long, random) LDAP passwords and SSH, and package uploads must be signed by GPG keys. Passwords are only emailed encrypted. Even if you have a developer's compromised home system, it's pretty hard to get things uploaded. And most developers don't have root or extra-privileged access to the machines.Numerous Debian Project systems compromised
Well, but as with any cracking or hacking, it's clear, that IT HAD A SECURITY HOLE, and that fact doesn't depend on the party or person, who states that, even, if that party were Micro$oft. Numerous Debian Project systems compromised
own directory, stop using /tmp as a place for temporary files(use /tmp/usrname with chmod 0660), make as many operations as monolitic as possible,
check the operations return value, when creating files, make sure, that, when something is writeable, then it's not executeable, and vice versa, don't use easily predictable filenames for temporary files(date, proccessID, userID, etc), filter the strings against string-attacks(a string, that is written by overflow and can be intepretad as a machine code), pay attantion to double-intepretation(for instance,
PHP forbidden, CGI writes PHP code which get's then executed as PHP), etc.
OpneBSD server? It's just a plain server, it doesn't have to be at such a
bleeding age? I think that it would be pretty OK, if OpenBSD were the final, robust, endproduct and various Linux distros were the development plathforms, where all new and exciting is tried and developed.
It's interesting to see the kernel attempt, the Debian attempt, and this paper[1] from Microsoft, all three in a quarter.Numerous Debian Project systems compromised
