Lessons from the Debian compromise

Recall the sequence of events. In the month of November, an unknown person developed a crack that exploited that now famous kernel flaw and found his way into a Debian developer's machine. Although it's not known if the attack was focused on the developer's machine in particular, it was quickly understood by the attacker that this PC presented a means to accessing the Debian servers. He installed the requisite tools that took over the machine and sniffed out the passwords. The attacker then obtained the password that enabled him to compromise a Debian project server. In quick succession, he penetrated a number of machines which spanned North-America and Europe.

It must be understood that up to this point the attack had not been detected. The machines were penetrated and had been successfully subverted. The attacks were executed in such a manner that none of the installed security mechanisms caught the activity. So why didn't the archives get compromised? And how was it that the attack, was even discovered?

The hand-crafted kernel exploit was not perfect. According to a group of Debian contributors who were interviewed at a recent Linux User's Group meeting (LUG), the exploit worked on all of the Intel machines but failed against one Sparc system, which is where the archives happen to reside. Another crack imperfection was that it generated strange messages in the log files which led to the attack's discovery. It turns out that one of the system administrators became uneasy as he was looking through the log files of one of his machines. He quickly understood that the messages were not normal and the other machines were checked out in short order. This is how the attack and its point of entry (the developer's compromised machine) were discovered.

What are the lessons learned?

• Crackers can make bad code: the existence of those log messages indicates a lack of professionalism and sloppiness that eventually led to the attack's discovery.

• The bio-diversity of mixed environments defeats mono-culture weaknesses: it's easy to criticize the fragility attributed by the dominance of a Microsoft centric work environment. But we seemed to have missed the fact that a Linux-only environment is monoculture too. Things could have been worse if it wasn't for the inherent differences between Intel and Sun System architectures.

• Good people make a difference: a sharp brain and active curiosity are a great combination. Given the time and resources, all exploits can be caught.

Has anything been learned from this event that can help us formulate a more proactive policy? That answer depends on how much we, the open source community, are willing to work to eliminate these violations. These kinds of people can exploit a hundred machines before they stumble over one that can really hurt us. And that's the irony, for every attack that is noticed there are ten more that are unseen. By increasing the diversity of our systems and the alertness of our administration, we improve our chances of detecting and shutting down this sort of attack before it does real damage.

> Another crack imperfection was that it generated strange messages
> in the log files which led to the attack's discovery. It turns out
> that one of the system administrators became uneasy as he was
> looking through the log files of one of his machines.

Note that a simple log checking program might have resulted in
much quicker detection.

Unless the attacker was clever enough to disable outgoing mail
(and then clean the logs).  Then you would need remote logging (as
available with syslog-ng), with the log checker running on the
logging server (and the logging server needs to be the most secure
server, e.g. only accessible to a few individuals who run very
secure workstations).

-scott