LWN: Comments on "Infrastructural attacks on free software"

Infrastructural attacks on free software

khim — Fri, 05 Dec 2003 04:38:13 +0000

..and avoid kernel exploits from compromising the system...

Huh? What crack are you smoking? No, really? Kernel exploit is kernel exploit. It's the end of story, period. Once you have kernel exploit you can easily break out of vserver and somewhat margnally harder is to break out of UserMode Linux.

And currenly kernel is big and monolitic. That's why the HURD is not abandoned, you know.

Dogfood

eread — Fri, 05 Dec 2003 03:26:08 +0000

I would say that Debian wants to be general purpose.

Perhaps they might consider using a special-high secure distro for critical systems.

Thanks.

Infrastructural attacks on free software

jaalwn — Thu, 04 Dec 2003 09:11:15 +0000

I think this hits the crux of the matter.

Irrespective of how the attack occurred, and what had to be done to restore the compromised servers, the real issue was that key Debian services went down, and some services are still down today.

I believe that the folks involved reacted rapidly and appropriately in dealing with the compromise, however, there appears to have been no resource allocated to maintaining continuous service, and hence the disruption to the community.

Imagine an alternative stream of events:

* some hosts discovered to be compromised
* these hosts are isolated / frozen / services go down
> integrity of services are verified
> redundant hosts are activated, service resumes
* forensics are performed on compromised hosts
* compromised hosts are purged / rebuilt
* rebuilt hosts are bought back on-line
* service is switched back to primary hosts

I feel that the biggest disruption was in the communication channels - many folks did not receive the 21st Nov announcement until 25th Nov - it was sent via a host that then taken down shortly after being posted.

There was also extreme pressure on those performing the forensics to diagnose and cure the problem - because services were down until the dianosis and cure could be implemented.

I don't think it is possible to guarantee publicly exposed internet hosts as secure. But it IS possible to minimise disruption and provide continuity of service, with some forethought and planning.

All in all, I believe the Debian effort was commendable, and I will be sticking with the dist. I do hope that next time [yes - it will happen!] the communication channels will be maintained - they are an essential part of the community's security.

Jeff

Infrastructural attacks on free software

Klavs — Wed, 03 Dec 2003 16:21:54 +0000

IMHO the big mistake was having users on a system that does anything else than serve users.

A good way to fix such an issue - an allow users to interact/develop software would be to use vservers (virtual servers see linux-vserver.org) or perhaps UserMode Linux (which runs seperate kernels for each virtual server) would do the trick - and avoid kernel exploits from compromising the system. This would also make the attempt to compromise a lot more easily detectable.
A more precise setup description would depend on the needs of these users, and indeed it's a good idea to devide them into groups (based on access needs) and give these groups seperate vservers.
I know of many people running linux-vserver.org on Debian servers (and Gentoo and RedHat :)

Infrastructural attacks on free software

daniels — Thu, 27 Nov 2003 23:58:10 +0000

Take people out of the loop?

No, seriously, if I had to choose between my family/SO/mates, and my GnuPG
passphrase, I know which one I'd be giving up. People suck, and anything
involving people will never be secure, ever.

Infrastructural attacks on free software

daniels — Thu, 27 Nov 2003 23:57:47 +0000

Err, a few security team members are also DSA (admin team), so it wouldn't have
been too difficult for them to extract the debian-security-announce subscription
list from murphy. They wouldn't have to had murphy up to send it out; therefore
the idea could well have worked.

I trust the security team and the DSA enough to believe that they would've come
up with a more-than-satisfactory solution to the hypothetical problem, had it
arisen. People don't get to DSA/security because they just went and drank with
the right people - they're there for a reason.

Infrastructural attacks on free software

NAR — Thu, 27 Nov 2003 19:43:18 +0000

While running the Debian servers on, say, OpenBSD would be somewhat more secure, it would also force Debian maintainers to keep up with OpenBSD developments and be expert users of two very different systems.

I disagree. Only the sysadmins of the servers should be "OpenBSD experts" - a simple user might not even notice if he's on OpenBSD instead of Debian. The basic UNIX tools are the same everywhere (and the GNU tools can be installed, if needed) and I don't think a package maintainer needs any more than that on the main servers.

Bye,NAR

Infrastructural attacks on free software

ajk — Thu, 27 Nov 2003 12:04:30 +0000

For an added bit of trust, Martin Schulze (or Matt Zimmerman, or another trusted and visible Debian developer) could have posted the MD5 sums of the updated packages onto the debian-security-announce list in a PGP/GPG signed email

The mailing list server machine was among the compromised, and thus the lists were out of order for several days. It would have taken a lot of time and effort to set up a temporary lists.debian.org, assuming that a backup system was not already set up. Therefore, your idea would not have worked.

Infrastructural attacks on free software

gleef — Thu, 27 Nov 2003 08:28:19 +0000

The author suggests:
Had a major security incident broken out while the Debian servers were unavailable, it would have been difficult or impossible for the project to respond quickly.

I disagree. The Debian project had control over DNS throughout the crisis, and had volunteers with servers and bandwidth willing to help. Had a major security issue hit, it would have been easy for them to set up a security repository, with a patch for the issue on it, on an volunteer's machine (or several), and point DNS for security.debian.org at that machine.

For an added bit of trust, Martin Schulze (or Matt Zimmerman, or another trusted and visible Debian developer) could have posted the MD5 sums of the updated packages onto the debian-security-announce list in a PGP/GPG signed email, so you can validate that a trusted person is saying that these are the signatures of the trusted packages. Actually, that's standard operating procedure in Debian, system compromise or not.

I, for one, feel that the Debian developers performed spectacularly in this crisis. They showed that not only do they have the infrastructure to keep things under control (and fix them quickly) when inevitable problems arise, but they have engineered room to spare if an even worse problem, or series of problems, were to come by.

solutions

Ross — Wed, 26 Nov 2003 18:30:14 +0000

It's hard to offer advice with so few details about what happened. But we
can always try and give generic advice :)

Besides the typical system security steps (disabling extra services,
applying patches in a timely manner, using good passwords, etc.) I can
think of four action which can help prevent problems like this:
minimize access, separate services, reduce dependencies,
diversify implementations, and increase redundancy.

The same people should not be responsible for all critical systems and the
number of people with access to any of those systems should be kept low.

The servers should be limited to one task. The Debian crew might already
be doing a good job of this. I'm not sure.

The server should avoid dependencies. Dependencies mean that a single
problem will cause outages on all systems.

This one is controversial. The systems shouldn't all be running the same
software. The problem with running the same software on every system is
that the chances they will all be vulnerable to an attack at the same
time is much higher. The downside is that managing the systems becomes
more difficult.

Each system should have a backup, probably in a different physical
location, connected to a different network, which can serve as a
replacement while the main system is being restored, examined, replaced,
etc.

One final piece of advice is to make installation simple and repeatable.
This may require careful documentation of what changes are made after the
system is installed or creating special install images. This can help you
restore a system quickly in many situations. Having an identical spare
system can be even more useful because you don't have to worry about
erasing evidence or valuable information on the original hard drive(s).

To carry a metaphor further...

mmarsh — Wed, 26 Nov 2003 15:34:30 +0000

If we really want to run with the "systems are living things" metaphor, perhaps we should start playing up the multiplicity of Linux distributions as a good thing. After all, mixing and matching the best of several distributions to make a new and better distribution is what evolution is all about, right? The really nice thing is that software packages with compatible licenses can be similarly "inter-bred". What becomes important then is having a good set of standards and an easy upgrade/downgrade/crossgrade path from distro to distro or app to app. I'm not sure to what extent the LSB addresses this, though my impression was that it was more concerned with standardizing configurations rather than mechanisms.

Infrastructural attacks on free software

freethinker — Wed, 26 Nov 2003 15:14:32 +0000

> Don't choose software for servers based on marketing or
> political considerations (i.e. we just have to run our
> distribution).

Your other points are good, but I have to take issue with this one. While running the Debian servers on, say, OpenBSD would be somewhat more secure, it would also force Debian maintainers to keep up with OpenBSD developments and be expert users of two very different systems. It would also make a critical part of the Debian infrastructure dependent on an entirely separate project. Neither is desirable.

Besides, while no one can match OpenBSD's record, Debian is no slouch. The distro has an excellent reputation for security. I very much doubt anyone will crack these machines again, now that they've had this wake-up call.

Dogfood

stuart — Wed, 26 Nov 2003 10:36:23 +0000

Quite,

But Debian is (or wants to be) the "Universal Operating System" and hence dogfood is very much in.

Stu.

Dogfood

Robin.Hill — Wed, 26 Nov 2003 09:20:23 +0000

That depends to a large extent on what market the distro is aimed at. If it's targeting the desktop/games market then you should definitely be picking another distro for the server.

Dogfood

walles — Wed, 26 Nov 2003 08:55:54 +0000

> Don't choose software for servers based on marketing or political
> considerations (i.e. we just have to run our distribution).

I have to disagree with the part about not necessarily running your own distro. If the distro isn't good enough for the distributor, how could it be good enough for anybody else? If a distro is too insecure, the distributor should fix it, not avoid it.

Dogfood is an excellent principle.

I agree on everything else you wrote though.

Infrastructural attacks on free software

proski — Wed, 26 Nov 2003 06:30:44 +0000

Don't put more than one service on the same machine. Don't put too many users with shell access on the same machine. Separate users into groups with different permissions to make it easier to find compromized accounts. Use remove logging so that the logs cannot be erased by successful attackers. Don't choose software for servers based on marketing or political considerations (i.e. we just have to run our distribution). Monitor critical files. Insulate development machines from the key infrastructure (web server, FTP). Use security features of the OS when possible (capabilities, ACLs, chroot, system levels).

Infrastructural attacks on free software

smoogen — Wed, 26 Nov 2003 03:21:15 +0000

And how should we fix it? It is easy to say 'things are too fragile and insecure..' but how hard it is to come up with sustainable ways to improve the situation.