LWN: Comments on "The brk() vulnerability"

How far back?

wildpossum — 2003-12-11T06:59:45+00:00

No that is not implied. What SUSE (and other vendors) do is patch the kernel that was released with that box version, rather than risk breaking things by upping the kernel version without adequate testing. So the bug was in the 9.0's 2.4.21 kernel, SUSE patched it when the alarm went out, and you got this update when you ran the YaST Online Update tool.

Bad Maintainance!

hazelsct — 2003-12-04T17:51:45+00:00

I cannot agree with you regarding this particular flaw, as the nature of the bug was not known to people maintaining the code.

But I do agree regarding your earlier example. Alan's practice with 2.2 has been to make a new point release as soon as a vulnerability is discovered and patched, a release which *only* fixes the vulnerability (saving all other developments for the subsequent point release). This greatly simplifies keeping track of secure kernel releases, rather than having to know exactly which revision fixes things for each separate distribution. (For example, "You have 2.4.19-15? Is that RedHat, SuSE, Mandrake or TurboLinux? Let's see, I can't remember exactly which one fixed that problem for your particular distribution...") Marcelo has not done this, and I wish he would commit to doing so moving forward.

The brk() vulnerability

IkeTo — 2003-12-04T12:21:03+00:00

Note that the patch cause a problem on the VMA of the process which calls brk(), and the actually memory read is done by /proc/<pid>/mem. In other words, the exploit depends on an interface of the kernel that will use the VMA without checking whether it is in the kernel region. If that interface does not exist, then the attacker will have a very hard time thinking about any way to use the wrongly placed VMA.

What are those interfaces, actually? I think there are just a few of them: the mem of the pid, and the core dump. Both not performance critical. Does it make sense for the kernel to just check the addresses also in these cases (i.e., not to trust its own VMA's) so that when such a problem occurs in the future, the attackers will not find it so easy to get through the kernel-user protection?

The brk() vulnerability

IkeTo — 2003-12-04T12:13:57+00:00

In other words, if you don't want it to happen to you, don't ssh from ssh'ed accounts.

How far back?

IkeTo — 2003-12-04T12:07:15+00:00

2.2 does not have the special case code for brk(), i.e., do_brk, which tries to speed things up because the end of the text segment is special. In 2.2, do_mmap is used to do brk, and do_mmap do all the checking. I think the kernel developers forget that the ELF-header can be modified to start the program such that the text segment might be at the end of the memory space, so they think that it is not possible to have brk() called in such a way that can extend through the end of the address space---normally, the virtual memory bound checking will stop anything wrong from happening.

The brk() vulnerability

brouhaha — 2003-12-04T05:52:29+00:00

The problem with doing kernel maintainance in public is that the bad guys are watching.

The problem with doing kernel maintenance secretly (e.g., proprietary kernels such as that of Windows) is that it's a little harder for the bad guys to find exploits, but it's a lot harder (effectively impossbile) for the good guys to audit the code to discover and fix weaknesses before they are used for attacks.

Like any engineering decision, it is a tradeoff. Most people that don't have a vested interest in promoting proprietary software believe that the open development model results in better security. The studies Microsoft has funded that claim Linux security is worse reach that conclusion based purely on the number of advisories issued, ignoring the fact that the number of advisories is not a measure or even a good predictor of the number of remaining vulnerabilities in the software.

Bad Maintainance!

iabervon — 2003-12-04T05:22:41+00:00

For vanilla kernels, I think the best idea is really a patch that applies
to all of the affected versions. Releasing 2.4.23 with that as the only
change is fine for x86 users with standard hardware, but if you're
running on a SA-1110 with a CS integrated NIC that requires a patch that
applies cleanly only to 2.4.8, a patched kernel which is very similar to
2.4.2x isn't going to be very useful. Furthermore, this sort of security
fix is generally only a line or two in a part of the kernel that doesn't
change very much.

What I think should happen in this situation is really that Marcelo
should bless a particular patch as the correct fix for the problem for
each 2.4 release, and kernel.org should distribute it, and it should get
into the next release. Fixing this sort of security problem should really
be orthogonal to the regular kernel development process.

Learn from this - flawfinder, RATS

dwheeler — 2003-12-03T17:11:34+00:00

A somewhat similar process is already in place. Folks like Viega and myself have developed static analysis tools; when we see an error, we try to modify our tools to find that too. My flawfinder tool (http://www.dwheeler.com/flawfinder) looks for previous mistakes, as does Viega's RATS. I'm sure more can be done, please do contribute!

The brk() vulnerability

mwh — 2003-12-03T11:45:06+00:00

It's interesting how this backs up the OpenBSD view that any bug is a potential security hole.

Learn from this!

walles — 2003-12-03T09:58:15+00:00

What should be done now is:

Assume you have a perfect kernel with an unknown problem in brk(). Ask yourself, what security measures (other than finding the bug and fixing it) could prevent the bug from being exploited?

When you have a good answer, go implement it and this won't happen again.

Our local Windows guru says that MS has started using the following procedure when security problems are found in their sources:
1. Report the problem to a group making static code analysis tools.
2. That group is responsible for making a tool that can find the bug that was just fixed.
3. The static code analysis tool is run on the sources at regular intervals.

Linux has the Stanford checker and another Free tool of which I cannot currently remember the name, but I don't know if the feedback works as outlined above. And regardless of whether MS actually manages to implement the above procedure or not, the Linux community should.

The brk() vulnerability

ridrid — 2003-12-03T08:33:44+00:00

Heres a hypthosesis from the debian-devel mailing list:

http://lists.debian.org/debian-devel/2003/debian-devel-200311/msg02170.html

Rich Dougherty

The brk() vulnerability

pwaechtler — 2003-12-03T08:21:05+00:00

I guess POP3 or HTTP (cvs repository?) with password

The brk() vulnerability

dlang — 2003-12-03T08:12:28+00:00

remember that the Debian project boasts 1000 developers. I don't know how many had access to the initial machine, but it's not unreasonable to consider that at least half of them did.

if these people each accessed this machine from 2 different machines (home and work for example) that means that there are 1000 different places that the inirial login password could have been aquired from.

remember that if you use ssh to connect to a machine that machine is only as secure as the least secure of the machines that are allowed to connect to it. chain this a few times, multiplied by a lot of people and you should figure that a determined enough person _will_ be able to get user access to it.

The brk() vulnerability

dkite — 2003-12-03T06:34:17+00:00

This hit the news here in Canada. Didn't see it, but a coworker said they
heard something during the lunch news.

This is a local exploit. From what I understand, it isn't exploitable
unless you can install and run an application that does the dirty work.

Which raises the question of how the servers were compromised. Obviously
this bug was not the only problem.

Derek

The brk() vulnerability

stuart2048 — 2003-12-03T05:45:58+00:00

I too would like to offer a big thumbs up on LWN's coverage of this event. Thanks guys!

One thing puzzles me though. How did the attacker gain unpriviledged shell access to the systems in the first place? Apparently this was done via a sniffed password. But how did this password get sniffed? Was it cleartext on the wire? If so, what was it -- telnet, rlogin, X11 keystrokes?

If this was a network sniff, what was the topology? Where did the sniff happen?

--Stuart

How far back?

Mithrandir — 2003-12-03T03:17:11+00:00

SuSE have applied 144 patches to that kernel, so it could hardly be called a vanilla 2.4.21. I'd be guessing that they applied the patch as a routine security patch at the time that it was released for 2.4.23-pre.

Bad Maintainance!

dlang — 2003-12-03T00:26:45+00:00

you see it as a good thing to discourage people from useing the vanilla tree and instead have everyone use a tree patched with some arbatrary set of patches? (last I heard Suse and Redhat both have >200 patches in their trees)

if every distro does this how in the world can anyone track down a bug report?

I agree that the way the 2.4 kernel releases have been done tends to encourage this, but I see this as a bad thing as it further fragments the testing base.

If a security hole is found and itentified as such (which I agree this was not) then the proper procedure is to apply the fix to the last released kernel, release a new kernel, and then take all the work that was done in the meantime and apply it to this new kernel and pick up where you left off (you can even keep the same preX number if you want) the use of source control systems makes this fairly trivial to do.

counting on people to be paying enough attention to every patch that is released to know that it's a security patch and then apply it themselves (or wait for a distro to do so) becouse you have a fixed 'release procedure' is being lazy

How far back?

pblanco — 2003-12-03T00:15:26+00:00

It's interesting that my SuSE 9.0 with the 2.4.21-144 kernel has the patch. The article implied that 2.4.22 and 2.4.23 had the patch, so I assumed that earlier releases wouldn't have it. What gives?

Bad Maintainance!

jafo — 2003-12-02T23:22:48+00:00

The issue is that the security implications weren't fully realized before the compromise of the Debian systems.

My problem with the 2.4.23 release is that it is (IMHO) much more than just a maintenance release. Adding USB gadget drivers, IPVS, SCTP, and others seems well beyond what I'd think of in a maintenance release. These can often cause other patches to fail to apply, making it difficult for admins to get their systems up to date.

Sean

4G/4G

brugolsky — 2003-12-02T23:22:37+00:00

So presumably systems with Ingo's 4G/4G patches are not vulnerable?
I hadn't thought of 4G/4G as an additional security barrier, but perhaps it is.

The brk() vulnerability

lacostej — 2003-12-02T23:00:11+00:00

Exactly. And with the open development of the kernel, it's not up to the maintainer to detect the risk of a fix, but it has to come has low/early as possible from the development process. Because as soon as a potential fix goes into the open, even before reaching the point where a maintainer must take a decision to include or not a patch, somebody may see and use this patch to find out a security problem.

The way I understand it, a maintainer is just follows some kind of agenda. He gets patches, like recommendations/ideas, and based on HIS own feelings, based on the fact that others have put their confidence on him to do a good job, he decides or not to accept/reject the fix. The more patches the less time to know about all the intricate details of every fix. That's why the security level of a patch must be assessed before they reach him, so that he can make the good decision. That's exactly like politics.

And to me, Marcelo does a great job, I am pretty happy with the 2.4 series. If he didn't put the fix in, it's because he didn't have all the information about the fix.

Because of the openess and increasing use of Linux and other open solutions, the number of attacks following this particular way of working will probably increase. It will also be used by the competition. This openess is one of the biggest weakness of open source, a weakness that also supposedly makes its greatness. People will try to turn that around. Make it an example.

Because of all that, perhaps some kind of patch security assessment policy should be introduced. Like patches should perhaps be tracked and reviewed with security in mind. That's thousands of patches, but we must probably be able to find a handful of detection hints sufficient to remove 80% of the risks.

Bad Maintainance!

LogicG8 — 2003-12-02T22:14:10+00:00

I disagree, I think Marcelo is doing the right thing.
It represents a fundamental shift in the way the
kernel is maintained, but I think it's for the best.
What this does is shift responsibility from the
vanilla source tree to individual distributions. All
the mainstream distros have their own kernel trees.
They should keep on top of security fixes and
immediately bump out a new release when something
happens. The vanilla sources aren't for everybody
anymore. I'm not trying to be elitist here, most
people just don't recompile their kernel. If you are
smart enough to compile your own kernel you are
probably smart enough to patch it when a fix comes
out. If not you might want to consider letting a
proffessional do it for you and use the kernel that
comes with your distro.

Stable releases should be for milestones in development
not for every security bug that comes out. Distributions
should provide security updates. To do otherwise dilutes
version numbers and makes them useless for indicators of
progress.

How far back?

alan — 2003-12-02T21:47:40+00:00

2.2 and 2.4 are so different, it likely wouldn't have mattered anyway

Bad Maintainance!

smoogen — 2003-12-02T21:29:49+00:00

Let me follow your logic:
Linus, Andrew, Alan, et al dont think of it as a security bug fix, but more of a cleanup and possible problem. They dont tell people to stop using 2.6.0.x and go to the next test version.
Marcelo applies the patch and is supposed to know that it is a security bug fix of great importance.

I have thought that Marcelo was pretty slow during his Connectiva days.. but this is a bit ridiculous.

Re: Bad Maintainance!

freethinker — 2003-12-02T21:20:41+00:00

I don't agree. Yes, we can have some people, who have the skills and time to track exploits, patch their systems. But wouldn't it be better if the maintainer patched the kernel and issued a new release with an appropriate message regarding the urgency? Then many more people would be aware and would upgrade. That's what the maintainer is for: to track these things, to be aware of urgent issues, and to do the right thing with them.

I won't go so far as to say that Marcelo fell down on the job here. I don't know all the circumstances. But the question should be asked.

The brk() vulnerability

elanthis — 2003-12-02T20:21:51+00:00

This isn't really true. It's similar to how bugtraq and other security organizations give companies time to release fixed software before the announcement of the vulnerability is even made - the fix exists and users can download it, and have had ample time to do so, before most attackers even know there is a hole.

With the open nature of the kernel developement, the "announcement" is out there before anyone but a few kernel developers have a chance to so much as know the patch exists.

Some good policy on keeping security fixes "secret" until after they're in a release (or at least -pre patch), and then keeping the details secret until the fix has been out for a while, could solve this problem. *could*.

Re: Bad Maintainance!

crimsun — 2003-12-02T20:03:48+00:00

I think it's a fundamental difference in how a maintainer follows the rigor of his release schedule. The argument has been made that people who track exploits closely will patch their systems regardless of whether a version bump is made to accomodate the release of a critical fix. I feel Marcelo's doing a fine job maintaining 2.4. Yes, Alan's policy with 2.2 has always been more finely-suited to security releases reflecting version bumps. That way there is no confusion.

The original closing hook really stands; we all need to be watching cset merges.

Bad Maintainance!

AnswerGuy — 2003-12-02T19:34:01+00:00

I'm increasingly unhappy with Marcelo's maintenance of the 2.4 kernels.

He has to learn that fixes that impact security in the core kernel must be released as a new kernel IMMEDIATELY! Pussy footing around with 5 or 6 pre-releases that contain lots of lesser fixes is a dis-service to the entire community. There are times to release a new stable kernel with a single bug fix, even it that's ONLY A SINGLE LINE.

I seem to recall there was a similar problem around 2.4.19 or 2.4.20 --- a known fix for a vulnerability was deferred in favor of the release based on the rc* series --- business as usual. That's not the way it should be done.

I long for the days when the "ping of death" elicited a new kernel IN LESS THAN ONE DAY! (Alan Cox).

Jim

The brk() vulnerability

stijn — 2003-12-02T19:20:49+00:00

Remember that Jon is looking at the article from the other side of your screen.

The brk() vulnerability

beejaybee — 2003-12-02T19:14:16+00:00

Sure - but it's better to do your washing in public than not to do it at all.

Let's prove ourselves better than the Windows admins by getting our systems patched reasonably quickly!

The brk() vulnerability

JoeBuck — 2003-12-02T19:04:17+00:00

The problem with doing kernel maintainance in public is that the bad guys are watching. Every commit to the source control system that fixes a flaw is an opportunity for a cracker to ponder whether unpatched systems can be exploited. I fear that publicity surrounding this particular exploit will alert others to this way of finding new cracks. Of course, the kernel maintainers are already aware of this possibility, so they will be more cautious when they are conscious that they are fixing a security hole. But any missing integrity check might make an exploit possible.

How far back?

ncm — 2003-12-02T18:43:44+00:00

When was this hole introduced? Apparently 2.2 has a proper range check, but that was deleted.

Whose patch deleted it, and when, and in which release?

The brk() vulnerability

corbet — 2003-12-02T18:29:10+00:00

No...I meant the other left...

The brk() vulnerability

parimi — 2003-12-02T18:28:06+00:00

Thank you for such an informative article on exploit! In the line The kernel organizes a process's memory in a way vaguely similar to the diagram at the left, should the ending portion of the line have been diagram at the right ?