|
|
Subscribe / Log in / New account

Stable kernel 2.6.32.8

Stable kernel 2.6.32.8 has been released. It is fairly large set of patches, with changes all over the tree, along with at least one security fix. Users of 2.6.32 are, as usual, very strongly encouraged to upgrade.

to post comments

Stable kernel 2.6.32.8

Posted Feb 9, 2010 15:29 UTC (Tue) by PaXTeam (guest, #24616) [Link] (56 responses)

which one's the security fix and how did you determine it?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 15:51 UTC (Tue) by jake (editor, #205) [Link] (49 responses)

> which one's the security fix and how did you determine it?

because I read Greg's announcement email (linked), part of which says:

combined with verifying that a security problem
really was fixed and backported properly

jake

Stable kernel 2.6.32.8

Posted Feb 9, 2010 20:08 UTC (Tue) by PaXTeam (guest, #24616) [Link] (13 responses)

> combined with verifying that a security problem
> really was fixed and backported properly

and how did you link that to the amd64 personality DoS based on the announcement email or the commit messages? does it mean that the connector bug, the e1000 bugs, etc are not security related at all?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 20:15 UTC (Tue) by jake (editor, #205) [Link] (12 responses)

umm, all I said was that there was at least one security fix in there, which was based on that quote from Greg's email. As we have all discussed here many times, there is no one who goes through the -stable patches specifically looking for all of the security problems, at least no one who reports what they find publicly.

jake

Stable kernel 2.6.32.8

Posted Feb 9, 2010 20:38 UTC (Tue) by PaXTeam (guest, #24616) [Link] (11 responses)

> umm, all I said was that there was at least one security fix in there,
> which was based on that quote from Greg's email.

right, in other words you didn't really answer the first part of my question: all we can know is that at least one security bug is fixed but not which one (good luck backporting it). that's all i wanted to point out. you could (and did, sort of) say 'it is business as usual', yes, but it doesn't make it any less sad.

> [...]there is no one who goes through the -stable patches specifically
> looking for all of the security problems[...]

except usually there's no hint whatsoever about security fixes, this time was a rare exception and one can wonder why stop halfway and not actually make it clear what the security fix was. one could also ask how you know security bugs are fixed at all when you can't point at commits that do so but i digress.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 21:30 UTC (Tue) by chad.netzer (subscriber, #4257) [Link] (4 responses)

Ok, how about *each* bugfix is a security fix. In a couple minutes of perusing I saw a kfree fix, a possible integer overflow fix, some NULL pointer dereferences, memset arg ordering fixes. All helpfully noted in the commit messages, by the way. So where would you like it all backported to? Would you care to help out?

1e8896049716fd580718bb9431c2ad3bddd114d7
78da404b13afa162e9da0384f553db5f19bc94b0
b260729c8a49452ae9491e3cb94750687f221d2b
e06fbe9a4092960a1db1fa973c9ec13a3ddce3f9
253f887cc8d719087f8de403cfe1a60b5e56b454

You just let us know where you want 'em, and whatever other whim of yours we can possibly satisfy.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 23:53 UTC (Tue) by PaXTeam (guest, #24616) [Link] (3 responses)

> Ok, how about *each* bugfix is a security fix.

i'm afraid you failed right here.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 0:01 UTC (Wed) by chad.netzer (subscriber, #4257) [Link] (2 responses)

In my best Jack Palance voice: "Prove it."

Stable kernel 2.6.32.8

Posted Feb 10, 2010 0:06 UTC (Wed) by PaXTeam (guest, #24616) [Link] (1 responses)

9c66557324ea4879abe8c9dde769a0061c81e1ac

Stable kernel 2.6.32.8

Posted Feb 10, 2010 1:18 UTC (Wed) by chad.netzer (subscriber, #4257) [Link]

That's a "bugfix", is it? Wow, convincing...

So, that's one (possible) non-security commit. Any more? Keep this up and you may just answer what you set out to ask.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 22:45 UTC (Tue) by ewan (guest, #5533) [Link] (5 responses)

right, in other words you didn't really answer the first part of my question: all we can know is that at least one security bug is fixed

That's all the summary text claimed! This really isn't complicated - Greg says there's a security fix, LWN reports that there's a security fix. No-one said they'd identified which one it was. I might equally well ask you how you determined that the moon is made of green cheese and populated by pink elephants and happy little elves. Since you didn't actually do that, you won't be able to point to specifics about it.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 22:56 UTC (Tue) by bojan (subscriber, #14302) [Link]

Given this text:

"combined with verifying that a security problem really was fixed and backported properly"

It is obvious that:

- one security problem was identified
- one security fix was backported and tested

In other words, people that released this update already know which bug that is and they obviously know how they determined that. The question is, why can't we?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 23:47 UTC (Tue) by PaXTeam (guest, #24616) [Link] (2 responses)

> That's all the summary text claimed!

not sure why you're bringing this up here, did anyone question that? what i asked since the beginning is to identify the security bug mentioned in the announcement. looks like you're none the wiser either though ;).

> Greg says there's a security fix, LWN reports that there's a security
> fix. No-one said they'd identified which one it was.

did i say that anyone had identified it? that's the very first question i'd asked in fact if you had cared to read the thread. according to nix it was a specific bug but then his guess is as good as mine since there're more security related bugs fixed as far as i can tell.

this is actually turning comical, are you saying that one can announce a fix for a security bug but not know which one it is? that's a new low in linux security.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 1:26 UTC (Wed) by ewan (guest, #5533) [Link] (1 responses)

did anyone question that?

Not question, so much as assume. If you read your original question "which one's the security fix and how did you determine it?" as being addressed to LWN, which is clearly the sense in which Jake answered it, you're assuming that LWN knows which is the security fix, and has some way of identifying it. That appears not to be the case.

did i say that anyone had identified it?

Well, yes, you did rather. To ask how something was done is to beg the question as to whether it was done at all.

are you saying that one can announce a fix for a security bug but not know which one it is?

I'm saying that LWN can report on an announcement of a security fix without knowing which one it is. If your question was addressed to GregKH rather then LWN, perhaps you'd have been better making that clear, or even sending it directly to GregKH rather than LWN?

Stable kernel 2.6.32.8

Posted Feb 10, 2010 14:55 UTC (Wed) by PaXTeam (guest, #24616) [Link]

> If you read your original question [...]

...i would see it addressed to the generic 'you' (the reader), and not to any particular person. apparently i'm not alone in that interpretation as nix managed to answer despite not being LWN staff.

> you're assuming that LWN knows which is the security fix, and has some
> way of identifying it. That appears not to be the case.

with s/LWN/the person answering the questions/, that's exactly the point i wanted to make: there's once again no information helping to identify the security fix(es).

> I'm saying that LWN can report on an announcement of a security fix
> without knowing which one it is.

you may not remember it, but LWN had once proudly proclaimed this:

> We also humbly suggest the LWN security page as a central place to look
> to keep up on security issues.

the source: http://lwn.net/1999/features/MSResponse.php3 . oh boy, how the tables have turned since...

Stable kernel 2.6.32.8

Posted Feb 10, 2010 1:12 UTC (Wed) by rahvin (guest, #16953) [Link]

I might equally well ask you how you determined that the moon is made of green cheese and populated by pink elephants and happy little elves.
I might well ask you how you determined it's NOT. After all, everyone knows the elephants are green and it's the elves that are pink, and they aren't happy, in fact they are quite suicidal. Sheesh.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 22:50 UTC (Tue) by xorbe (guest, #3165) [Link] (34 responses)

Oh goodness don't feed the troll, he posts the same thing to every LWN kernel release thread.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 0:25 UTC (Wed) by bojan (subscriber, #14302) [Link] (32 responses)

When did asking sensible questions become trolling?

Stable kernel 2.6.32.8

Posted Feb 10, 2010 8:36 UTC (Wed) by k3ninho (subscriber, #50375) [Link] (1 responses)

In the context of a project where the full source is available, and where updates improve
the whole tree, asking which fixes are security fixes identifies holes in the Linux kernel.

If you're following 2.6.32, you can upgrade because you get a lot of fixes in .8. If you're
trying to learn where the weaknesses lie in Linux 2.6.32, the process doesn't track that
information. If you're trying to call out the Linux security process (and think like a
script.kiddie looking for direct vulnerabilities to abuse), then you ask which fixes are
security fixes, trolling like PaD did.

K3ninho.
[Dang Android line wrapping]

Stable kernel 2.6.32.8

Posted Feb 10, 2010 15:16 UTC (Wed) by PaXTeam (guest, #24616) [Link]

> If you're trying to learn where the weaknesses lie in Linux 2.6.32, the
> process doesn't track that information.

fortunately for the rest of the world, there's http://cve.mitre.org/cve/cve.html and http://web.nvd.nist.gov/view/vuln/search to help one look for "linux 2.6.32" and not come away empty-handed.

makes one think whether the US government is out to help script kiddies or perhaps there's some deeper need for such information out there? food for thought before you make silly statements next time...

Stable kernel 2.6.32.8

Posted Feb 10, 2010 13:57 UTC (Wed) by pflugstad (subscriber, #224) [Link]

When the same point is brought up over and over and over and over and OVER.

We've gotten the point he's trying to make - probably 18 months and dozens of point releases ago, we got the point.

Honestly, at this point he's proven, at least on LWN, that he's a troll. He does not add anything to the discussions here.

The stable tree maintainers are not going to change what they are doing based on his posts here (and probably not based on anything he does).

Stop responding to him and maybe he'll go elsewhere (of course, now he'll respond to me that we're burying our heads in the sand, or something like that).

Stable kernel 2.6.32.8

Posted Feb 10, 2010 16:31 UTC (Wed) by nix (subscriber, #2304) [Link] (28 responses)

This is someone who claims at length not to understand the difference
between 'at least one of this set of patches is a security fix' and 'we
know exactly which of this set of patches are security fixes'... and this
despite the fact that reporting the latter would require precognitive
powers or constitute lying, given the number of believed-nonexploitable
bugs that later turn out to be exploitable (something PaXTeam surely well
knows).

Looks like a troll to me.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 23:00 UTC (Wed) by bojan (subscriber, #14302) [Link] (27 responses)

> 'at least one of this set of patches is a security fix'

> 'we know exactly which of this set of patches are security fixes'

Rubbish. When an announcement says that a (meaning one) security fix was backported, then it is obvious that it is known which one that is.

Nobody is saying anything about potential security problems related to other fixed bugs. Everybody understands that sometimes bugs that didn't look like security issues turn out to be ones. Announcement can simply says so, so that any potential liability may be avoided.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 23:18 UTC (Wed) by dlang (guest, #313) [Link] (26 responses)

the kernel developers and stable team have decided not to try and judge which patches are security fixes and which are merely bugfixes.

They believe that if they did so, people would only apply the fixes marked as security, and so would end up being exposed to security problems fixed by other things that weren't marked that way.

in addition, they are tired of being harassed about the issue, so now they make a point not to call out particular fixes as being security fixes. This is human nature at work.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 23:49 UTC (Wed) by PaXTeam (guest, #24616) [Link] (3 responses)

> the kernel developers and stable team have decided not to try and judge
> which patches are security fixes and which are merely bugfixes.

you're wrong. they didn't have that decision to make to begin with because they're simply not qualified to make such judgement calls due to lack of expertise. what you wanted to say is that they decided to not share what others tell them about security bugs. we call that a coverup in other areas of life.

> They believe that if they did so, people would only apply the fixes
> marked as security

and they have presented exactly zero evidence for such a belief. not to mention that it's outright insulting to assume that people would be that dumb.

> in addition, they are tired of being harassed about the issue, so now
> they make a point not to call out particular fixes as being security
> fixes.

they are tired of being held accountable for their treatment of security bugs. something proprietary vendors have also had to learn, mind you but they at least did. their reaction is nothing short of 'punishment' of their own userbase, a rather childish attitude at that.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 2:08 UTC (Thu) by mfedyk (guest, #55303) [Link] (2 responses)

> > They believe that if they did so, people would only apply the fixes
> > marked as security

> and they have presented exactly zero evidence for such a belief. not
> to mention that it's outright insulting to assume that people would
> be that dumb.

There is always someone that dumb (using your term). Look at all of the unpatched windows boxes, or even the uproar over firefox 3.7, err I mean 3.6.37. Also look at the latest hack to get debian stable working for their purpose (I used debian for over 8 years and more than 50% of debian users use debian testing or sid)

Once you come down from your lofty ivory tower, you'll see the reality of the individual that believes they're right, no matter what anyone else thinks.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 10:12 UTC (Thu) by nix (subscriber, #2304) [Link]

Hell, I ran into a Linux box recently at a friend's, on the Internet, running Red Hat 5.0. That's not RH*EL* 5.0 or Fedora 5.0, note: that's Red Hat 5.0. Genuine 1997 vintage 2.0.29-ish kernel and libc5 userspace IIRC, never upgraded. Said friend wasn't even aware it *could* be upgraded. And it was in use as a firewall.

So, no, this sort of thing is not unheard of in the least.

(I dislike automatic upgrades that you can't turn off, but automatic upgrades *by default* seem like a very good idea to me. People who don't know or care about security might be secure-by-default then.)

Stable kernel 2.6.32.8

Posted Feb 12, 2010 0:35 UTC (Fri) by PaXTeam (guest, #24616) [Link]

> Look at all of the unpatched windows boxes[...]

it seems you're confused. unlike binary Windows updates, the patches are not useful for end users but rather distro builders, sysadmins and other people maintaining their own kernel (for the better or worse, let's not digress into the costs/benefits of not being on latest -stable). they had better be competent at what they're doing.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 23:54 UTC (Wed) by bojan (subscriber, #14302) [Link] (21 responses)

> the kernel developers and stable team have decided not to try and judge which patches are security fixes and which are merely bugfixes.

This is not true. Read the announcement:

> combined with verifying that a security problem really was fixed and backported properly

Obviously, it was identified that a particular bug/fix was a security problem.

The only question is, if they identified it already, why can't we know what it is?

Stable kernel 2.6.32.8

Posted Feb 12, 2010 16:20 UTC (Fri) by hmh (subscriber, #3838) [Link] (20 responses)

Well, you could have asked Greg, but it should be really obvious if you look at the patches, and I suppose anyone who really wanted to know did just that. And probably found out there is more than one security fix patch in there.

The problem is that people are really bad at logic as a rule, and will infer that:

1. a patch was marked as a security fix
2. the others were not

so

3. the others are not security fixes

Which is incorrect. It is also dangerous if they follow with "so, I will apply/backport only the one marked as a security fix".

There is more: while I personally mark my patches as security fixes when I am aware they fix a security problem, I certainly won't claim that a patch of mine that I have NOT marked as a security fix isn't a security fix, as I might not realize I am fixing a security hole.

Stable kernel 2.6.32.8

Posted Feb 12, 2010 21:33 UTC (Fri) by PaXTeam (guest, #24616) [Link] (19 responses)

> The problem is that people are really bad at logic as a rule, and will infer that:

have you actually seen evidence of such thinking? among the group targeted by patches (see my post above: http://lwn.net/Articles/374094/)?

> It is also dangerous if they follow with "so, I will apply/backport only the one marked as a security fix".

why is it dangerous? do they somehow *reduce* the security of their systems? sure, they could do better than that (still under the assumption that such dumb people actually exist, which i seriously question) but any improvement in system security is still improvement, whatever way you look at it.

or another perspective: if you follow your own logic, file system corruption bugs should not be announced as such either since you can't assure people that other commits don't fix such corruption as well (as a sidenote, a memory corruption bug that often lies behind a security bug is also a good candidate for file system corruption given how much data/metadata lives in kernel memory). yet such bugs are properly described in the commits. how do you explain that?

Stable kernel 2.6.32.8

Posted Feb 13, 2010 4:24 UTC (Sat) by hmh (subscriber, #3838) [Link] (18 responses)

>> The problem is that people are really bad at logic as a rule, and will infer that:

> have you actually seen evidence of such thinking?

Yes. And not just with kernel bugs. And certainly not just with users.

There are some rather unprepared people taking care of software for others out there. You learn a LOT when your main hobby for the last 10 years was to be a large distro downstream maintainer, and not just from the mistakes of others.

Back to your reply, yes, a partially patched system is usually better than an unpatched system, but that DOES assume the skipped patches were not shadow dependencies of the ones that were not skipped. And I know you do know better than to assume this can't happen. However, that was not the point. I consider "selective backporting" dangerous because I DO mean it is often being done on behalf of many others, without proper care.

>or another perspective: if you follow your own logic, file system corruption bugs should not be announced as such either...

Now, don't go making flawed "logic" assumptions. If you're going to use such weak "logic", let me remind you that I could as well just be advocating that "selective patching" that gives higher priority to security bugs is a very bad practice and should be severely frowned upon. Which happens to be the correct answer, but neither it or your "another perspective" can be properly derived (through logic) from what I wrote.

As far as I know, the bias against disclosing security bugs exists due to the security theater circus.

Stable kernel 2.6.32.8

Posted Feb 13, 2010 13:43 UTC (Sat) by spender (guest, #23067) [Link] (15 responses)

You are aware that every single distro's kernel, as well as the stable tree, is "selective backporting" of security fixes and other bugfixes.

So you've established that some of these people doing backporting are "unprepared" -- do you think they know that refcount overflows are exploitable? Do you think they'll know they should backport a fix for a remote DoS when the commit message is simply "Enhance frame detection support"? Remember, these are the same people you claimed were too illogical to understand that if some bugfixes are marked as security fixes, that no other security fixes possibly exist.

In other words, how is the current policy of deliberate obfuscation *better* than mentioning that one found a security bug when that was known at the time to be the case? The vendors and others that have to do backporting aren't happy with this policy. What's being done to make their job easier, being as its the result of their work that decides the kernel that is run on 99% of Linux machines? Is it responsible to stick one's head in the sand and claim no responsibility for the problem?

-Brad

Stable kernel 2.6.32.8

Posted Feb 14, 2010 21:06 UTC (Sun) by hmh (subscriber, #3838) [Link] (14 responses)

> You are aware that every single distro's kernel, as well as the stable tree, is "selective backporting" of security fixes and other bugfixes.

Yes. I happen to think that is a bad thing, and I said as much, so I wonder why the heck you brought that up.

>So you've established that some of these people doing backporting are "unprepared" -- do you think they know that refcount overflows are exploitable? ...

Many of the regular kernel developers won't know that (or recall that fact), so I wouldn't expect most of the distro maintainers to, either. That would be the reason why the distros are supposed to have some very security-minded people in their kernel teams.

>In other words, how is the current policy of deliberate obfuscation *better* than mentioning that one found a security bug when that was known at the time to be the case?

I believe the point is that it forces *every* -stable patch to be backported which is a fine goal by itself, at least IMHO.

That won't help with the stuff that never goes to -stable, but those are very likely to be bug fixes where the developer doing the fixing doesn't even realize he fixed a security hole, coupled to said developer either having a weak "let's keep -stable fixed" commitment, or feeling like the fix is too invasive for -stable.

That, of course, means that the case where someone would see a security fix targeted at mainline, realize that older kernels are vulnerable but the maintainer didn't notice that (maybe they're vulnerable in a slightly different way), and alert someone that -stable needs fixing becomes much harder to happen.

> The vendors and others that have to do backporting aren't happy with this policy

Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.

> What's being done to make their job easier

The -stable tree.

> Is it responsible to stick one's head in the sand and claim no responsibility for the problem?

That's not what I see happening.

However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".

Why can't we have an independent report from you or other kernel security specialists, published regularly, with information on the security implications of every patch hitting -stable (-mainline would be too much)? You know you can get early access to all those patches, don't you? Anyone can, they just need to ask to be in the review team. And I am pretty sure you know your report WOULD be read, and acted upon.

Why can't we see a series of articles from you or other kernel security specialist here in LWN, teaching the kernel developers about all the security issues they should know to identify at a glance, with examples of exploits for each class of pattern to get the point home?

It is completely fair if you don't do that because you have better things to do with your time, or because it doesn't suit your (or your employer's) agenda. But it is also completely fair if the kernel developers don't do what you want because they believe their way is better (or because it doesn't suit their, or their employer's agendas, whatever).

Stable kernel 2.6.32.8

Posted Feb 14, 2010 21:56 UTC (Sun) by nix (subscriber, #2304) [Link] (1 responses)

People don't even need to ask to be in the review team to get access
to -stable patches, do they? Just pulling stable-queue on a daily basis
will do that. (IIRC. I'm not sure if things land in stable-queue before or
after reviewers see them: I thought it was before...)

Stable kernel 2.6.32.8

Posted Feb 15, 2010 15:35 UTC (Mon) by BenHutchings (subscriber, #37955) [Link]

Patches are added to the queue before being sent to reviewers.

Stable kernel 2.6.32.8

Posted Feb 14, 2010 23:59 UTC (Sun) by spender (guest, #23067) [Link] (11 responses)

> Yes. I happen to think that is a bad thing, and I said as much, so I wonder why the heck you brought that up.

I brought it up because if you claim a bug is just a bug, and at the same time say that selective backporting is wrong, then you're ignoring the implications of your views in reality. You expect vendors to inspect (to see if it applies) and backport every single bugfix made to the mainline kernel? *That* is unreasonable. And just to reiterate, you say that "selective backporting" is a bad thing, but -stable is a good thing.

> > The vendors and others that have to do backporting aren't happy with this policy

> Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.

Eugene Teo's website (member of Red Hat's security team, and the main person responsible for backported kernel security fixes):
http://www.kernel.sg/
"Interesting dailydave/blog post about a kernel vulnerability I spotted that was fixed silently upstream. Unfortunately, this happened too frequently."

http://www.openwall.com/lists/oss-security/2010/01/20/3
> I don't think CVE ids can be assigned to this without more information. I'm
> not knowledgeable enough, nor do I have the time to properly understand
> this list.

Eugene says:
"And upstream continues to give us grief..."

And don't you remember Eugene's quote that caused Linus' famous "security people are leaches [sic]"?:
http://comments.gmane.org/gmane.comp.security.dailydave/3946
On Sun, 19 Jul 2009, Eugene Teo wrote:
>
> If the upstream development community can start doing their part by
> differentiating normal bug fixes to the security ones, I think most of
> us will benefit from it.

Since you forgot, this was Linus' calm and measured response:
>Ok, so this is a perfect example of the kind of IDIOTIC blathering that I
>hate to hear from security people.

>Quite frankly, people who state things like that ARE FUCKING MORONS.

>I'm sorry, but it's true. Learn it. Think about it. Deeply, and long.

>This who security exploit is a prime example of exactly why anybody who
>says something stupid like that is so stupid and so WRONG."

So yes, if you stick your head in the sand and don't listen to anything the vendors have to say, shouting expletives at voiced concerns about how things are done, things would appear to be just fine.

> However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".

How can anyone do anything about the problem when it originates from the top down? Linus runs the show, and when he sets his anti-security environment of "a bug is just a bug," it legitimizes it and spreads it to the other kernel developers. I report on all the silently fixed vulnerabilities on my twitter, but most don't get CVEs. I report issues directly to developers and they don't get fixed. I requested a CVE for the SELinux/mmap_min_addr vulnerability and never received a response (Red Hat months later got a CVE for it after I released 8 public exploits).

What changes at all in how Linux kernel development operates if I do any of the things you mention? Do you honestly think anyone will change Linus' stubborn mind? I can publish all day on all the silently fixed vulnerabilities in Linux, but the vendors don't care unless I start publishing exploits for them too. Why would anyone assist the kernel developers for free in the area of security when the developers are actively working against the people trying to improve security? Many of those involved in Linux kernel development (and vendor security teams) get paid to do their work. Isn't this their responsibility? The only way things will improve is from the top down, and it starts with the developers' attitudes towards security. Until then, all Linux users are the ones losing from this stupidity.

If you want to use history as a lesson on how security improves in Linux, let me give you a recap:

security@kernel.org was created some weeks after the PaX team and myself complained (and it made the news, and thus bad publicity) were unable to get any response from Linus and Andrew Morton via email (after several weeks and repeated emails) about some vulnerabilities that were reported.
http://lkml.org/lkml/2005/1/12/138

mmap_min_addr was added after I published a null-ptr-dereference exploit that compromised the machine and disabled SELinux (the bugclass was known for years prior to be exploitable, there were presentations given on it at security conferences, it was published in Phrack, and PaX had already provided protection against it). Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)

-fno-delete-null-pointer-checks was added after I released another exploit for yet another silently fixed vulnerability (again this one generating bad publicity for Linux)

Red Hat changed the default behavior of SELinux to not bypass mmap_min_addr only after 8 public exploits were released (7 wasn't enough -- only Fedora had changed at that point, but after the 8th public exploit for pipe() which affected a huge number of systems, the bad press of SELinux making exploitation easier made them change their tune).

Minor hardening of copy_from_user (using __builtin_object_size) was only added after my perf_counter() exploit was released.

ASLR added by duplicating the work of the PaX team (NIH syndrome)

exec-shield written by duplicating the work of the PaX team (NIH syndrome)

SSP support in the kernel didn't even work until the PaX team fixed it (again, the common symptom of just throwing in features to look good, but not actually caring if they work or not), this was after it was claimed that SSP would have prevented the vmsplice exploit which 1) it couldn't have as the feature was broke, and 2) SSP wouldn't have even prevented exploitation if it were working

What "hard" things have you done to improve the security of Linux?

-Brad

Stable kernel 2.6.32.8

Posted Feb 15, 2010 13:33 UTC (Mon) by eparis (guest, #33060) [Link] (6 responses)

> Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)

While I agree with the facts (if not your interpretation) of most of your statements this one is a bold faced lie. The "original author" (me) has committed fixes for mmap_min_addr bugs as have developers from the Debian community. You have discovered a number of vulnerabilities in the mmap_min_addr implementation over time although have never, that I can recall, reported any of them to me in any form other than an md5 hash, which was obviously indecipherable. I can't recall any fixes for the code either (I might be wrong), but obviously those are the easy part which I gladly do and have done if you do the hard part of finding the problems in the first place.

Maybe you would like to change this situation? I see you say that you told a 'prominent kernel developer' about other issues, which may also affect the mmap_min_addr implementation. If you tell me, a not so prominent kernel grunt, I'll try to get to the bottom of the situation.

Stable kernel 2.6.32.8

Posted Feb 15, 2010 13:52 UTC (Mon) by spender (guest, #23067) [Link] (5 responses)

Committed the fix, yes obviously. But where did those fixes come from?

http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-07/m...
SELinux/mmap_min_addr

http://lkml.indiana.edu/hypermail/linux/kernel/0711.2/027...
Fix for a number of vulnerabilities because the initial mmap_min_addr check was in the wrong place completely

We then even sent you additional emails in private to have you fix additional methods (a.out was one, I can paste the emails if necessary)

http://blog.cr0.org/2009/06/bypassing-linux-null-pointer....
personality bypass from Google

The "Debian community" individual I believe you're talking about is Kees Cook, part of the security team for Ubuntu, not someone I would call a kernel developer. His mmap_min_addr change was to require CAP_SYS_RAWIO to modify the sysctl entry, so that it was consistent with CAP_SYS_RAWIO being required to bypass the limitation.
http://osdir.com/ml/linux-security-module/2009-11/msg0004...

So remind me again which vulnerability fix for mmap_min_addr was your idea? Maybe you have faulty memory.

-Brad

Stable kernel 2.6.32.8

Posted Feb 15, 2010 15:07 UTC (Mon) by eparis (guest, #33060) [Link] (4 responses)

I think I rightly gave you credit for the harder job of finding a number of vulnerabilities. Although, often I wish that I could better interpret your reporting of issues. "3812e371986ad24ace67bab90fd07ca4" does not count as reporting a flaw, even if it was accurate. For anyone interested he really did send me an e-mail with that in it. I had no idea what it meant. A number of months later I learned it was the md5 hash of a sentence explaining how my code didn't work and was quite shitty. Could have had all that cleared up a lot faster if you just told me.

> So remind me again which vulnerability fix for mmap_min_addr was your idea? Maybe you have faulty memory.

I made no claims to having been the first person to find one of your vulnerabilities, usually I have to be aimed to find them. But you seem to be changing your story. I questioned your quote:

> Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)

Maybe it's because I separate finding the vulnerability and fixing the problem. In many situations finding the bug is the easy part and fixing it is the hard part. In security usually finding the bug is the hard part and fixing it is the easy part. Google did both. You have not. You have found vulnerabilities and I don't think I've done anything to discredit your ability to find problems, I just question how you can stand by the above quote. Clearly the author "cares" and has worked to fix every vulnerability he has learned about rapidly (even if you are unhappy with my company's reporting at times)

How about reporting those security issues you mentioned telling a 'prominent kernel engineer' to a lowly engineer who will help get to the bottom of it?

Stable kernel 2.6.32.8

Posted Feb 15, 2010 15:56 UTC (Mon) by spender (guest, #23067) [Link] (3 responses)

Just to clarify, given your response to my question about which vulnerabilities in your mmap_min_addr were found by you, you discovered none of them.

You didn't care to try to find any bugs in the feature you developed. All the bugs were discovered by other people. It's the typical style of non-security people adding security features to the kernel. Particularly when it comes from a company, the main goal is just to be able to say "we have X" and not to care about whether "X" actually works as intended or not. That's the case here.

If you want to be congratulated for the trivial work of writing a couple lines of patches, all for bugs that were either reported to you in private or made public with lots of attention being made to them, you won't find any congratulation from me. It's called doing the job that you're paid to do.

On that note, I don't get paid to do your job, yet I keep reporting vulnerabilities and what I get back is being called a "bold face liar". I'm not sure if Linux leadership/development has caught on to this fact yet, but there is basically no relationship between the security industry and Linux. Linux is viewed as a joke stuck in 1990s security mentality, hence the constant pwnie award nominations and last year's win.

If you really want to improve security, you can take your misguided accusations and direct your attention toward upstream development. If the developers want to be treated like reasonable adults by the security community, they need to take an adult approach to security (instead of calling security people "fucking morons", "masturbating monkeys", "bold face liars", "leaches", "idiots who do not understand development", "idiots who do not understand basic facts", or whatever else the epithet of the day is). It's certainly not the way you entice anyone to do your (paid) work for free.

In case you haven't figured it out yet, I'm not interested in helping anyone who doesn't want to help themselves (any time I've reported extra vulnerabilities have been when a developer showed some initiative/interest in security, or took the handling/classification of security bugs more seriously). What I know for sure is that upstream isn't interested in taking any initiative, but they're very good at reacting to bad publicity and public exploits (at least until that exploit stops being newsworthy, then suddenly the interest in security disappears). So that's what you'll get: more exploits, more bad publicity -- and you'll continue to re-learn the lesson that preventing entire classes of exploits is more important than pretending the developers will always get things right so you can save a CPU cycle.

As for reporting those security issues to you, at least one of them has a patch out there somewhere for it. In the words of Linus, releasing a patch is disclosure, so it should be easy for you to fix it, right? Just as it's easy for any vendor to backport silent security fixes from mainline. If you have a problem with this style of disclosure, you know who to complain to.

-Brad

Stable kernel 2.6.32.8

Posted Feb 15, 2010 16:38 UTC (Mon) by eparis (guest, #33060) [Link] (2 responses)

> (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)

This is quite simply untrue and I thought by now you realized so. Call me a fool. Say that I don't have the time, knowledge, or skill to find the problems that you find and I might not argue. But saying that I don't care is a bold face lie. Claiming that I hadn't found a vulnerability in my code would also not warrant disagreement, but saying that I haven't fixed a single thing is a fabrication.

I don't think pointing out how something you said is untrue should get lumped in with the ridiculous "epithet of the day[s]" you have historically been called.

Stable kernel 2.6.32.8

Posted Feb 15, 2010 17:01 UTC (Mon) by spender (guest, #23067) [Link] (1 responses)

It's very simple, as I explained in my previous post. If you cared about it working, you would have at bare minimum found the first couple vulnerabilities yourself (because they were so juvenile -- within 30 minutes after you published the feature I had a second version of my same exploit, with 2 lines changed that worked just as fine as the original). Your claims of caring about it are very weak, as both you and I agree you didn't discover any of the vulnerabilities yourself. As I explained in the previous post, simply fixing things that are reported doesn't mean that you cared about it working correctly, it's just called doing your job.

You're getting stuck up on my use of the word "fix" when after my first reply I didn't contest that you committed the actual fixes, but the point was that those fixes would have never existed were it not for other people that actually found the vulnerabilities. You had two ways of interpreting it, knowing that 1) neither you nor any other kernel developer discovered a bug that resulted in a fix, and that 2) you committed all but two of the fixes yourself (something you know I'm aware of since you remember our emails).

Replace "fix" with "bug discovery" if you like, with the knowledge that no fixes would exist without previous bug discovery. And I stand behind what I said in general and in this specific case about not caring about added security features actually working. Fixing reported bugs after the fact doesn't explain it away.

-Brad

-Brad

Stable kernel 2.6.32.8

Posted Feb 17, 2010 20:48 UTC (Wed) by nix (subscriber, #2304) [Link]

So, you told Eric about bugs by... not telling him about them, and thus
it's his fault that he didn't fix the bugs you didn't tell him about?

Stable kernel 2.6.32.8

Posted Feb 16, 2010 11:32 UTC (Tue) by hmh (subscriber, #3838) [Link] (3 responses)

>> Yes. I happen to think that is a bad thing, and I said as much, so I wonder why the heck you brought that up.

>I brought it up because if you claim a bug is just a bug, and at the same time say that selective backporting is wrong, then you're ignoring the implications of your views in reality. You expect vendors to inspect (to see if it applies) and backport every single bugfix made to the mainline kernel? *That* is unreasonable. And just to reiterate, you say that "selective backporting" is a bad thing, but -stable is a good thing.

Oh, I wish vendors would do that, but it is sort of like wishing for world piece: I fully expect it to be outright impossible.

Most developers don't do a good enough job of sending all fixes they could/should to -stable, and that's the only place such an effort would have a good chance of doing a lot of good... and would still not be the same as screening every mainline commit.

I failed to make it blatantly clear that I consider selective backporting from the already too-selectively-backported set that ends up being -stable to be a very bad thing, while selectively backporting from mainline is sort of unavoidable in the short term (in the medium or long term, you can check for regressions and switch to a newer kernel).

>>> The vendors and others that have to do backporting aren't happy with this policy

>> Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.

>Eugene Teo's website (member of Red Hat's security team...

Thank you for the pointer. Looks like some people in the teams don't like it, but I still wonder why we don't see clear messages from the vendors themselves. Otherwise, RedHat would pressure the people they pay to do kernel work, to tag the security bugs very clearly in the commit logs. Well, maybe they _do_ have that policy, but I personally haven't heard of it.

It would be cool to be proven wrong on this, though.

>> However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".

>How can anyone do anything about the problem when it originates from the top down? Linus runs the show,

What do you take us for? If developers tag something as a security fix, that commit log is NOT censored. Not by the subsystem maintainers upstream, and certainly not by Linus.

However, there are two important factors: we (as in "your typical Linux kernel developer") usually either fail to recognize the security concerns of a fix, OR we don't want to go the early-full-disclosure route, so we have to submit the fix to someone instead of just sending it up for merging. In that case, it is the security contacts (be them your immediate upstream, kernel.org's, or a distro's) who are the ones responsible for requesting a CVE number, etc.

I worry a lot more about your claims to report stuff and they remaining unfixed.

I have no idea about what is happening in that area, but I can suggest that you could CC these reports to the kernel security teams of the distros themselves as well. AFAIK, if you do that with a good explanation of why it is exploitable, you are pretty much guaranteed to get a CVE out of it.

Or do you mean reports of far more elaborate stuff that hardens the kernel so that it reduces the damage of bugs instead of fixing specific problems (such as much of what the PaX patch is)? Those are often the stuff that would go through the (hard) peer-review and politics of LKML, and indeed would not be merged through security contacts...

I do get the points you made about NIH, I have seen a lot of that in LKML, and it seldon has far more to do with egos than specific technical reasons. That's sad, but it is also something we see everywhere in the software industry, so it is not that surprising.

>Why would anyone assist the kernel developers for free in the area of security when the developers are actively working against the people trying to improve security?

We are a far more diverse group than you take us for. A lot of developers do kernel work as a hobby in their spare time, for example.

Developer education is also the only thing that has a very good chance to help reduce the amount of crap getting into mainline at the current development pace, since even the screening done by subsystem maintainers can only scale so far.

Stable kernel 2.6.32.8

Posted Feb 16, 2010 13:34 UTC (Tue) by spender (guest, #23067) [Link]

> Thank you for the pointer. Looks like some people in the teams don't like it, but I still wonder why we don't see clear messages from the vendors themselves. Otherwise, RedHat would pressure the people they pay to do kernel work, to tag the security bugs very clearly in the commit logs. Well, maybe they _do_ have that policy, but I personally haven't heard of it.

Well, I think it's telling that the most direct request for a change in the policy had to occur on a private mailing list. Linus telling security people on the list to "f*ck me, shut up about idiotic things like that already" likely sent the message that any further complaints would be fruitless. We also have to consider that for the security teams, this is their career. They likely won't get a raise for getting upstream to change their ways, but they'll certainly generate unwanted attention toward their company if they upset Linus enough that another round of "masturbating monkeys" makes the news.

I hear all the time (generally in private emails to me) of people who disagree with it (and you'd be surprised by who some of them are) but that disagreement remains just that -- private. Like you, I would also like to know why the vendors aren't setting policies for the developers they employ, or why they aren't making a more concerted effort in changing policy.

> What do you take us for? If developers tag something as a security fix, that commit log is NOT censored. Not by the subsystem maintainers upstream, and certainly not by Linus.

The problem is not censorship of already-created commit messages, the problem occurs at the time of the commit message, as was established last year during much discussion (you can go back and read the LWN, LKML threads for all the evidence and admissions). It was back then in the summer of 2008 that the official policy was declared (in contradiction to the published Documentation/SecurityBugs) to be that of deliberate obfuscation of changelogs involving security fixes. For this most recent amd64 DoS, POC code was sent, the bug was known to be a security bug, and yet any mention of it was omitted from the changelogs. In fact, from the changelogs themselves, there's no indication that anything is being done other than code cleanup. This is just a fact -- Linus admitted he engages in this policy, and it's in fact the official policy: the existence of the patch is considered "disclosure," so according to him nothing actually needs to be said security-wise about it.

If you think nothing needs to be said about it, then you assume everyone that needs that security fix is capable of understanding the code to figure out that it's a security fix. Imagine if Microsoft took the approach of "we fixed it -- we don't need to tell anyone what we fixed" and what kind of backlash that would cause. It's obvious in the Windows world that security people/blackhats are fully capable of diffing binaries before a patch and after a patch to determine what vulnerabilities are fixed (a more difficult task than with Linux where the source exists). Why do people pretend this doesn't happen with Linux?

> Or do you mean reports of far more elaborate stuff that hardens the kernel so that it reduces the damage of bugs instead of fixing specific problems (such as much of what the PaX patch is)? Those are often the stuff that would go through the (hard) peer-review and politics of LKML, and indeed would not be merged through security contacts...

No, I mean direct vulnerability fixes with attached POC code to trigger them.

The reason I don't email lists (like vendor-sec for instance) with vulnerability reports is because I don't know who reads them, and I don't trust the integrity of the lists. I've pasted private emails from both vendor-sec and the security@kernel.org lists over the course of several years, obviously to the displeasure of the people on the lists (as they'd wished the kinds of things that go on in private never see the light of day). So to me, sending vulnerability reports to lists is equivalent to sending it out to everyone before any fix exists (and for the one vulnerability, a fix wasn't trivial), and long before that fix filters down to end-users.

The current situation is just silly as everyone agrees that people doing backporting don't necessarily have the knowledge they need to spot many of the vulnerabilities unless they're marked as such. I've demonstrated that security people (and blackhats) are very good at spotting these silent fixes and developing exploits for them before vendors are even aware of the flaw, and long before vendors issue updated kernels. Most of my exploits were developed within an hour of finding the commit message. If you don't omit the security information you're aware of, nothing changes on the blackhat side because they can already spot developer weasel words when the developers are committing a vulnerability fix. So who is really benefiting from the current policy?

-Brad

Stable kernel 2.6.32.8

Posted Feb 18, 2010 3:30 UTC (Thu) by eteo (guest, #36711) [Link] (1 responses)

> Thank you for the pointer. Looks like some people in the teams don't like
> it, but I still wonder why we don't see clear messages from the vendors

Some of my quotes Brad pasted in his previous reply are my personal opinions
and not my employers. Deliberate obfuscation of security-relevant changelogs
is not the right thing to do.

Thanks, Eugene

Stable kernel 2.6.32.8

Posted Feb 19, 2010 8:53 UTC (Fri) by mjcox@redhat.com (guest, #31775) [Link]

> Deliberate obfuscation of security-relevant changelogs
> is not the right thing to do.

At Red Hat, if we know something is a security issue in any package, and we fix it, we'll get a CVE for it (if it's missing one) and label it in our updates as well as mention it in public bugs/mailing lists etc.

If something we fix later turns out to have a security relevance we didn't know about at the time, we'll retrospectively add the CVE to the relevant errata.

We definitely encourage clear and complete upstream changelogs that highlight security relevant issues.

Stable kernel 2.6.32.8

Posted Feb 15, 2010 15:50 UTC (Mon) by PaXTeam (guest, #24616) [Link] (1 responses)

>> have you actually seen evidence of such thinking?

> Yes.

is that anecdotal evidence only? just asking because my anecdotal evidence is the exact opposite.

> And not just with kernel bugs.

we are talking about *only* kernel bugs. and that's because other sw developers (learned to) understand the importance of security bugs and they do *not* in general cover them up (quite the contrary in fact).

> And certainly not just with users.

did you mean 'users' in the sense i used it above (i.e., 'patch users', not the grandma kind)? because then it's irrelevant what others think, they get to use what the 'patch users' give them. all the more important to make the life of these 'patch users' easier because any missed security fix will directly affect all their end users.

> There are some rather unprepared people taking care of software for others out there.

then go shame them in public, let the world, and more importantly, their end users know! in case you missed the past decades of the full disclosure debate 'out there', this is exactly what happened to unresponsive and/or uncooperative vendors and look at where we stand today.

> I consider "selective backporting" dangerous because I DO mean it is
> often being done on behalf of many others, without proper care.

so -stable is dangerous as well since it's selective backporting. i don't see why properly reporting known security issues changes any of this danger then, in other words, this is again not a reason for coverup.

> Now, don't go making flawed "logic" assumptions.

i didn't but maybe you misunderstood the 'your logic' reference. it wasn't about the bad implication per se, but what you used it to justify: that people are bad at logic therefore it's somehow dangerous to document security fixes in commits/announcements (look at bojan's question that you replied this to, if it's still not clear). now *this* logic is flawed and i gave you a counterexample where this flawed logic, quite rightly, is not applied and those file system corruption bugs are not covered up - the same should happen with security fixes as well, but for some reason doesn't.

> [...]I could as well just be advocating that "selective patching" that
> gives higher priority to security bugs is a very bad practice and should
> be severely frowned upon. Which happens to be the correct answer

actually you're wrong in the 'very bad practice' part, only because you happened to use the right term for a change: higher priority. notice how it suddenly doesn't make the whole backporting business black&white (something you and other kernel developers make it appear in their 'arguments'), suddenly it's about priorities and it is very correct (and about the only practical way) to classify fixes and order them somehow given that noone has enough resources to do everything (identify/backport/test/release/support/etc) all at once.

> As far as I know, the bias against disclosing security bugs exists due
> to the security theater circus.

now this is a new one and if true, the most childish of all i've heard from kernel developers. what exactly is this 'security theater circus'? and how does it prevent them from disclosing security bugs?

Stable kernel 2.6.32.8

Posted Feb 16, 2010 13:06 UTC (Tue) by hmh (subscriber, #3838) [Link]

>>> have you actually seen evidence of such thinking?
>> Yes.

>is that anecdotal evidence only? just asking because my anecdotal evidence is the exact opposite.

Then we will have to agree to disagree there, as it is indeed anecdotal. It is likely that the groups we have more contact with (and thus are the source of the anecdotal evidences) are different, though.

>> And certainly not just with users.
>did you mean 'users' in the sense i used it above (i.e., 'patch users', not the grandma kind)?

I did mean distro maintainers and patch users. Note that I haven't seen any evidence of skill shortcomings in the kernel maintainers of any of the larger distros, and said as much.

>> I consider "selective backporting" dangerous because I DO mean it is often being done on behalf of many others, without proper care.

>so -stable is dangerous as well since it's selective backporting.

If you trust it to have all important fixes (which is a superset of "known-to-be-security fixes" and also of "security fixes") backported in the long run? Yes, they are. However, they're the best compromise we have so far, IMO better even than getting stuck to very old distro kernels in many cases, as long as you can do your own regression testing.

>> Now, don't go making flawed "logic" assumptions.
> i didn't but maybe you misunderstood the 'your logic' reference. it wasn't about the bad implication per se, but what you used it to justify: that people are bad at logic therefore it's somehow dangerous to document security fixes in commits/announcements

I see. Well, I don't quite agree with that "justification" and I did say as much. But it is one of the main justifications that are put forward when this issue is brought up, and I can certainly see it as a valid reason to prefer to not tag security bugs as such when they are already going to be scheduled to -stable, regardless of whether I'd do it like that or not. And I am quite sure I don't know the entire picture and there are other reasons involved, on both sides.

>suddenly it's about priorities and it is very correct (and about the only practical way) to classify fixes and order them somehow given that noone has enough resources to do everything (identify/backport/test/release/support/etc) all at once

Well, when you see someone asking which fixes are the security ones, because he is only interested in backporting those post-haste, and you look at the commit logs and there are, e.g., RCU or list corruption fixes in there, it gets pretty clear that there are severe drawbacks to the mentality that only security fixes are of critical priority.

I did say I would rather see that issue addressed in other ways than silent security fixes, though.

>> As far as I know, the bias against disclosing security bugs exists due to the security theater circus.

>now this is a new one

No, it isn't. It is an old complain. It refers to two things: the amount of attention and priority that security issues (even non-major ones) are given when compared to non-security issues of similar damage potential, coupled to the importance given to gray-area metrics like "number of security bugs identified" (which is too easy to misuse) and the effects of those by the media.

I better make it clear that I am strongly against downplaying security issues, but I am also against overplaying them (because that, in practice, implies in downplaying data-corruption and system stability issues).

Note that I don't claim metrics are bad. There are metrics that are always useful and hard to misuse, such as "time to deploy fixes", and "fraction of available fixes effectively applied", and obviously, "fraction of issues with fixes available" and "serious regressions introduced by fixes". And even the metrics that are easily misused have their value.

Also note that I don't think security bugs and other classes of critical bugs are exactly the same thing, as by definition the security bugs have one extra component that makes their handling a lot more difficult: the fact that they are, by definition, something that can specifically be targeted and triggered in order to cause damage. Which brings the whole early-disclosure/full-disclosure versus responsible-disclosure (or whatever you want to call these, I apologize if I am not using the correct terms) deal.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 16:44 UTC (Wed) by jcm (subscriber, #18262) [Link]

+1

Jake's doing a good job. There's absolutely no need to fly off the handle because he didn't individually inspect every patch for security impact. Even if LWN did do that, I'm sure they'd not want to say "this has no security impact" on some patch that later does. Far easier to not get into that.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 17:16 UTC (Tue) by nix (subscriber, #2304) [Link] (5 responses)

It's hardly been hidden. There's just been an *entire article* on this very site about it!

Stable kernel 2.6.32.8

Posted Feb 9, 2010 20:13 UTC (Tue) by PaXTeam (guest, #24616) [Link] (4 responses)

> It's hardly been hidden. There's just been an *entire article* on this
> very site about it!

no kidding. had you not been just trolling as usual, you would have noticed that i'd actually read that article and even asked a related question there too. but you can too answer what i asked Jake above: let's see how you figured out the security bug Greg was talking about based on solely the information he provided. btw, i don't recall that reading LWN articles became a mandatory exercise to learn about linux security bugs, maybe you want to submit a patch to Documentation/SecurityBugs?

Stable kernel 2.6.32.8

Posted Feb 10, 2010 17:28 UTC (Wed) by nix (subscriber, #2304) [Link] (3 responses)

So, in other words, your original question in this thread was purely
rhetorical? You already knew the answer? Yet you proceeded to ask the same
damn question you have in response to virtually every announcement of any
stable kernel here for the last two years or so?

That's the very *definition* of destructive trolling.

(oh, and, btw, have you ever made any constructive contribution to this
site's comment threads? Have you ever posted anything that wasn't an
attack on someone? Even once?)

Stable kernel 2.6.32.8

Posted Feb 10, 2010 19:22 UTC (Wed) by PaXTeam (guest, #24616) [Link] (2 responses)

> So, in other words, your original question in this thread was purely
> rhetorical?

no. it was also two questions, not one.

> You already knew the answer?

no, i didn't. i still don't in fact.

> Yet you proceeded to ask the same damn question you have

again, it's two questions and i don't recall ever asking the second one but feel free to find something to the contrary on google. in any case, it's certainly not every time given that it's definitely rare that a security bug is hinted at in the -stable announcement.

> in response to virtually every announcement of any stable kernel here
> for the last two years or so?

my guess is that it's less than one in a dozen announcements but feel free to prove me wrong (google with site:lwn.net would be a good start).

so now that your 'very *definition*' lost its legs, will you actually keep on topic and answer my second question? after all you must have had a reason to pick the amd64 bug against the others i mentioned elsewhere in the thread.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 20:29 UTC (Wed) by nix (subscriber, #2304) [Link] (1 responses)

Yes, I picked the amd64 thing because it was a security bug. A
widely-announced security bug. One which you had, as you admit, already
noticed. You wanted to know a security bug in this release: I told you
about one.

(sheesh.)

Stable kernel 2.6.32.8

Posted Feb 10, 2010 21:30 UTC (Wed) by PaXTeam (guest, #24616) [Link]

> Yes, I picked the amd64 thing because it was a security bug.

i see. so you're saying you read this in the -stable announcement. where exactly? also, since you seem to be a very sharp-eyed reader, what other security bugs did you see in the announcement?

> A widely-announced security bug.

define widely. the -stable announcement doesn't seem to be part of it.

also what do you have to say about security bugs that are not 'widely-announced'? how's one supposed to learn about them?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 16:32 UTC (Tue) by pranith (subscriber, #53092) [Link] (11 responses)

woah!! a stable update containing "changes all over the tree"??? What's going on here? Can you please elucidate why so many changes were needed for a "." update?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 16:56 UTC (Tue) by sbohrer (guest, #61058) [Link]

The changes are all over the tree, because there were bugs all over the tree. Would you expect the bugs to localized to one specific subsystem?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 18:17 UTC (Tue) by chad.netzer (subscriber, #4257) [Link]

http://git.kernel.org/?p=linux/kernel/git/stable/linux-2....

Some driver fixes (pretty normal for a "." release), miscellaneous error handling, backported ACPI features, some filesystem work (non-ext3/4), some odds and ends, and the important "personality" fixes. "All over the tree" is really an overstatement, it's just not a trivial tweak of the last point release (of the "oops, it didn't boot anymore" variety, for example).

This stable series is likely to be used in a couple upcoming distro releases, so it's good to have more of these patches roll in now.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 18:46 UTC (Tue) by MisterIO (guest, #36192) [Link] (8 responses)

Actually it's always like this, I can't really see why you're so surprised.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 19:11 UTC (Tue) by pranith (subscriber, #53092) [Link] (7 responses)

I was under the impression that only _important_ fixes were done to the stable kernel mostly bugs and security. Seeing that there is only one security update and the rest are bugs, it's a bit strange as to the number of bugs which are present in the release.

May be the rate of change of kernel code is too huge to actually filter out all the bugs. Should this not be a matter of concern?

Stable kernel 2.6.32.8

Posted Feb 9, 2010 19:59 UTC (Tue) by dlang (guest, #313) [Link] (3 responses)

it is a concern and the -stable branches are the way those concerns are being addressed.

It has been discovered (by many different projects) through years of trying different things that not all bugs can be discovered before a release, no matter how long that release is delayed.

So what the kernel developers do is that Linus makes a judgement call when they have hit the point of diminishing returns. Linus then makes a release and more people start using it (on a wider variety of hardware and loads). As bugs are discovered the fixes are developed and put into the main tree, and then Greg and the -stable team backport them to the -stable branches of the older kernels.

This large number of patches this time is that various distro kernel teams got their acts togeather and submitted fixes that they had not previously sent upstream. Since most of those patches meet the criteria for the -stable branch, they were also applied there for this release.

Yes, the number of fixes in each stable release seems to be going up to me as well. I don't know if that means that more bugs are getting through, or that more of the bugs that are being fixed upstream are being applied to the -stable kernel as well.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 22:53 UTC (Tue) by drag (guest, #31333) [Link] (2 responses)

The reality of the situation is, unfortunately, the -stable branch is anything but stable.

If you want to use a reliable operating system you simply cannot depend on the kernel developers not to break something you depend on. Too many changes, too many regressions. You have to rely on the distributions specific kernels and those people backporting fixes and driver updates to keep a sane system.

Those people provide much better support and much faster turn around for fixes then the kernel developers. The major downside is that for new hardware drivers to show up it can take upwards of a couple years to trickle back down to end users.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 22:57 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

heh, my view is that I can't trust the distros to not do short-term hacks that break as much as they fix, so I prefer to stick with the kernel.org kernels.

the responsiveness that I've gotten from the kernel developers directly when bugs have been reported there far exceeds what I have gotten from any distro when reporting issues.

to each their own.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 10:15 UTC (Thu) by nix (subscriber, #2304) [Link]

Exactly. I've had regressions of some sort with every single kernel since 2.6.29... and every single regression I've reported has been fixed within days, most within hours, with the exception of a nasty suspend-resume corruption the fix to which is slowed by its unreproducibility by anyone but me and a 12-hour timezone difference. I think I can live with that. I just have to schedule major kernel upgrades for a time when I don't mind a few regressions for a while, and stagger them so I always have a few systems working.

It's the price one pays for being at the bleeding edge.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 20:31 UTC (Tue) by chad.netzer (subscriber, #4257) [Link] (2 responses)

And which commits specifically are "unimportant" in this release? What bugs shouldn't have been fixed?

Stable kernel 2.6.32.8

Posted Feb 10, 2010 9:03 UTC (Wed) by pranith (subscriber, #53092) [Link] (1 responses)

Not saying that anything is unimportant. Just surprised that there are so _many_ important changes in a .8 release. Wish the bugs should had been caught earlier.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 19:43 UTC (Wed) by chad.netzer (subscriber, #4257) [Link]

These bugs *are* being caught "early", in that this kernel release has yet to go into wide usage (ie. Distributions generally aren't released with it.) It is not yet "stable", it is being stabilized. That .8 will turn into .50-ish someday, so just be thankful for those people who are using and testing it now.

Stable kernel 2.6.32.8

Posted Feb 9, 2010 20:30 UTC (Tue) by drag (guest, #31333) [Link] (1 responses)

Ohhh.. I hope that one of these fix the problem I have been having with my Intel video lately in Debian unstable!

 drivers/gpu/drm/drm_gem.c                 |   13 
 drivers/gpu/drm/i915/i915_debugfs.c       |    2 
 drivers/gpu/drm/i915/i915_dma.c           |    4 
 drivers/gpu/drm/i915/i915_drv.h           |    2 
 drivers/gpu/drm/i915/i915_gem.c           |   54 --
 drivers/gpu/drm/i915/i915_irq.c           |   32 -
 drivers/gpu/drm/i915/i915_reg.h           |    8 
 drivers/gpu/drm/i915/intel_crt.c          |    2 
 drivers/gpu/drm/i915/intel_display.c      |   26 -
 drivers/gpu/drm/i915/intel_dp.c           |    6 
 drivers/gpu/drm/i915/intel_hdmi.c         |    5 
 drivers/gpu/drm/i915/intel_sdvo.c         |    3 
 drivers/gpu/drm/i915/intel_tv.c           |    2 


I want to be able to allocate all my 4GB of RAM again!

Stable kernel 2.6.32.8

Posted Feb 10, 2010 0:28 UTC (Wed) by nof (guest, #61716) [Link]

I also had some problems with "Intel Corporation Mobile 945GM/GMS/GME"
on Debian Sid.
I hope this will fix suspend to RAM on my laptop, and no more "reboot
required"..

Ketchup running now :-)

Stable kernel 2.6.32.8

Posted Feb 9, 2010 22:50 UTC (Tue) by bojan (subscriber, #14302) [Link] (14 responses)

Weren't all bugs created equal? What's this talk of "combined with verifying that a security problem really was fixed and backported properly"? Must be some black magic or something :-)

Stable kernel 2.6.32.8

Posted Feb 10, 2010 1:55 UTC (Wed) by njs (subscriber, #40338) [Link] (13 responses)

I don't think they ever said that they don't care about fixing security bugs once they learn they exist, just that determining whether arbitrary bugs are security bugs isn't worth the effort.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 4:20 UTC (Wed) by spender (guest, #23067) [Link] (12 responses)

Isn't worth the effort for who? I'd say it's worth the effort for:
Red Hat (RHEL/Fedora)
Novell (SuSE)
Canonical (Ubuntu)
Debian
Gentoo

Otherwise how can you have any kind of reasonable expectation of security for your vendor kernels?

Why should they all individually have to be playing a guessing game on things that kernel developers already know but deliberately obfuscate? And again, before anyone brings it up again, it's not about the developers themselves trying to figure out if every bug is a security bug -- it's about not hiding or obfuscating what they are clearly aware of. In this case, for starters, it was clearly obvious to both Linus and Greg that a DoS was fixed for amd64 for which a trivial exploit existed. The developers were told by the bugfinder that it was a DoS issue and were sent code to trigger it. It seems obvious stopping this insane policy of deliberate obfuscation would reduce the ridiculous lengths each individual vendor has to go through to turn kernel developer weasel words into actual assessments of bugs.

As an example, the mremap "fixes" that went into the stable tree recently -- a huge set of patches with only vague descriptions of what was being fixed: Red Hat assigned it a cvss2 score of 7.2, saying that it was more than just DoS fixes (7.2 puts it in the highest category of severity, if you weren't aware). Ubuntu (and I believe another distro) just released an advisory for the same set of fixes recently, saying that they only fixed a memory consumption problem leading to a DoS. This is the kind of confusion "a bug is just a bug" causes.

What we know for sure is that "they" (being the kernel developers) don't care about telling us about security bugs once they learn they exist. Greg didn't mention the amd64 DoS by name, and all the commit messages involved in fixing it avoid mentioning it as well (Linus' work here).

While we're on the topic of fixing security bugs though, one security vulnerability that wasn't fixed in 2.6.32.8 was the move_pages() infoleak, which affects kernels >= 2.6.18. If you're running RHEL 5.4 x64, you're vulnerable to the infoleak, which unlike the recent leaks of tiny amounts, involves a leak within a 512MB range. It was posted to oss-sec 3 days prior to the release of 2.6.32.8, and the fix was a trivial 2 lines.

I wrote up an exploit for the vulnerability within an hour of the oss-sec email, available here: http://grsecurity.net/~spender/exp_sieve.txt and integrated into the enlightenment exploit framework here: http://grsecurity.net/~spender/enlightenment.tgz

Finally, I reported two security issues to a prominent kernel developer that affect all 64bit kernels since 2.6.26 several months ago and they've yet to be fixed. You'll be shouting your "a bug is just a bug" mantra until it's revealed months from now that there's yet another vulnerability in the mmap_min_addr code (making null pointer dereference bugs exploitable again) that would have been prevented if the issues I reported months ago had been fixed.

-Brad

Stable kernel 2.6.32.8

Posted Feb 10, 2010 4:38 UTC (Wed) by njs (subscriber, #40338) [Link] (7 responses)

> You'll be shouting your "a bug is just a bug" mantra

Dude, all I did was correct the record on what they said, not make any claim myself. Why you gotta sweep me up into some drama in your head?

Stable kernel 2.6.32.8

Posted Feb 10, 2010 4:42 UTC (Wed) by spender (guest, #23067) [Link] (6 responses)

It was the plural 'you', referring in general to those who agree with that completely misguided notion, and in doing so enable it to continue to be official policy.

-Brad

Stable kernel 2.6.32.8

Posted Feb 10, 2010 8:53 UTC (Wed) by chad.netzer (subscriber, #4257) [Link] (5 responses)

For those wanting more context:
http://kerneltrap.org/Linux/Security_Bugs_and_Full_Disclo...

I personally think Linus's position is more compelling:
http://kerneltrap.org/mailarchive/linux-kernel/2008/7/15/...

Stable kernel 2.6.32.8

Posted Feb 10, 2010 21:07 UTC (Wed) by ballombe (subscriber, #9523) [Link] (4 responses)

Yes, but note that the situation of the stable kernel team is different from Linus situation.

Linus argues that for most patches, it is unknown whether it fixes a security issue, and so it is misleading to mark a small subset of them as 'security fix', and that cherry-picking patches marked 'security fix' create a false sense of security.

However, the stable kernel team job is precisely to cherry-pick patches, and it can be expected that some of them have been picked because they fix a security issue.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 8:00 UTC (Thu) by chad.netzer (subscriber, #4257) [Link] (3 responses)

Surely it is an issue of style and not substance, is it not? If a bugfix is later
shown to be a security fix as well, should it be rewritten to indicate it as such,
or is it not easier to just track this elsewhere where commits can be
unambiguously referred to? If the issue is that security fixes and commits
aren't being disseminated or discussed, surely that can be addressed outside
of commit messages.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 10:16 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

Rewriting commit messages once they land in the stable kernel tree (as opposed to the stable patch queue) is not possible: you'd break everyone pulling the stable tree.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 18:13 UTC (Thu) by chad.netzer (subscriber, #4257) [Link] (1 responses)

Right. Which is another reason *not* to tie any security analysis to a commit message, since the known security impact of a commit can change.

Stable kernel 2.6.32.8

Posted Feb 12, 2010 0:03 UTC (Fri) by nix (subscriber, #2304) [Link]

Well, sort of. It's a reason never to say 'this has no security impact',
but attacks only get worse, never better: it's safe to say 'this is a
security hole', because it's not going to *stop* being a security hole.

Stable kernel 2.6.32.8

Posted Feb 10, 2010 22:25 UTC (Wed) by vonbrand (subscriber, #4458) [Link]

OMG, not this nonsense again...

Work flow is that somebody notices a bug, and fixes it. Somebody else (stable team) goes through the set of patches and picks those they consider "important enough" and "non-intrusive" to include in the next point release. Here "important enough" certainly includes potential security problems, but nobody goes through the (not trivial) work of checking if it is a real security problem or just a "can't ever happen, but the code is badly written anyway" or even creating an exploit. In parallel, somebody figures out something is a (potential) security problem, and gets it a CVE number. There is no connection between the commit fixing the problem (which is probably taken almost verbatim from vanilla, just to be on the safe side) and any CVE announcement and/or exploit writing. Ergo, that information rarely (if ever) shows up in changelogs.

This is not some world-wide conspiracy to hide security problems, the kernel's developers are doing their best to fix known problems (which includes security problems). Want a more stable, secure system? Then you have to check it more (you are certainly welcome to help out) and/or beat more on it (i.e., wait longer while testing and looking it over).

Perhaps the tools should be enhanced so PaXteam (or others) can publish annotations to the commits in the stable (or even vanilla) kernel tree (without touching the originals) tieing the commits to whatever CVE or other anntotations they deem appropiate.

Stable kernel 2.6.32.8

Posted Feb 11, 2010 13:46 UTC (Thu) by tao (subscriber, #17563) [Link] (2 responses)

"While we're on the topic of fixing security bugs though, one security vulnerability that
wasn't fixed in 2.6.32.8 was the move_pages() infoleak, which affects kernels >= 2.6.18. If
you're running RHEL 5.4 x64, you're vulnerable to the infoleak, which unlike the recent leaks
of tiny amounts, involves a leak within a 512MB range. It was posted to oss-sec 3 days prior
to the release of 2.6.32.8, and the fix was a trivial 2 lines."

And was your trivial 2 line fix merged in the 2.6.33-tree (reasonably early) before the release
of the 2.6.32.8 kernel? If not, your argument fails. The -stable kernels only merge fixes that
have already been integrated in the mainline kernel (sometimes more simple or hackish
fixes than the mainline fix, but if the issue isn't fixed in mainline, then the fix doesn't go into
-stable either).

Stable kernel 2.6.32.8

Posted Feb 11, 2010 23:58 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

3 days prior to the release of 2.6.32.8 seems to me likely to be after the
three-day-long review period for 2.6.32.8 started. Patches don't normally
land in a given stable release after the review period starts for that
release unless specifically requested.

Stable kernel 2.6.32.8

Posted Feb 15, 2010 15:50 UTC (Mon) by BenHutchings (subscriber, #37955) [Link]

It does happen occasionally, e.g. if a reviewer points out a dependency of another patch, but in general you are right.


Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds