|
|
Subscribe / Log in / New account

Dirty COW and clean commit messages

By Jonathan Corbet
October 21, 2016
We live in an era of celebrity vulnerabilities; at the moment, an unpleasant kernel bug called "Dirty COW" (or CVE-2016-5195) is taking its turn on the runway. This one is more disconcerting than many due to its omnipresence and the ease with which it can be exploited. But there is also some unhappiness in the wider community about how this vulnerability has been handled by the kernel development community. It may well be time for the kernel project to rethink its approach to serious security problems.

Dirty COW (which, naturally, has its own logo and web page, though this one is a bit on the satirical side) is a race condition in the kernel's memory-management subsystem. By timing things right, a local attacker can exploit the copy-on-write mechanism to turn a read-only mapping of a file into a writable mapping; with that, a file that should not be writable can be written to. It doesn't take much imagination to see how the ability to overwrite files could be used to escalate privileges in any of a number of ways.

The core code with the vulnerability is present in every Linux system (excluding, perhaps, no-MMU systems, which lack the sort of protection that has been defeated here in the first place). The known exploit depends on access to /proc/self/mem, which is not universally available, but, even on systems that do not provide that access, there are other ways to exploit the bug. The exploit happily bypasses almost every hardening technique out there; strong sandboxing with mechanisms like seccomp() might slow it down, but counting on that is probably not wise either. The only real protection is to upgrade to a kernel containing the fix.

Given that the nature of the vulnerability was known when the fix was applied, one might think that the development community would go out of its way to inform its users of the nature of the problem. The actual fix, when applied to the mainline, was grouped with a set of "cleanups"; Linus's merge commit for that set mentioned this fix at the very end: "Additionally, there's a fix for an ancient bug related to FOLL_FORCE and FOLL_WRITE by me." The fix itself is not much more illuminating:

This is an ancient bug that was actually attempted to be fixed once (badly) by me eleven years ago in commit 4ceb5db9757a ("Fix get_user_pages() race for write access") but that was then undone due to problems on s390 by commit f33ea7f404e5 ("fix get_user_pages bug").

The fix was (properly) rushed out into a set of stable updates, but there was no mention of the vulnerability there either. So users of the kernel are entirely reliant on others to inform them that an important vulnerability has been fixed and that they should really be updating their systems.

There is nothing new about this practice; Linus and others have long had a habit of, at best, neglecting to mention vulnerabilities that have been fixed in released kernels. There are a number of reasons given for operating this way, starting with a general disdain for the "security circus" and the industry that lives on responding to yesterday's vulnerabilities. Every kernel release fixes a great many serious bugs, they say, some of which certainly have security implications that nobody has (publicly) noticed yet. Highlighting specific vulnerabilities only draws attackers' attention to them while glossing over the fact that the only way to get all of the important fixes is to run the latest releases. Security bugs are just bugs, and we fix them like every other bug.

Everything expressed in that viewpoint could be said to be entirely true (though it glosses over the new vulnerabilities that can also come with the latest releases), but it is, in your editor's view, insufficient to justify this practice.

One of the core tenets of kernel development is that every change must describe the problem that is being solved and what the impact of the change will be. The core SubmittingPatches document, to which all developers are referred, says:

Describe user-visible impact. Straight up crashes and lockups are pretty convincing, but not all bugs are that blatant. Even if the problem was spotted during code review, describe the impact you think it can have on users.

Core reviewer Andrew Morton asks developers to add information about user-visible impacts so often (example) that one assumes he has it bound to a special key sequence. The idea that a patch should describe what it does, and why, is deeply ingrained in the community; it's the only way that we can judge patches properly, and it's the only way that developers will know why a change was made when they encounter it in the repository years later. It's also the only way for users and distributors to know how important a particular patch is known to be. Failing to document the security impact of patches violates that rule at a time when that information is needed the most.

When, for example, a kernel bug can cause random corruption and loss of data within a filesystem, the fix describes the problem and its consequences. That way, users and distributors understand the importance of the change. Security bugs are bugs too, and their consequences should be spelled out as well. The fact that developers are not always aware that a bug has security implications does not excuse omitting that information when it is available.

It has been a long time since all but the laziest of attackers grepped the changelogs for explicit mention of vulnerabilities. As a rule, they are far more interested in the changes that introduce the vulnerabilities in the first place, or those that quietly fix vulnerabilities that the original developer might not even realize are there. Suppressing information about security fixes may keep vulnerabilities out of the awareness of those who are not actively looking for them — the users who may be exploited via those vulnerabilities — but it doesn't fool the exploit creators.

What suppressing security-related information does do is this: it calls into question the community's seriousness about dealing with security issues in general. When users are not told about the known effects of bugs, they rightly assume that they will be unaware of other silently fixed problems. So they can never know that they have the fixes for all known security issues unless they run the absolute latest releases. Staying on the leading edge is not an option for a huge portion of our user base, so they are forced to live with the worry that important fixes have been hidden from them. That is not a situation that engenders confidence in the kernel in general.

To summarize: the suppression of information about the security implications of a change runs counter to the kernel community's principles, makes life harder for our users and distributors, does little or nothing to slow down attackers, and hurts the kernel's reputation in general. A practice that might have made sense (but probably didn't) against script kiddies in the early 1990s is clearly out of place in 2016. Your editor humbly suggests to the kernel development community that the time has long since come to reconsider how security-related patches are handled.

Index entries for this article
KernelDevelopment model/Security issues
SecurityBug reporting
SecurityLinux kernel


to post comments

Dirty COW and clean commit messages

Posted Oct 21, 2016 17:40 UTC (Fri) by Kamiccolo (subscriber, #95159) [Link] (2 responses)

I think that there is one more side of this story and argument for not leaving low hanging fruits for "script kidies" --- the vast amount of devices running ancient kernels without any option to patch and/or no proper maintainance and communication from manufacturers.

Dirty COW and clean commit messages

Posted Oct 21, 2016 18:00 UTC (Fri) by lambda (subscriber, #40735) [Link] (1 responses)

This issue has been all over the news due to active exploits in the wild. Hiding it from script kiddies clearly hasn't worked.

Furthermore, there are a large number of troubling attacks these days that are not merely from script kiddies, but from both sophisticated organized crime operations and nation-state level actors, who have the resources to monitor and analyze patches for security relevance. Not publicly announcing the severity makes it harder for users, administrators, and distros to learn about and prioritize deploying the fixes, while leaving the window open for exploitation by malicious actors who have noticed the relevance.

well funded super predators in the cyber jungle

Posted Oct 23, 2016 18:40 UTC (Sun) by Garak (guest, #99377) [Link]

+1, Snowden happened

Dirty COW and clean commit messages

Posted Oct 21, 2016 18:00 UTC (Fri) by ware (subscriber, #83607) [Link] (5 responses)

I appreciate the idea that the way security vulnerabilities in the Linux kernel are handled in a way that’s really out of the guidelines required for all other bugs. That said, I see nothing here that implies a better way of handling them. Are skilled security researchers looking at all the committs looking for changes that fix a security issue? Absolutely. Should the kernel community in their committ messages shoot up a flare identifying a security vulnerability so that everyone sees it? I don’t think so. The moment they do that, not only is everyone still vulnerabile until a distro update (or a recompile of your kernel if you’re rolling your own), but you’ve told everyone about it both good and bad. I’m sure the kernel development community would be happy to entertain specific proposals that cover these concerns.

The process that currently exists where distros get fixes out before security vulnerabilities are generally know usually works. It has broken down a bit in this particular case, but it generally works. I think what really needs to take place though is that the kernel community needs to be better about Responsible Disclouse once fixes are available. I don’t have any problems with fairly innocuous commit messages that are later updated to reflect the full significance of a security patch.

Embargos

Posted Oct 21, 2016 18:26 UTC (Fri) by corbet (editor, #1) [Link]

If the article reads like it's against embargoes, that was not the intent. Not everybody likes embargoes either, but I think a limited pre-disclosure time to get the updates together makes complete sense. That's unrelated to the topic at hand here; the commits and releases in question happened after the embargo ended.

Dirty COW and clean commit messages

Posted Oct 21, 2016 18:29 UTC (Fri) by smoogen (subscriber, #97) [Link] (1 responses)

It is simple. Just say what you know when you make the commit. If you know the problem has security implications, then say it. If you don't then don't. Trying to hedge around a bug that you know has security implications is ridiculous because of the following items.

1. There are hundreds of red teams out there tearing apart every kernel to see what advantage they can get. They will either figure it out when you close off a working exploit they already had or when they see the commit.

2. The vast number of linux users aren't going to be left out of the cold because there was no time window between the commit and it being know to fix a security problem. They are going to be left out because their phone company will never ship them an updated kernel... and no amount of information blackout window is going to fix that.

Dirty COW and clean commit messages

Posted Oct 22, 2016 1:33 UTC (Sat) by bojan (subscriber, #14302) [Link]

Spoken with such clarity and simplicity! +1

Dirty COW and clean commit messages

Posted Oct 21, 2016 22:35 UTC (Fri) by clemensg (guest, #94377) [Link] (1 responses)

The problem is that the security implications become public anyway. Also, if patches quickly find their way into the stable tree and if Linus himself authored them, I'd look at them if I were an attacker.
Security by obscurity does not work, as we can see.

The problem is not the guys who read the commit description of the fix, the problem is all those guys who already knew about the bug years before. Wasn't this bug discovered because it was actively exploited in the wild? And not the other way around..
Imagine how many 0-day exploits the NSA found and keeps secret and how much it bums them out if these expensive 0-day bugs are discovered by someone else and then publicly disclosed. :-)

Dirty COW and clean commit messages

Posted Oct 22, 2016 15:35 UTC (Sat) by raven667 (subscriber, #5198) [Link]

> Imagine how many 0-day exploits the NSA found and keeps secret

It would better for the world I think if the US Security services like the NSA put all their effort into disclosing bugs and closing holes rather than keeping them secret, even if the other worlds services do the opposite, as it will greatly raise the costs for the bad actors and maybe open up some conversations about code review, quality and serviceability from dispassionate experts rather than involved vendors. Making the Internet more hostile to bad actors by securing more systems is more worth the money we spend than hoarding exploits.

Dirty COW and clean commit messages

Posted Oct 21, 2016 18:03 UTC (Fri) by davidstrauss (guest, #85867) [Link] (3 responses)

I've posted an additional mitigation path via syscall filtering using service configuration:

https://medium.com/@davidtstrauss/mitigating-dirtyc0w-wit...

Dirty COW and clean commit messages

Posted Oct 22, 2016 2:42 UTC (Sat) by nnewton (subscriber, #40661) [Link]

Nice, was wondering if that would work. Thanks.

Dirty COW and clean commit messages

Posted Oct 23, 2016 16:07 UTC (Sun) by justincormack (subscriber, #70439) [Link] (1 responses)

madvise is used by a lot of things, at least some versions of glibc use it to return memory to the OS.

Dirty COW and clean commit messages

Posted Oct 23, 2016 17:02 UTC (Sun) by davidstrauss (guest, #85867) [Link]

You're absolutely right; use of madvise may be widespread. I did not test the mitigation against most popular daemons to see which use it directly or via something like glibc.

I do know that some daemons and runtimes have models for allocation or freeing that may not require it:

(1) PHP-FPM allocates whenever memory in a worker exceeds the pool already allocated for that process. It then periodically destroys and recreated the worker, freeing the worker's memory through means of the process not existing.

(2) Some runtimes like the JVM seem to allocate a pool, as needed, up to a specified size. They then manage garbage collection within that memory.

(3) Some alternative memory libraries (like but not including jemalloc) may avoid madvise calls to the point of not using them at all.

So, I figured it wasn't impossible to work for some people. I mostly wanted to talk about syscall whitelisting, though.

Dirty COW and clean commit messages

Posted Oct 21, 2016 22:21 UTC (Fri) by mst@redhat.com (subscriber, #60682) [Link] (3 responses)

> Your editor humbly suggests to the kernel development community that the time has long since come to reconsider how security-related patches are handled.

As all problems in computer science we can solve this
with another level of indirection.
In particular report the issue in the kernel bugzilla.
I would be very surprised if there will be any resistance
to listing the bugzilla link in the commit log.
CVE and other info can be attached to the bugzilla
entry when and if it becomes available.

Dirty COW and clean commit messages

Posted Oct 23, 2016 7:16 UTC (Sun) by error27 (subscriber, #8346) [Link] (2 responses)

It has been a long time since I cared but back in the day red hat marked all these bugs as access denied. It's rarely useful for reviewing CVEs. I use the Ubuntu CVE list.

Anyway, in this case the bug was first reported to security@kernel.org then everyone was like this is important and we should do coordinated release through linux-distros. There was no thought about hiding it. Everyone expected that this was a webpage and logo bug because it was so serious.

Dirty COW and clean commit messages

Posted Oct 25, 2016 21:00 UTC (Tue) by lacos (guest, #70616) [Link] (1 responses)

> It has been a long time since I cared but back in the day red hat marked all these bugs as access denied.

MST was most likely not talking about the Red Hat Bugzilla, but the upstream kernel Bugzilla, at https://bugzilla.kernel.org/.

Dirty COW and clean commit messages

Posted Oct 27, 2016 0:25 UTC (Thu) by ajdlinux (subscriber, #82125) [Link]

Which subsystems actually use the kernel bugzilla actively? I know mine (powerpc) doesn't.

Dirty COW and clean commit messages

Posted Oct 22, 2016 0:14 UTC (Sat) by pj (subscriber, #4506) [Link] (15 responses)

>When users are not told about the known effects of bugs, they rightly assume that they will be unaware of other silently fixed problems. So they can never know that they have the fixes for all known security issues unless they run the absolute latest releases.

I think you strike closest to Linus' rationale here. There are always more 'silently fixed problems' to both known and unknown security issues; the only benefit to calling out fixes to known security issues would be to give a false sense of security that 'all known bugs are fixed' - even though many other security issues may lie dormant and undiscovered in the exact same code. Even the latest release with 'no known bugs' may be turned into 'most hackable version ever' the next day by the exposure of some heretofore unknown exploit.

>Staying on the leading edge is not an option for a huge portion of our user base, so they are forced to live with the worry that important fixes have been hidden from them.

No fixes are hidden - they're all in the leading edge. If people can't keep up with the leading edge, they likely can't upgrade at all, so sorting out 'important' from 'unimportant' fixes is moot.

Dirty COW and clean commit messages

Posted Oct 22, 2016 5:53 UTC (Sat) by marcH (subscriber, #57642) [Link]

> If people can't keep up with the leading edge, they likely can't upgrade at all,

There's not just one but a few worlds in the interval between kernel developers' workstations and dirt cheap, throw-away embedded systems. What's there is typically called "products".

Dirty COW and clean commit messages

Posted Oct 22, 2016 8:18 UTC (Sat) by PaXTeam (guest, #24616) [Link] (8 responses)

> the only benefit to calling out fixes to known security issues would be to give a false sense of security that 'all known bugs are fixed' [...]

first of all, i don't think you meant to say that as is, because if true it'd mean that some of the *known* bugs are *not* fixed and i'm sure i wouldn't be the only one who'd like to know just what those known but unfixed bugs are.

second, assuming you just meant 'all bugs are fixed', is that a statement that you can back up with evidence? i've seen this 'argument' before and every time i asked for proof, i got exactly nothing. will you fare better? ;)

also, let me ask you about another variant of your statement:

> the only benefit to calling out fixes to known file system corruption issues would be to give a false sense of security that 'all file system corruption bugs are fixed' [...]

would you agree with the above statement too? if yes, then your kernel developers are doing the exact opposite, if no, you have a nice contradicton on your hands...

> No fixes are hidden - they're all in the leading edge.

can you give me a list of filesystem corruption fixes that went into 4.9-rc1? remember, by your own claims, you must be able to give me an exhaustive list. if you miss one you'll have defeated your own argument.

> If people can't keep up with the leading edge, they likely can't upgrade at all[...]

so you're saying that the only way to upgrade (a kernel) is to update to the leading edge. why do the -stable trees exist then? why do vendor kernels exist? is Red Hat going out of business?

PS: when you go see a doctor, do you expect her to *not* tell you the disease(s) she identified on grounds that she may very well have missed some? because it's that kind of crazy that you're advocating for.

Dirty COW and clean commit messages

Posted Oct 22, 2016 17:28 UTC (Sat) by flussence (guest, #85566) [Link] (5 responses)

>PS: when you go see a doctor, do you expect her to *not* tell you the disease(s) she identified on grounds that she may very well have missed some? because it's that kind of crazy that you're advocating for.

Since we're on the subject of tortuous security circus analogies, and you're here.

When I go to a doctor, they usually act in my best interests. The medical science world has the Hippocratic Oath, and practising doctors are not ego-driven sociopaths for the most part. Not even Red Cross volunteers.

If a clinic were to prescribe preventative medication and it turns out to cause a near-fatal side effect in 99% of patients, it would be very odd if anyone who tried to warn the doctor and public of this, along with anyone else who received the news, received a lifetime ban from the clinic. That would be a cut and dry case of medical malpractice, in fact.

I shouldn't have to spell out the correct way to handle such a colossal fuckup is to apologise. We're all mature adults here, right? Right?

Even in an alternate universe where the law offers no checks and balances for such actions, it would still be a career-ending move once people start wondering what other horrors the mad doctor is trying to hide from his test subjects.

Food for thought, @grsecurity.

Dirty COW and clean commit messages

Posted Oct 22, 2016 18:19 UTC (Sat) by PaXTeam (guest, #24616) [Link] (4 responses)

> Since we're on the subject of tortuous security circus analogies, and you're here.

and i lost you about here too. can you write up your thoughts in a somewhat more coherent way that mere mortals can make sense of as well?

Dirty COW and clean commit messages

Posted Oct 24, 2016 9:41 UTC (Mon) by FLHerne (guest, #105373) [Link] (3 responses)

I assume this is about the Twitter thing a few months ago, where whoever controls @grsecurity tried to suppress a bug far more explicitly than the upstream kernel ever has.

Someone tweeted "How to panic a current @grsecurity kernel as any user: $ script /dev/null </dev/zero (seriously, WTF)" and you:
- Blocked his account.
- Blocked his IP from your website.
- Blocked the accounts (and IPs?) of anyone else mentioning the bug, or the blocking.
- Removed @grsecurity and all its past tweets.

Naturally, the Streisand effect put it all over the Internet.

With that in recent memory, your complaint that _upstream_ handles security bugs poorly -- while IMO entirely true -- seems rather hypocritical.

Dirty COW and clean commit messages

Posted Oct 24, 2016 10:49 UTC (Mon) by PaXTeam (guest, #24616) [Link] (2 responses)

> [...]@grsecurity tried to suppress a bug far more explicitly than the upstream kernel ever has.

i see, so let's do some fact checking then. the bugreport is here: https://forums.grsecurity.net/viewtopic.php?f=3&t=4342 and you tell me what was suppressed there.

> and you:[...]

sorry to burst your bubble, but i didn't do any of that, it was spender's doing and he was simply cleaning up the trash (minus the IP banning which was obviously useless). as for the triggering tweet for all that shitstorm, it wasn't the one you cited (that was merely poking fun at him, or rather, us, considering it was my bug to begin with, not spender's) but the later outright *lies* that marcan spread. that had to be countered and it was, quite successfully in fact (he even came on IRC to explain himself and failed at it, perhaps someone who logged the channel can share it with the rest of us). he's still so upset that i exposed him as a liar that he's been trying to 'get even' ever since, last time he tried to sabotage the upstreaming of gcc plugins by fake licensing 'concerns' that too failed.

> With that in recent memory, your complaint that _upstream_ handles security bugs poorly -- while IMO entirely true -- seems rather hypocritical.

what's hypocritical in 1. acknowledging the bug, 2. describing its nature, 3. fixing it without lying about it? if we had failed at any of those steps then you'd have a point but as the facts show, we didn't.

Dirty COW and clean commit messages

Posted Oct 26, 2016 22:08 UTC (Wed) by nix (subscriber, #2304) [Link] (1 responses)

sorry to burst your bubble, but i didn't do any of that, it was spender's doing and he was simply cleaning up the trash (minus the IP banning which was obviously useless)
*BOGGLE*. I don't see how anyone can seriously characterize a mass of account blocking and deletion as 'cleaning up the trash'. It was utterly obviously a panicky cover-up to anyone older than the age of five. (And anyone over the age of ten could predict its inevitable failure.)

Dirty COW and clean commit messages

Posted Oct 27, 2016 17:41 UTC (Thu) by Arach (guest, #58847) [Link]

> It was utterly obviously a panicky cover-up

No sht, Sherlock. That's why the original bug report is still on the forum. And that's why spender stopped posting on twitter. Because all about it was is a cover-up, obviously.

Dirty COW and clean commit messages

Posted Oct 24, 2016 13:31 UTC (Mon) by pj (subscriber, #4506) [Link] (1 responses)

>second, assuming you just meant 'all bugs are fixed', is that a statement that you can back up with evidence? i've seen this 'argument' before and every time i asked for proof, i got exactly nothing. will you fare better? ;)

You're correct, I meant to say "the only benefit to calling out fixes to known security issues would be to give a false sense of security that 'all bugs are fixed'". As for 'proof'... prove what? That if you say you've fixed 'all known bugs' then unsophisticated users think that means 'all bugs'? I think that's fairly self-evident.

>also, let me ask you about another variant of your statement:
>> the only benefit to calling out fixes to known file system corruption issues would be to give a false sense of security that 'all file system corruption bugs are fixed' [...]
>would you agree with the above statement too? if yes, then your kernel developers are doing the exact >opposite, if no, you have a nice contradicton on your hands...

Are they? Any number of memory corruption or other types of bugs could cause file system corruption (in the sense of bad data making it to disk), but I don't think the fs developers list them all as 'file system corruption bugs'. It's an interesting analogy though; I think one distinction it has from 'security bugs' is that bugs in filesystem code could corrupt data in the normal sequence of events, while security bugs take a malicious actor.

>> No fixes are hidden - they're all in the leading edge.
>can you give me a list of filesystem corruption fixes that went into 4.9-rc1? remember, by your own claims, you must be able to give me an exhaustive list. if you miss one you'll have defeated your own argument.

Sure, every fix is a filesystem corruption fix, so the list of fixes is identical to the list of commits. If it's not in the leading edge, it's not production ready, having not passed muster with enough maintainers to make it into the leading edge.

> If people can't keep up with the leading edge, they likely can't upgrade at all[...]
so you're saying that the only way to upgrade (a kernel) is to update to the leading edge. why do the -stable trees exist then? why do vendor kernels exist? is Red Hat going out of business?

I'm saying that these days there's little point in upgrading to anything other than the latest version - it has all the fixes and all the hardware support.

-stable trees exist, as far as I can tell, to provide an unchanging base against which to do longer-term development work before then forward-porting said work to mainline.

vendor kernels exist to be able to pull in non-mainline code, or to standardize build options across said vendor.

Whether or not RH is going out of business is a non-sequitur.

>PS: when you go see a doctor, do you expect her to *not* tell you the disease(s) she identified on grounds that she may very well have missed some? because it's that kind of crazy that you're advocating for.

I disagree; I think that when I go see a doctor, I expect her not to tell me the disease(s) she identified _that have already been treated_, which in this tortuous analogy is the analogue for bugs already fixed.

Dirty COW and clean commit messages

Posted Oct 25, 2016 8:51 UTC (Tue) by PaXTeam (guest, #24616) [Link]

> That if you say you've fixed 'all known bugs' then unsophisticated users think that means 'all bugs'? I think that's fairly self-evident.

yes, provide proof of that statement as it's not self-evident. just so you don't diverge into grandma territory, the users of commit messages are people whose task is to build kernels for other people (users of distros and customers of companies). are you calling them unsophisticated? on what grounds?

> Are they?

yes, e.g., b8927721ae9d5ac0582d29d7b8c267d465ad5f00 quite explicitly says 'Fix bugs that could cause kernel deadlocks or file system corruption' so clearly they don't think that your 'unsophisticated users' may mistake such statements as saying that other commits can't possibly fix similar issues.

> I think one distinction it has from 'security bugs' is that bugs in filesystem code could corrupt data in the normal sequence of events,
> while security bugs take a malicious actor.

security bugs can also be triggered accidentally, there's no need for a malicious actor.

> Sure, every fix is a filesystem corruption fix.

then explain how the bug fixed by commit 3fc6a642e4355abef986b2dd11672216fb18212e can cause file system corruption.

> If it's not in the leading edge, it's not production ready, having not passed muster with enough maintainers to make it into the leading edge.

that's a very interesting take (though not in your favour as you'll see), so hold that thought ;).

> I'm saying that these days there's little point in upgrading to anything other than the latest version - it has all the fixes and all the hardware support.

and all the new bugs and regressions that didn't exist before.

> -stable trees exist, as far as I can tell, to provide an unchanging base against which to do
> longer-term development work before then forward-porting said work to mainline.

interesting, my reading of Documentation/stable_kernel_rules.txt doesn't at all suggest that, can you submit a patch to clear up the new rules?

> vendor kernels exist to be able to pull in non-mainline code, or to standardize build options across said vendor.

vendor kernels exist to satisfy the needs of paying customers, including out-of-tree code is orthogonal to that (and usually frowned upon).

in any case, putting together your statements you're basically saying vendor kernels have not passed muster and there's little point in upgrading to them.

> Whether or not RH is going out of business is a non-sequitur.

as you can see, you drew that conclusion yourself by virtue of calling their kernels inadequate for use which means people should abandon them driving such vendors out of business. the fact that it's not happening should tell you something about your premises...

> > PS: when you go see a doctor, do you expect her to *not* tell you the disease(s) she identified on grounds that she may very well have missed some?
> I disagree; I think that when I go see a doctor, I expect her not to tell me the disease(s) she identified _that have already been treated_,
> which in this tortuous analogy is the analogue for bugs already fixed.

no, the analogy is this: bug = disease, bugfix = cure, commit message = disease *and* cure description. you're arguing that describing certain details of the particular disease is bad because that would somehow mislead people.

Dirty COW and clean commit messages

Posted Oct 23, 2016 21:59 UTC (Sun) by drag (guest, #31333) [Link] (3 responses)

> the only benefit to calling out fixes to known security issues would be to give a false sense of security that 'all known bugs are fixed' - even though many other security issues may lie dormant and undiscovered in the exact same code.

The benefit to having security fixes called out is that they get priority to getting patched by end users.

I deal with a large number of systems and for the most part they are pretty 'steady state'. Meaning they run on pretty much the same hardware using the same virtual machine technology running the same OSes and run similar software written with similar libraries and languages in similar ways.

If there was something like a file system corruption bug that could of affected us it would of already caused problems for us. So even if there was a bug fix available for a issue like a file system corruption that then there isn't going to be a huge business reason to fix that. If a bug doesn't affect us it doesn't affect us; period end of story. So it'll get fix next round of scheduled OS updates. No need for disrupting business, operations, or developers with kernel updates.

However unlike most of our servers the external threat isn't so 'steady state'. Security threats are ever evolving and ever changing. Things like CVEs and dorky websites and articles being written make it much easier for us to justify the disruption and downtime kernel updates cause.

I also go out of my own way to keep track of security issues, but it's done on my own time. Most systems engineers I work with have a half a dozen different projects they work on and really have no time to be scanning mailing lists and such things for hidden security problems. We depend very much on the honesty of our OS vendors.

Of course we practice 'security in depth', just like we back up our important data just in case of some yet unknown bug or hardware failure. I understand that all bugs are potentially security bugs, but that doesn't mean that it's a waste of time to update against known security problems.

If OS vendor is not being honest then that increases our risk and increases our cost. It may be useful for their marketing, but it makes it more difficult to justify spending money on support licenses.

Dirty COW and clean commit messages

Posted Oct 24, 2016 19:27 UTC (Mon) by pj (subscriber, #4506) [Link] (2 responses)

Shouldn't you be upgrading continuously, anyway? And not just when "oh no, bad bug!" ? In order to get not only bug fixes but also performance fixes and new useful features?

>the disruption and downtime kernel updates cause.

...should be mitigated by having a non-disruptive method to doing so. "What you do often, you get good at." So get good at doing kernel updates by figuring out a non-disruptive way to do them (testbed, slow-rollout, maintenance window, etc) and then doing them regularly.

Dirty COW and clean commit messages

Posted Oct 24, 2016 22:24 UTC (Mon) by drag (guest, #31333) [Link] (1 responses)

> Shouldn't you be upgrading continuously, anyway?

We upgrade at regularly scheduled intervals. I am not going to tell you those intervals.

If you are talking about running 'while true; do yum -y update ; sleep 30; done'... no I am not going to do that.

> ...should be mitigated by having a non-disruptive method to doing so.

The non-disruptive way is to it at normally scheduled intervals. We don't update the entire company in one go, however, if that is what you are talking about.

I can be disruptive, however, if I need to be. Important security updates are important.

Dirty COW and clean commit messages

Posted Oct 25, 2016 4:09 UTC (Tue) by spender (guest, #23067) [Link]

I'm sure everyone in the world updated to 4.8.4 immediately, 3 days after having already updated to 4.8.3. This of course happens in the same fantasy world people like PJ live in where the latest kernels only bring new fixes, and not new bugs.

Meanwhile in the real world, anyone can look at the following patches generated over the same timeframe:

diffstat patch-4.8.1-2 | tail -n 1
52 files changed, 487 insertions(+), 213 deletions(-)

diffstat patch-4.7.7-8 | tail -n 1
48 files changed, 425 insertions(+), 176 deletions(-)

diffstat patch-4.4.24-25 | tail -n 1
27 files changed, 313 insertions(+), 137 deletions(-)

and see the same thing any honest person who does any backporting work knows: that the majority of bugs being fixed in the latest kernels are for recently introduced bugs. If Greg were an honest person, he would admit this. But I realize he's got a lot of conflicting interests to juggle: to take money from the Linux Foundation to create the stable kernels, to keep people doing QA on the latest kernels by telling them the latest upstream kernel has the most fixes, to tell people not to use the old distro kernels because of a fake story he concocted about the exploitability of a tty bug, and to ensure nobody takes him seriously by intentionally covering up vulnerabilities for no reason other than a failed attempt at propping up the facade of "a bug is a bug".

One other thing I find interesting that nobody talks about is that a lot of the unpublicized security fixes Greg backports land in his -stable kernels months after they appear upstream (visible to defenders and attackers alike). Other LTS trees that depend on Greg's solely for backports are even worse. I know this from seeing merged fixes months later from things I've already backported, or for issues I've reported/been cc'd on as I then get automated mails from the stable maintainers. It's interesting how much belief is happening and how little critical thinking goes on, because I'm not sure how people can merge those facts with Greg's propaganda into a straightforward narrative. Pay no attention to the man behind the curtain for now, maybe in a couple years LWN will repeat everything I've said (much like the article I'm commenting on now) and everyone will have a sudden epiphany.

-Brad

Dirty COW and clean commit messages

Posted Oct 24, 2016 7:15 UTC (Mon) by mjthayer (guest, #39183) [Link]

> No fixes are hidden - they're all in the leading edge. If people can't keep up with the leading edge, they likely can't upgrade at all, so sorting out 'important' from 'unimportant' fixes is moot.

Brings the evil word "micro-kernel" to mind - especially after the article on user-space drivers.

Dirty COW and clean commit messages

Posted Oct 23, 2016 4:22 UTC (Sun) by citypw (guest, #82661) [Link] (2 responses)

Some exploitable bugs seems unlikely to exploit in an easy way, e.g: CVE-2014-9322 so called "BadIRET", it was fixed by kernel community in Nov 23 2014:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linu...

Coundn't find any public discussion in English world until nergal's write-up in Feb 2 2015:

https://labs.bromium.com/2015/02/02/exploiting-badiret-vu...

Then we realized there was a public PoC( it may be the 1st one) released in a .ru forum in Dec 17 2014:

https://rdot.org/forum/showthread.php?t=3341

According to nergal's write-up, the most people think it's hard to exploit. But the exploit writer always trying harder 2 be themselves;-)

http://blog.pi3.com.pl/?p=509

Thanks to pi3's work and we know how strong defense of PaX/Grsecurity does in the most cases.

There are tons of kernel vulnerabilities doesn't have a brand name( yet?) like CVE-2016-5195( Dirty*F*Cow) and it doesn't mean they can't be exploited. KSPP is trying to follow what PaX/Grsecurity does for a decade, which is killing a class of bug/vuln by adding mitigation. We've been suffering from the philosophy of "a bug is bug" for too long. Hope this dirty fuc* cow can change the situation of "security through obscurity" in linux kernel community. Like corbet said, we are living in 2016 but 90s;-)

Dirty COW and clean commit messages

Posted Oct 24, 2016 7:12 UTC (Mon) by mjthayer (guest, #39183) [Link] (1 responses)

> Then we realized there was a public PoC( it may be the 1st one) released in a .ru forum in Dec 17 2014:

https://rdot.org/forum/showthread.php?t=3341

But if I understand right, this shows that there is no real problem. People wanting to know which patches to apply (and budding script kiddies) should simply follow .ru forums rather than the kernel change log. (Interesting for a non-Russian from a linguistic point of view, from "инстант ребут", to wondering what "poc:" meant before realising it was Latin script.)

Dirty COW and clean commit messages

Posted Oct 25, 2016 16:22 UTC (Tue) by sorokin (guest, #88478) [Link]

For those who are not familiar with Cyrillic script: "инстант ребут" is a transliteration of "instant reboot" in english. Transliterated English words are commonly used by programmers in Russia. I believe about 75% of the words mentioned in the referred page are so. To name a few: "фикс", "баг", "федора", "эксплойт", "РУТануть".

Rolling LWN articles

Posted Oct 23, 2016 13:20 UTC (Sun) by swilmet (subscriber, #98424) [Link]

A bit off-topic, but it's nice to have such LWN articles released early, not necessarily on the Thursday's Weekly Edition. In the past there were already articles released another day, but it seems that it is now more frequent after Some news from LWN in August.

So, I wanted to report a positive feedback from this change, thanks :)

Dirty COW and clean commit messages

Posted Oct 24, 2016 15:58 UTC (Mon) by jezuch (subscriber, #52988) [Link]

Well, I'm no security researcher nor a black hat hacker, and still I immediately knew something's up with that commit... by how small the stable release was :) It almost screamed "this is important!".

KSM and Unix Shared Memory

Posted Oct 25, 2016 16:32 UTC (Tue) by jhhaller (guest, #56103) [Link]

We ran into a bug where we built a semaphore in shared memory (the old Unix shared memory shm_* API). When we enabled Kernel Same Page Merging (KSM), and ran the code in a couple of VMs, the semaphore sometimes broke. Not needing KSM and not having time to investigate, we just turned off KSM. But, it does make me wonder if we ran into the same issue. The page with the semaphore was generally zero, so likely to be selected for KSM with some other zero page. KSM operates by making the shared page read-only, so it would handled in a way that would bring it close to the broken Dirty Cow code.

Dirty COW and clean commit messages

Posted Oct 25, 2016 17:35 UTC (Tue) by vapier (guest, #15768) [Link]

> (excluding, perhaps, no-MMU systems, which lack the sort of protection that has been defeated here in the first place)

this is a common misconception/conflation. lack of virtual<->physical translation does not mean there is no memory protection. a number of embedded no-MMU CPUs have memory protection (MPUs) which means they can protect kernel/user space from each other, and protect processes from each other.

https://blackfin.uclinux.org/doku.php?id=bfin:mpu


Copyright © 2016, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds