LWN: Comments on "Known-exploit detection for the kernel"

Known-exploit detection for the kernel

nix — Wed, 08 Jan 2014 16:38:58 +0000

Imagine that someone has found a way to exploit, say, procmail, or some other daemon run on the user's behalf that accepts code from the network and is ultimately invoked from the network.

The right thing to do in that situation is probably to halt mail delivery and just queue everything -- but your proposal would lock the entire account. An attacker that can determine what accounts exist (perhaps via said exploit) could then DoS-attack the entire system trivially.

(But, of course, if they can execute arbitrary code as one user they can probably do that anyway, in about a million ways, and probably get root too. So perhaps my concerns are unjustified. It might well elevate a failed breakin, via an exploit that doesn't actually work, to a partial DoS, but I'm finding it hard to be too worried about that.)

Known-exploit detection for the kernel

speedster1 — Tue, 07 Jan 2014 02:59:42 +0000

That makes no sense to me, perhaps you can explain more... are these DoS victims naive users who are running random things that script kiddies left for them? Or a daemon which has been tricked into running evil code? Either way, if a script kiddie gets to run stuff from an account, that account has been compromised and deserves to be locked out until it can be cleaned up. Maybe instead you are thinking of a way to lock out a non-compromised account with this feature?

Known-exploit detection for the kernel

nix — Mon, 06 Jan 2014 13:22:30 +0000

Locking an account *is* a DoS for the account's legitimate owner.

Known-exploit detection for the kernel

speedster1 — Mon, 06 Jan 2014 03:03:14 +0000

> The problem with that sort of thing is that as soon as it becomes popular enough that attackers can speculate that you might be running it (which is not very popular at all), it suddenly becomes a wonderful DoS tool.

There shouldn't be much DoS potential for script kiddies to abuse if there were a reliable mechanism for automatic account-locking like tshow wanted.

On the other hand, count me among those who predict this feature will quickly become worked-around by all the popular exploit kits -- at least on any systems lacking admins who are big enough on security to be running custom kernels with generic uname info. Those admins who do tweak their uname and hide /boot /lib/modules are probably not the ones who the kernel devs need to worry about protecting from script kiddies (their custom kernels probably include grsecurity...)

Known-exploit detection for the kernel

lamawithonel — Sun, 29 Dec 2013 08:06:44 +0000

this patch set uses the audit framework without any rate limiting, and in the somewhat more structured audit format. that sounds like what you want.

+ audit_log_format(ab, "exploit id=%s pid=%u uid=%u auid=%u ses=%u comm=",
+                  id, pid, uid,
+                  from_kuid(&init_user_ns, audit_get_loginuid(task)),
+                  audit_get_sessionid(task));

Known-exploit detection for the kernel

nix — Thu, 26 Dec 2013 22:22:31 +0000

... aand my point is proved. I'd rather bite my own nose off than work with people this nasty.

Known-exploit detection for the kernel

PaXTeam — Wed, 25 Dec 2013 23:56:48 +0000

did you move to the southern hemisphere recently? i can't think of many other places where it's hayfever season. keep up the nonsense and spender and i wish you a merry christmas (try without the drugs though).

Known-exploit detection for the kernel

nix — Wed, 25 Dec 2013 23:03:52 +0000

The problem with that sort of thing is that as soon as it becomes popular enough that attackers can speculate that you might be running it (which is not very popular at all), it suddenly becomes a wonderful DoS tool.

Known-exploit detection for the kernel

nix — Wed, 25 Dec 2013 23:02:14 +0000

If only some grsec features could be included in the kernel.

That seems unlikely to happen while the grsec people remain incapable of working with other people or taking criticism of any kind without blowing up like two-year-olds. (Apologies to my two-year-old niece, who blows up very rarely and is usually quite charming and sweet.)

Known-exploit detection for the kernel

dgm — Tue, 24 Dec 2013 09:22:33 +0000

The question is what percentage of the threats a Real World box is exposed to is "trivial". This is not a tool to make the OS invulnerable, but to improve the bottom-line.

Known-exploit detection for the kernel

drag — Tue, 24 Dec 2013 06:34:04 +0000

The most critical step in that is to have alarms that are actually meaningful and functional.

In most of those cases you mentioned you'd find that those alarms were disabled because they had so many false positives. A alarm that goes off once because a train left it's doors open is useless when it also goes off a thousand times when the doors are actually closed.

Known-exploit detection for the kernel

drag — Tue, 24 Dec 2013 06:30:01 +0000

> The question remains: will it detect some subset of unsophisticated attacks, enough to make it worth including, or not?

The social aspect of it is the problem.

Sure the kernel implementing features like this may succeed at detecting ham-fisted attacks, but the problem you run into is that people will naturally assume that this security feature has merit and will unfortunately actually try to depend on it.

People will write blog posts on how critically important it is to make sure you use a kernel version with this detection enabled in it and so on and so forth. It'll end up in Google's cache when people search 'make sure Linux is secure' or such things and then it'll just get increasingly stupid and self-defeating from then on.

You can see the same thing happen all over the place with various 'linux security features'.

A perfect example of this is things like 'chkrootkit'. People may end up with a hacked website and then install chkrootkit and run it thinking that somehow it will actually be effective at detecting root kits, which it is not. Or schemes that involve checking file checksums using 'rpm' utility. Or things like 'fail2ban' being used to 'secure' ssh, or port knocking, or any other number of silly and hokum things that people try to do to 'improve' their security.

Like anti-virus in Windows there is a actual limited useful application for some of this stuff, but most people are not able to know what that is.

Software security is already hard enough without creating features that add to the confusion with no real benefit.

Known-exploit detection for the kernel

drag — Tue, 24 Dec 2013 06:18:25 +0000

I am sure that there are other ways to determine what version of he Linux kernel that is running.

'uname -a' comes to mind.

This 'threat detection' is just one of those things that, if you are lucky, can be used to detect some old hack a script kiddie may try on your computer as they just randomly try different things. But any threat beyond the most trivial it would quickly become completely worthless.

Known-exploit detection for the kernel

dlang — Fri, 20 Dec 2013 18:47:14 +0000

> I'd want it for automated response, with a subsequent notification to the admin that the automated response has been made.

There are a whole lot more tools available to do the alerting based on syslog message than there are going to be if you invent a new protocol.

> If someone is script kiddying their way through a series of exploits to see what sticks, I want the userspace daemon to be able to log them out and ban their IP (and possibly lock their account) *now*, this microsecond, before they get a chance to try the next one. Yes, that does imply potential scheduling hackery to get the data out the exploit alert pipe to the userspace daemon before control is allowed to return to the offending process.

This requires much more significant changes, and really is going to require that you put your response code into the kernel inside the detection routine.

> In theory you'd only get flooded with alerts if the system was flooded with exploit attempts. You'd probably want to know if that was happening, but rate limiting could be done in the userspace daemon.

given that the effort involved in generating the log compared to the action that triggers the log, not having this be rate limited would result in a very good DOS vector.

> Feeding the userspace daemon through syslog is kind of silly; a dedicated pipe the daemon could connect to would be better. Avoiding syslog as a transport means:

> - being able to specify a consistent communication protocol

why can't you use a consistent protocol in your syslog message?

> - possibly being able to avoid parsing entirely, but at the least the protocol can be designed to make the parsing trivial

you can make the parsing trivial in syslog as well

> - not needing to worry about some other syslog output source being tricked into spoofing valid-looking but manufactured exploit logs

you can do this inside syslog with filters today (only send to this output if the source is the kernel and the message contains blah)

> - the daemon being able to sleep on data availability in the pipe rather than having to constantly chew on syslog and spit out the (1 - epsilon) it doesn't need, so the daemon will mostly just be asleep until its needed

again, with filtering in syslog, your userspace daemon can do this today.

And as I noted up at the top, there are a lot more alerting engines around that understand syslog than ones that will understand your new protocol.

Any significant company is already going to have an alerting tool in place, it's far better to integrate with that than to require a new tool be written.

Known-exploit detection for the kernel

tshow — Fri, 20 Dec 2013 16:21:38 +0000

I'd want it for automated response, with a subsequent notification to the admin that the automated response has been made.

If someone is script kiddying their way through a series of exploits to see what sticks, I want the userspace daemon to be able to log them out and ban their IP (and possibly lock their account) *now*, this microsecond, before they get a chance to try the next one. Yes, that does imply potential scheduling hackery to get the data out the exploit alert pipe to the userspace daemon before control is allowed to return to the offending process.

In theory you'd only get flooded with alerts if the system was flooded with exploit attempts. You'd probably want to know if that was happening, but rate limiting could be done in the userspace daemon.

I think you'd want to be able to differentiate between local and remote exploits; if someone logged in is trying an exploit it's a very different kettle of fish from someone trying a remote exploit against the external interface.

Feeding the userspace daemon through syslog is kind of silly; a dedicated pipe the daemon could connect to would be better. Avoiding syslog as a transport means:

- being able to specify a consistent communication protocol
- possibly being able to avoid parsing entirely, but at the least the protocol can be designed to make the parsing trivial
- not needing to worry about some other syslog output source being tricked into spoofing valid-looking but manufactured exploit logs
- the daemon being able to sleep on data availability in the pipe rather than having to constantly chew on syslog and spit out the (1 - epsilon) it doesn't need, so the daemon will mostly just be asleep until its needed

Known-exploit detection for the kernel

roblucid — Fri, 20 Dec 2013 15:10:17 +0000

I don't understand your objections, it's a fallacy to suppose that the guy who maintains this patch set, would necessary be capable of solving unknown kernel issues.

Reading logs manually is way too slow, so good sysadmins generate alarms filtering for where action may be required.

It might not be only Internet facing boxes, where for example remote or local root exploit attempts might be of interest. Engineers & other employees to get curious and try things, sometimes things they really are not supposed to. Just because the most clueful and careful attacker won't trip something doesn't mean a measure is useless.

Known-exploit detection for the kernel

tialaramex — Fri, 20 Dec 2013 10:52:12 +0000

The issue of ignoring alarms is a far more widespread one than can be covered in this thread, or indeed LWN at all.

In the maritime industry almost every other accident report will involve alarms that were disabled, non-functional, or ignored. In other transport industries it's less bad, but e.g. a London Underground driver overrode or disabled all the safety alarms in his train and was only alerted to the fact that he'd driven out of a station with the doors still open by the screaming of passengers. They couldn't alert him using the provided passenger alarm because he'd switched that off too.

Teaching people to investigate alarms rather than ignore them, and to test and maintain alarm systems so that they have confidence that the alarms reported are real is a bunch of work, but it's not impossible.

Known-exploit detection for the kernel

shmget — Fri, 20 Dec 2013 04:03:40 +0000

or the good old gentoo way of not leaving /boot mounted. ?
to mount /boot you need to be root.. and if you are root already what's the point....

Known-exploit detection for the kernel

PaXTeam — Fri, 20 Dec 2013 01:54:37 +0000

i'm not entirely sure whether you grasped the proposed feature and spender's criticism (the perfection strawman and silly real life non-examples indicate otherwise) but here it is in a nutshell:

detecting and reacting to incidents is a useful thing (grsec has been probably doing it for longer and more efficiently than most) but this feature will not achieve anything, not just because there're so many ways to know when a kernel is equipped with it or because the potential for false positives, but also because in real life nobody reacts to logs on systems where script kiddies are considered an actual threat (these are the lowest value systems, cf. the kernel.org compromise whose dmesg with the sign of a backdoor laid in public for weeks before anyone figured out that something wasn't right with it). and where they're not a threat, the remaining attackers are sophisticated enough to use 0-day to begin with (many of which the mechanisms in grsec will handle unlike the proposed feature).

Known-exploit detection for the kernel

tialaramex — Fri, 20 Dec 2013 00:57:22 +0000

"despite it being useless"

Like dusting for prints is useless, because criminals all wear gloves, right? Like it's pointless to own a burglar alarm because any "real" burglar would know how to disable it, right? You're actually making the same mistake the people you don't like usually make, assuming perfection where it's unwarranted.

Ordinarily it's the good guys who have to be perfect. One missed special case and you're exploitable. But in this sort of situation the roles are reversed, one slip by the bad guys and the alarms go off. _Everything_ they try has to be properly insulated so that it can't trigger this sort of alarm, because if just one thing isn't, and the relevant alarm is installed, game over.

Known-exploit detection for the kernel

Trou.fr — Thu, 19 Dec 2013 23:16:04 +0000

I perfectly understand the reasoning. But I think this is quite representative of the current mindset of developpers regarding security : no consideration for effective measure and it sometimes seems no consideration for actual security.

The single "recent" feature which led to actual security improvements in the kernel I can think of is seccomp-bpf, which is a brilliant and very efficient idea.

As you said, developpers would probably turn their attention to something else entirely. If only some grsec features could be included in the kernel.

Known-exploit detection for the kernel

dlang — Thu, 19 Dec 2013 20:45:05 +0000

you can feed it to your userspace daemon through syslog pretty darn fast, how fast do you think it needs to be acted on? and are you talking about an automated action or a manual action? if it's a manual action, the time taken to get a human's attention is FAR longer than the delay that syslog will introduce.

In addition, I'm not sure this is something you want to jump on every time (that the reason for rate limiting the alerts), you may just get flooded with alerts.

Known-exploit detection for the kernel

tshow — Thu, 19 Dec 2013 20:35:45 +0000

That's what I'm implying; it would be nice if this could export via some sort of file descriptor through which you could feed a userspace daemon, rather than shuffling stuff off to the logger. Logging is great, but if there's an exploit attempt I want to deal with it now, not after something gets done parsing the logs.

Known-exploit detection for the kernel

RobSeace — Thu, 19 Dec 2013 18:09:10 +0000

Sounds like something better suited to a userspace daemon parsing the kernel log messages and acting on them... Sort of like "Denyhosts" does now with failed SSH logins...

Known-exploit detection for the kernel

Funcan — Thu, 19 Dec 2013 16:30:28 +0000

A sufficiently advanced attacked can also break in and steal the log server. I doubt most people are facing that level of APT most of the time though...

Known-exploit detection for the kernel

tshow — Thu, 19 Dec 2013 16:25:39 +0000

It seems to me that rather than just sending something to syslog, this would be more useful if it could feed into userspace hooks and things like firewall rules. If I could rig exploit() to:

if(user is nonlocal)
{
log them out & lock the account
add their IP to the ban list
send the admin a "user tried exploit" message
}

I could see that being useful. Obviously there's potential pitfalls (people boobytrapping each other, or managing to get daemons kickbanned if the system isn't smart enough), but a tripwire hooked to a nuke can be a useful line in a layered defense.

Known-exploit detection for the kernel

dgm — Thu, 19 Dec 2013 16:06:51 +0000

> So the real workaround you propose is to determine the distro and kernel version and then check the intended exploits against it

Spender's code shows that it's actually easy to look for signs of fixes in the kernel image before trying any exploit, thus defeating the detection scheme.

This could be prevented by making the kernel image unreadable except for root, and in case of using SELinux, only for the bootloader tools.

Known-exploit detection for the kernel

vegard — Thu, 19 Dec 2013 14:39:23 +0000

Hi Brad,

I'm really grateful that we have people like yourself who are both competent and view security with a critical eye. Your feedback is appreciated. Keep up the good work!

Vegard

Known-exploit detection for the kernel

josh — Thu, 19 Dec 2013 14:32:41 +0000

> Some trivial static analysis (the likes of which I already do in enlightenment for other purposes) will defeat that "workaround" easily.

I'd be curious what kind of static analysis you have in mind.

> It's clear to me many people want more than anything to believe that this feature will work

I don't think anyone believes this mechanism will stop or detect sophisticated attacks. The question remains: will it detect some subset of unsophisticated attacks, enough to make it worth including, or not?

Known-exploit detection for the kernel

iq-0 — Thu, 19 Dec 2013 12:44:21 +0000

You seem to suggest that not doing this would change the focus of said kernel developers to do more kernel security improvements. That is rather wishful thinking, they would do something different but not necessarily that.

The idea itself is also not bad. It's the equivalent of doing authentication attempt logging and detecting possible attacks. Does it increase security? No. Does it tell you there might be something going on? Yes. Can attackers circumvent those checks? Yes. Does that make them less valuable? No, the attackers have to take more precautions, take more actions which might be detected or risk tripping an alarm.

An analogy would be to install a video cameras in your house. Any burglar could easily either circumvent them, disable them or ignore them and afterwards erase the tapes. But it makes live more difficult for them (even if slightly), might deter them to look for easier targets or increase the chance of detection because of all the steps involved. And there is a good chance some are not that diligent and get caught in the act.

You can never assume to be secure, you can only try and hope that it's as uneconomically possible for others to abuse the weaknesses.
But often it's just as important to just know somebody might have (tried) to do that, so you can take appropriate measures to minimize possible (indirect) damage.

Known-exploit detection for the kernel

spender — Thu, 19 Dec 2013 12:35:12 +0000

I gave you the exact file and function to look at, and it's clear you haven't even done that, so this will be my last post in this thread. You don't want to learn, you want validation that you could somehow wrap this turd of a feature with the appropriate metal lining to make it useful. It's just not going to happen.

What you propose (which is also what Ingo proposed) will not prevent invisible detection of the vulnerable kernel. I have *many* other methods I can use even in the event everything you mention is implemented.

This is my promise: if any form of this security theater makes its way upstream and into any distro, I will write any code necessary to expose it for what it is.

-Brad

Known-exploit detection for the kernel

spender — Thu, 19 Dec 2013 12:19:45 +0000

Some trivial static analysis (the likes of which I already do in enlightenment for other purposes) will defeat that "workaround" easily. Obviously I'm not going to waste time writing code for every possible change that could be made to the "feature" but the fact remains that there's no asymmetry of information here. Whatever eventually gets merged will be known to all and defeated. I guarantee it.

It's clear to me many people want more than anything to believe that this feature will work, regardless of any facts.

-Brad

Known-exploit detection for the kernel

zlynx — Thu, 19 Dec 2013 09:36:05 +0000

A really well informed attacker can try to jam the log server with nonsense UDP or TCP resets. He'd need access to the log server network of course.

If he can DOS the log server, it won't record anything except a pile of junk. Once he gets root he can kill -9 the log service, clean the logs and restart it.

Just another thing to watch out for.

Known-exploit detection for the kernel

Trou.fr — Thu, 19 Dec 2013 09:34:43 +0000

This is so stupid. I rather wish the kernel developers spend the time lost on actually improving the kernel security.
I find it baffling that even security-minded people like Kees Cook support the idea.

Known-exploit detection for the kernel

ncm — Thu, 19 Dec 2013 08:10:09 +0000

So the real workaround you propose is to determine the distro and kernel version and then check the intended exploits against it before attacking the real kernel. Maybe you use a table of all the likely kernel versions and their well-documented (in-)vulnerabilities.

Suppose, then, that in addition to making /boot and /lib/modules inaccessible, uname() is made to lie or be otherwise uninformative. Are you left with any alternative than to try out your catalog of exploits? If you try the newest one first, the kernel under attack might not have that fault inserted yet. If it fails, you have not exposed yourself, but you still have to try the older ones, and risk exposure. Or maybe any EINVALID return is logged, with the PC where it was noted, so the first try exposes you anyway.

Maybe you restrict yourself to faults that have always been there, and only recently patched, if at all. Then, either you get in silently, or you trip the alarm. This still seems better than being able to try out your whole catalog without worry of being noticed. No doubt I'm still missing something crucial.

Known-exploit detection for the kernel

dlang — Thu, 19 Dec 2013 07:50:24 +0000

the better shops also ship their logs off of the local systems so that attempts to scrub the logs will fail.

Known-exploit detection for the kernel

josh — Thu, 19 Dec 2013 07:17:24 +0000

You might want to add support for non-gzipped kernels, but yeah, it does look about that simple.

The obvious workaround would be to not put the CVE number in the kernel, but to instead just print "exploit attempt detected at $file:$line" and let it be decoded later. Still easily worked around if you're targeting a particular distro kernel, but if you're targeting a *particular* distro kernel anyway then you could just know which exploits to run against it.

Known-exploit detection for the kernel

noxxi — Thu, 19 Dec 2013 07:11:46 +0000

From my understanding the code is not to fight new exploits, but to detect if a known and fixed exploit was tried. And because the exploit was fixed the script kiddie will have no way to clean the logs (unless it has another and better exploit).
Thus the log entry can be used as a canary by the admin to detect if an account might be compromised and lock it down before worse exploits will be tried.

Known-exploit detection for the kernel

wahern — Thu, 19 Dec 2013 06:14:30 +0000

Script kiddies, by definition, don't need to worry about cleaning the logs. That's what their scripts are for. If the scripts don't do it now, they will. Script kiddies do the minimum possible. If you raise the bar, there's no reason to believe that it slows them down one iota.

Adding any security which can be easily circumvented is no security at all. All it takes is one person to write the circumvention code and to share it.

There are people doing the thankless job of actually preemptively scanning the code for vulnerabilities and fixing them. Those guys are priceless. Why they keep doing it when all the fame and adulation goes to this kind of stuff, port knocking, and other crazy schemes.... well, I wish their commitment could spread the same way interest in these schemes do.

These schemes only work as long as they're not widely adopted. Once they're widely adopted, they get added into the kiddie scripts. Then you're left with a bunch of useless code which only adds to your attack surface.

Known-exploit detection for the kernel

PaulWay — Thu, 19 Dec 2013 02:23:44 +0000

As I see it, this makes sense for sysadmins who:

* use kernels with the CONFIG_EXPLOIT_DETECTION turned on (either rolling their own or using a distro's enabled kernel)
* have systems that are likely to be probed by script-kiddies
* want to know if they're being probed
* watch their logs

Now, that isn't everyone - but it's a lot of people, including me. Even if I discover the probing after the fact, there's a good chance that the script-kiddie won't clean the logs, so I can then shut the system down and restore from a good backup.

So I for one think this a good idea.

Have fun,

Paul