Disallowing perf_event_open()
An Android security measure that limits the ability of processes to access the perf events subsystem ran up against some, perhaps surprising, resistance when it was recently proposed for the mainline. The patch simply provides another setting for the kernel.perf_event_paranoid sysctl parameter to disallow unprivileged processes from accessing the perf_event_open() system call at all. It is currently used in both Android and Debian kernels, but some kernel developers see it as too much of a "big hammer" approach.
Jeff Vander Stoep posted the patch on July 27. It adds a another value that can be set for the sysctl parameter (i.e. kernel.perf_event_paranoid=3) that restricts perf_event_open() to processes with the CAP_SYS_ADMIN capability. Currently, perf_event_paranoid is set to 2 by default, which disallows access to some perf features (raw tracepoint access, CPU event access, and kernel profiling) to processes without the proper capabilities; the patch does not change the default. He also submitted another patch that would allow configuring the kernel to make 3 be the default perf_event_paranoid value.
In the first patch, he noted five vulnerabilities worthy of CVE numbers that have recently been found in perf and argued that allowing access to it increases the attack surface of the kernel. For production kernels, that may not make sense, so the patch is intended to allow administrators to restrict access to perf, while still providing a means for developers and others to access the tool as needed (by granting CAP_SYS_ADMIN). The patches are based on the grsecurity PERF_HARDEN feature and were first proposed by Ben Hutchings back in January. At that time, he said it had been running in the Debian kernel since August 2015 with no complaints.
It is a fairly simple and straightforward change, but Peter Zijlstra objected that
providing a way to turn off perf because of some bugs was heavy-handed:
"We have bugs we fix them, we don't kill complete infrastructure
because
of them.
" He also thought that it would inhibit new and innovative
uses for the tool:
I would much rather have an LSM hook where the security stuff can do more fine grained control of things. Allowing some apps perf usage while denying others.
Daniel Micay noted that the functionality would still be available to privileged processes and that Android will allow access by unprivileged processes, but that capability must be enabled from the adb shell. Furthermore:
This patch is now a requirement for any Android devices with a security patch level above August 2016. The only thing that not merging it is going to accomplish is preventing a mainline kernel from ever being used on Android devices, unless you provide an alternative it can use for the same use case.
Micay was skeptical that an LSM-based approach would work, as was Kees
Cook, who said: "I'm not against an
LSM, but I think it's needless complexity when
there is already a knob for this but it just doesn't go 'high' enough.
" He also noted that bugs live a long time ("an
average of 5 years from introduction to fix
") and they can last even
longer when you take product update lifecycles into account. He argued
that administrators need the ability to reduce the attack surface of their
systems:
Now, obviously, these API have huge value, otherwise they wouldn't exist in the first place, and they wouldn't be built into end-user kernels if they were universally undesirable. But that's not the situation: the APIs are needed, but they lack the appropriate knobs to control their availability.
The use case Zijlstra mentioned might be a good reason to change the value of the setting, Cook said, but there are other use cases where administrators want to be able to reduce their systems' attack surface when running a pre-built kernel. But Zijlstra disagreed; he is concerned that having this knob available will mean that administrators blindly apply it. That would have the effect of stopping the development of use cases like he described:
Cook was undeterred, however, saying that
the feature is based on a risk analysis of the attack surface, and that
there are
"hundreds of
millions of end-users for whom perf is not needed
". Beyond that,
though, Zijlstra's argument assumes that the knob is not available, but
that simply isn't true:
Ingo Molnar agreed with Zijlstra that the
approach was "too coarse
". He suggested that perf is not just
a "narrow debugging mechanism
" that can simply be turned off,
but that it is a "wide
scope performance measurement and event logging
infrastructure that is being utilized not just by developers but by apps and
runtimes as well
".
Micay pointed out that the wide scope of
perf is part of its problem from a security perspective. Because it has
been a "frequent source of vulnerabilities
", it has been
disabled by some distributions. Part of the problem also lies outside of
the core kernel, he said: "It's extended by lots of vendor code to
specific to platforms too, so it
isn't just some core kernel code that's properly reviewed.
"
The coarseness of the setting also concerned Eric W. Biederman. He suggested
that many of the features to reduce the attack surface amount to a
"system wide off switch
" for features like user namespaces and
perf. The result is that new applications cannot take advantage of these
features, which turns the attack-surface reduction into "great big
denial of
service attacks on legitimate users
". He also suggested several ideas
for ways to make the feature less coarse:
"I vote for sandboxes. Perhaps seccomp. Perhaps a per userns sysctl.
Perhaps something else.
"
That's about where things stand at this point. The second patch to allow configuring the kernel to default to denying access to perf has seemingly been dropped. The first will undoubtedly live on in grsecurity, Android, and Debian (at least), which seems to undermine the concern that Zijlstra, Molnar, and Biederman have—as Cook said, the change has already happened in some places. Whether a more fine-grained approach emerges remains to be seen, but it is a little hard to see who would work on it. Distributions already have their solution at this point.
Index entries for this article | |
---|---|
Kernel | Performance monitoring |
Kernel | Security/Kernel hardening |
Security | Linux kernel/Hardening |
Posted Aug 4, 2016 12:52 UTC (Thu)
by deater (subscriber, #11746)
[Link] (9 responses)
At the same time, the perf_fuzzer can still reliably crash any x86 machine within a few hours despite years of trying to get that fixed, so it probably is a good idea to provide a way to block the syscall.
I see both sides of the issue here, as having perf_event disabled by default is really going to cause annoyance for high-performance computing users (it hasn't started problems yet because most users aren't using Debian). If you start requiring root or sysadmin action in order for perf to work, people are going to go back to skipping perf altogether for tools like LIKWID (they also usually require root or sysadmin intervention).
Posted Aug 4, 2016 20:01 UTC (Thu)
by nix (subscriber, #2304)
[Link]
I have no idea why there isn't a config knob to compile perf out entirely. There surely should be. Not everything needs it; not everything can benefit; and those things that don't are purely harmed by its distinctly non-zero-sized vulnerability surface.
Posted Aug 4, 2016 22:51 UTC (Thu)
by ballombe (subscriber, #9523)
[Link] (7 responses)
Posted Aug 5, 2016 0:17 UTC (Fri)
by deater (subscriber, #11746)
[Link] (6 responses)
I always forget that most people do not run unstable.
Posted Aug 5, 2016 0:57 UTC (Fri)
by creemj (subscriber, #56061)
[Link] (1 responses)
The pain level has to be quite high before I would report a bug or issue back to Debian --- I hate the reportbug interface and find 'apt-get purge crappy-buggy-package' a much easier and quicker solution in most cases.
I am wondering, however, whether there is a kernel option to change the default? That would be a neater solution.
Posted Aug 5, 2016 4:36 UTC (Fri)
by nybble41 (subscriber, #55106)
[Link]
Why a systemd init config? Isn't this exactly why we have /etc/sysctl.conf and /etc/sysctl.d/?
# echo "kernel.perf_event_paranoid = 0" > /etc/sysctl.d/perf_events.conf
Posted Aug 6, 2016 17:35 UTC (Sat)
by anton (subscriber, #25547)
[Link] (3 responses)
Posted Aug 6, 2016 18:06 UTC (Sat)
by spender (guest, #23067)
[Link] (2 responses)
-Brad
Posted Aug 8, 2016 15:09 UTC (Mon)
by deater (subscriber, #11746)
[Link] (1 responses)
The fun part is that the serial console only manages to print
I'll try reporting this to linux-kernel, but the reports are usually ignored and the deep seated problems the fuzzer is tickling are a bit out of my league to track down.
Posted Aug 9, 2016 15:34 UTC (Tue)
by deater (subscriber, #11746)
[Link]
The bug mentioned previously that quickly locked the machine turns out to be a known bug (found with trinity a few weeks ago) with a working patch that hasn't hit upstream yet. Once I patched for the bug I let perf_fuzzer run again.
It lasted 6 hours before completely crashing. Along the way it found:
2 known WARNings
and then it finally crashed hard.
Posted Aug 15, 2016 13:41 UTC (Mon)
by MarcB (subscriber, #101804)
[Link]
However, there is a significant, other reality that has nasty places like regular end user devices (usually called "smartp hones") or even shared hosting environments.
The latter ones face malicious code and malicious, local users on a daily basis. The operating system running them needs to cope with this or it is simply unfit for this purpose.
Now, if this were about removing perf from the kernel, I could understand the opposition. But it is simply about giving the option to disable it. Arguing that "bugs will be fixed" is irrelevant: The risk is still increased. Arguing that administrators are clueless is hubris: Experienced administrators are lazy. If the default configuration works for them, they stick with it. If they change it - or even start patching the kernel - there must be a very good reason and kernel developer would be well-advised to try to understand it.
Disallowing perf_event_open()
Disallowing perf_event_open()
Disallowing perf_event_open()
or maybe because perf is not disabled by default on Debian.
Disallowing perf_event_open()
Disallowing perf_event_open()
Disallowing perf_event_open()
# update-initramfs -k all -u
I run stable, and I find that the default restrictions of perf are broken for my uses, so I have this nice line in /etc/rc.local:
Disallowing perf_event_open()
echo 0 >/proc/sys/kernel/perf_event_paranoid
Disallowing perf_event_open()
And it turns out that 4.8-rc1 falls over even more quickly than normal with the perf_fuzzer.Disallowing perf_event_open()
[163405.758086] BUG: unable to handle kernel
before completely locking up.Disallowing perf_event_open()
2 new WARNings
3 gpf/slab poisoning warnings
Disallowing perf_event_open()
Nowadays, mainline Linux *is* unfit for such environments - and the unavoidable perf is a major reason for this.