LWN: Comments on "Disallowing perf_event_open()"

Disallowing perf_event_open()

MarcB — Mon, 15 Aug 2016 13:41:31 +0000

Somehow, I get the impression some kernel developers have disconnected from reality. Or maybe live in a different one, that consists only of developer workstations, back-end systems and maybe super-computing. Obviously, perf is always fine and useful in this reality.

However, there is a significant, other reality that has nasty places like regular end user devices (usually called "smartp hones") or even shared hosting environments.

The latter ones face malicious code and malicious, local users on a daily basis. The operating system running them needs to cope with this or it is simply unfit for this purpose.
Nowadays, mainline Linux *is* unfit for such environments - and the unavoidable perf is a major reason for this.

Now, if this were about removing perf from the kernel, I could understand the opposition. But it is simply about giving the option to disable it. Arguing that "bugs will be fixed" is irrelevant: The risk is still increased. Arguing that administrators are clueless is hubris: Experienced administrators are lazy. If the default configuration works for them, they stick with it. If they change it - or even start patching the kernel - there must be a very good reason and kernel developer would be well-advised to try to understand it.

Disallowing perf_event_open()

deater — Tue, 09 Aug 2016 15:34:08 +0000

For those keeping track, I ran the perf_fuzzer on a Haswell machine running 4.8-rc1.

The bug mentioned previously that quickly locked the machine turns out to be a known bug (found with trinity a few weeks ago) with a working patch that hasn't hit upstream yet. Once I patched for the bug I let perf_fuzzer run again.

It lasted 6 hours before completely crashing. Along the way it found:

2 known WARNings
2 new WARNings
3 gpf/slab poisoning warnings

and then it finally crashed hard.

Disallowing perf_event_open()

deater — Mon, 08 Aug 2016 15:09:36 +0000

And it turns out that 4.8-rc1 falls over even more quickly than normal with the perf_fuzzer.

The fun part is that the serial console only manages to print
[163405.758086] BUG: unable to handle kernel
before completely locking up.

I'll try reporting this to linux-kernel, but the reports are usually ignored and the deep seated problems the fuzzer is tickling are a bit out of my league to track down.

Disallowing perf_event_open()

spender — Sat, 06 Aug 2016 18:06:05 +0000

I like that we have two comments now in this thread alone demonstrating how people have no problem at all with the serious security side-effects of the coarseness of the toggle in the other direction, yet all the complaints are focused on the mere option of an additional, more secure setting. When you consider that even CONFIG_NET can be disabled, but PERF_EVENTS cannot, it's obvious that all the pushback has to do with an arch maintainer also being the maintainer of a pet project he wants to force on everyone, spending more time on adding new bells and whistles and not on (for instance) fixing known issues deater has been talking about here for years now exposed by simple fuzzing, meanwhile offering users no way to mitigate those issues.

-Brad

Disallowing perf_event_open()

anton — Sat, 06 Aug 2016 17:35:18 +0000

I run stable, and I find that the default restrictions of perf are broken for my uses, so I have this nice line in /etc/rc.local:

echo 0 >/proc/sys/kernel/perf_event_paranoid

Disallowing perf_event_open()

nybble41 — Fri, 05 Aug 2016 04:36:59 +0000

> …but I managed to write my first ever systemd init config to write a sensible value to /proc/sys/kernel/perf_event_paranoid on boot up…

Why a systemd init config? Isn't this exactly why we have /etc/sysctl.conf and /etc/sysctl.d/?

# echo "kernel.perf_event_paranoid = 0" > /etc/sysctl.d/perf_events.conf
# update-initramfs -k all -u

Disallowing perf_event_open()

creemj — Fri, 05 Aug 2016 00:57:59 +0000

I upgraded to testing two or three weeks ago and had to spend some time working out why my profiling code stopped working. It was a bit annoying to find that perf events was disabled in the kernel from scratch, but I managed to write my first ever systemd init config to write a sensible value to /proc/sys/kernel/perf_event_paranoid on boot up so that I wouldn't be pissed off every time I rebooted.

The pain level has to be quite high before I would report a bug or issue back to Debian --- I hate the reportbug interface and find 'apt-get purge crappy-buggy-package' a much easier and quicker solution in most cases.

I am wondering, however, whether there is a kernel option to change the default? That would be a neater solution.

Disallowing perf_event_open()

deater — Fri, 05 Aug 2016 00:17:21 +0000

> or maybe because perf is not disabled by default on Debian.

I always forget that most people do not run unstable.

Disallowing perf_event_open()

ballombe — Thu, 04 Aug 2016 22:51:35 +0000

> (it hasn't started problems yet because most users aren't using Debian)
or maybe because perf is not disabled by default on Debian.

Disallowing perf_event_open()

nix — Thu, 04 Aug 2016 20:01:41 +0000

Quite. That's not the only use case either. My firewall, in common with much semi-embedded hardware, has no useful PMUs and runs no software that would benefit (and even if it did, I frankly don't care if I break self-tuning JITs on a firewall: it's not doing anything else so the performance of things like that is of minimal interest). But as firewalls tend to be, it is network-exposed to hostile attackers and really should not have huge unused lumps of security-critical code on it at all, let alone exposed to unprivileged users.

I have no idea why there isn't a config knob to compile perf out entirely. There surely should be. Not everything needs it; not everything can benefit; and those things that don't are purely harmed by its distinctly non-zero-sized vulnerability surface.

Disallowing perf_event_open()

deater — Thu, 04 Aug 2016 12:52:09 +0000

Any word on how the CVEs were found? Via a fuzzer or some other way? It is a bit annoying to find internal bugs in your own architectural driver and your solution is to disable a global interface for everyone.

At the same time, the perf_fuzzer can still reliably crash any x86 machine within a few hours despite years of trying to get that fixed, so it probably is a good idea to provide a way to block the syscall.

I see both sides of the issue here, as having perf_event disabled by default is really going to cause annoyance for high-performance computing users (it hasn't started problems yet because most users aren't using Debian). If you start requiring root or sysadmin action in order for perf to work, people are going to go back to skipping perf altogether for tools like LIKWID (they also usually require root or sysadmin intervention).