LWN: Comments on "Kernel page-table isolation merged"

Kernel page-table isolation merged

Cyberax — Wed, 10 Jan 2018 01:50:53 +0000

This should be a quote of the week.

Kernel page-table isolation merged

immibis — Wed, 10 Jan 2018 00:31:24 +0000

> Speculation is fun :-)

Foreshadowing!

Kernel page-table isolation merged

microchp — Tue, 09 Jan 2018 16:47:04 +0000

Sorry, I was multitasking in the Spectre/Meltdown discussions and meant RHEL7 on the 3.10.x kernel.

Kernel page-table isolation merged

mjg59 — Thu, 04 Jan 2018 02:11:05 +0000

The embargo was lifted 2 hours before your post :)

Kernel page-table isolation merged

rahvin — Thu, 04 Jan 2018 00:33:59 +0000

I don't know if it's been added, but the patch was posted by Ken with AMD and he says pretty explicitly that AMD isn't vulnerable. Maybe it was premature and they actually are I don't know but I wish they'd lift the embargo and tell us. Particularly given the reports of Intel executives making large share sales.

https://lkml.org/lkml/2017/12/27/2

Kernel page-table isolation merged

MarcB — Wed, 03 Jan 2018 22:02:55 +0000

I think, I get that now ;-)

Kernel page-table isolation merged

rahulsundaram — Wed, 03 Jan 2018 20:36:36 +0000

RHEL 3, 4 and 5 is EOL. Extended support is available for 5 IIRC

Kernel page-table isolation merged

cesarb — Wed, 03 Jan 2018 20:10:10 +0000

> When this is back-ported to older kernels, such as in RHEL 3.x

Aren't RHEL 3.x and 4.x completely out of support already?

Kernel page-table isolation merged

Lionel_Debroux — Wed, 03 Jan 2018 19:14:52 +0000

PaX and therefore grsecurity have been taking advantage of PCID for years, to get either a stronger (default) or a faster version of the MEMORY_UDEREF protection. But indeed, the mainline Linux kernel only started using PCID very recently.

Kernel page-table isolation merged

deater — Wed, 03 Jan 2018 18:40:18 +0000

> Are there more such silicon features ignored by Linux?

Definitely. See the sad story that was AMD's Light-Weight Profiling instructions. Though it looks like they gave up on Linux support ever being added and have dropped support for the feature.

Kernel page-table isolation merged

Wol — Wed, 03 Jan 2018 18:37:06 +0000

Normal development timetable ...

One layer adds a nifty feature. Next layer takes maybe a year to realise the feature exists and work out how to take advantage of it. Allow a year for debugging. Next layer up realises ... rinse lather and repeat.

Innovations typically take TWENTY TO THIRTY YEARS to filter through. How long did it take before the internet (in the form of the WWW) became ubiquitous? Around in the 70s, pushed and *financed* by Gore in the 80s, the web invented in the 90s (as a minor improvement on Gopher!), and ubiquitous by the 2000s. How long is that?

Cheers,
Wol

Kernel page-table isolation merged

excors — Wed, 03 Jan 2018 18:20:29 +0000

There can be - for example any code that tries to exploit this vulnerability will 'work' on Intel hardware and not AMD. But if you exclude code that has been specifically designed to exploit it, it seems unlikely.

CPUs are usually allowed to fetch whatever memory they fancy whenever they fancy (which gives them freedom to continually improve caches, prefetchers, speculative execution, etc), which is safe because it has no effect on application behaviour except in terms of instruction timing (and performance counters etc). Intel and AMD fetch memory quite differently, and different generations and models of CPUs by a single vendor fetch memory quite differently, so nobody writes applications that depend precisely on instruction timing. Except for people intentionally using timing attacks to extract information that exists inside the CPU pipeline but that wasn't meant to be observable to applications, which is presumably the problem here. (And except for buggy code with race conditions that get triggered by these timing changes, but that's so broken anyway that it doesn't really matter.)

Kernel page-table isolation merged

microchp — Wed, 03 Jan 2018 18:12:12 +0000

Thankyou, I missed that.

Kernel page-table isolation merged

zdzichu — Wed, 03 Jan 2018 18:04:00 +0000

On a side note, PCID seem to be feature of Intel CPUs for 8 (eight!) years now, yet it seem it only get used with kernel 4.14 and later. It is strange to see so many years pass before it got used. Are there more such silicon features ignored by Linux?

Kernel page-table isolation merged

MarcB — Wed, 03 Jan 2018 18:03:34 +0000

This is unlikely to be a simple, direct read of forbidden memory. Most likely, it is a timing side-channel.

So, no existing code will be affected (unless there is an existing exploit, of course).

Kernel page-table isolation merged

Paf — Wed, 03 Jan 2018 17:51:36 +0000

Do you know your exact CPU? Do you have PCID support? (I guess that means Westmere and newer)

Kernel page-table isolation merged

jezuch — Wed, 03 Jan 2018 17:39:13 +0000

...I also wonder whether there's any (can there be any?) code out there that relies on Intel's behaviour and does not work on AMD hardware :) Or maybe this kind of problem only affects software API's...

Kernel page-table isolation merged

jezuch — Wed, 03 Jan 2018 17:33:53 +0000

Interesting. If that's not considered a violation of x86 architecture spec then Intel really screwed up.

Kernel page-table isolation merged

vstinner — Wed, 03 Jan 2018 17:02:53 +0000

I like the last sentence of the blog article: "Further we see that speculative execution does not consistently abide by isolation mechanism, thus it’s a haunting question what we can actually do with speculative execution." Anders Fogh, July 2017

Kernel page-table isolation merged

vstinner — Wed, 03 Jan 2018 17:00:35 +0000

"There will be a nopti command-line option to disable this mechanism at boot time."
https://lwn.net/Articles/741878/

Kernel page-table isolation merged

microchp — Wed, 03 Jan 2018 16:56:37 +0000

When this is back-ported to older kernels, such as in RHEL 3.x, will there be a toggle to preserve the old behavior?

I am asking for the folks that will need time to mitigate the performance hit and may have servers with a lower risk profile.

Kernel page-table isolation merged

sorokin — Wed, 03 Jan 2018 12:12:38 +0000

The Register published an article about the issue: Intel processor design flaw forces Linux, Windows redesign. The article refers to a blog post worth reading: Negative Result: Reading Kernel Memory From User Mode.

Kernel page-table isolation merged

cesarb — Wed, 03 Jan 2018 10:54:30 +0000

> they've also added a flag to exempt AMD hardware so I'd presume AMD hardware is not vulnerable.

Has the commit adding that test been merged already? So far, I've only seen it on the mailing list, but not on the kernel repository, so as far as I can see, AMD hardware is not yet exempted.

Kernel page-table isolation merged

deater — Wed, 03 Jan 2018 04:58:48 +0000

The recent PAPI 5.6 performance library release apparently happened just in time.

It added code using rdpmc while doing self-monitoring reads of the perf_event performance counters.

On haswell a counter read:
Using rdpmc (avoids kernel): 142 cycles
4.14 kernel using read() syscall: 913 cycles.
4.15-rc6 kernel using read() syscall: 1360 cycles.

That's a huge increase when you're trying to do low-latency measurements.

Kernel page-table isolation merged

rahvin — Wed, 03 Jan 2018 03:03:25 +0000

The benchmarks being reported could be catastrophic (5% to 50% performance degradation depending on workload), they've also added a flag to exempt AMD hardware so I'd presume AMD hardware is not vulnerable.

By all reports this is worse than the Intel lights-out firmware bug and allows user space code to read protected kernel memory, conceivably allowing one VM to read the memory of another VM per one of the scenario's I've seen. This has the potential to be heart-bleed plus a remote exploitable memory read that can be executed by user space code including javascript running in a browser. And it's hard coded in Intel silicon requiring the need to use the OS to separate the kernel and user space cache system resulting in major performance hits. Talk about ugly and just like the firmware it's in every processor Intel has built for more than a decade.

This is beyond brutal and I expect it's going to exacerbate the AMD processor shortage, good news for AMD at least. Bad news for anyone running an internet connected server.

Kernel page-table isolation merged

andresfreund — Tue, 02 Jan 2018 22:32:54 +0000

> > The change isn’t in Linus’s tree yet, so I guess we’ll have to wait and see, but if the performance impact is significant on Intel processors and Ryzen and EPYC aren’t affected it could spell a serious problem for Intel.

> I don't think it'll be that bad. The patchset has ASID / PCID support on capable hardware reducing the syscall overhead on capable hardware (apparently sandy bridge onwards).

Some numbers for postgres workloads that more likely to suffer (i.e. very short statements with plenty syscalls, rather than analytics workloads):
https://www.postgresql.org/message-id/20180102222354.qikj...

readonly pgbench (tpch-like), 16 clients, i7-6820HQ CPU (skylake):

pti=off:
tps = 236629.778328

pti=on:
tps = 220791.228297 (~0.93x)

pti=on, nopcid:
tps = 198959.801459 (~0.84x)

To get closer to the worst case, I've also measured:

pgbench SELECT 1, 16 clients, i7-6820HQ CPU (skylake):

pti=off:
tps = 420490.162391

pti=on:
tps = 350746.065039 (~0.83x)

pti=on, nopcid:
tps = 324269.903152 (~0.77x)

So yea, definitely not fun :(

Kernel page-table isolation merged

riel — Tue, 02 Jan 2018 22:12:09 +0000

Speculation is fun indeed.

Kernel page-table isolation merged

Wol — Tue, 02 Jan 2018 13:46:05 +0000

> there's no conceivable upgrade path to skip from 6 to 8 over 15 years worth of development.

As has been pointed out here before, I believe ... how many systems are installed and never upgraded? If you have no plans to upgrade the system ever, why start now?

Although, in that case, it would probably make sense to start migrating whatever apps you have that matter, across to a new system.

Cheers,
Wol

Kernel page-table isolation merged

sync — Tue, 02 Jan 2018 13:27:26 +0000

Fedora rawhide released a PTI enabled kernel (4.15.0-0.rc6.git0.1.fc28.x86_64). I did some test with it. Syscalls are much slower with PTI enabled.

bench_03_getpid_vdso 2.7x slower
bench_11_read_vdso 2.0x slower
bench_21_write_vdso 2.3x slower

I had used this benchmark https://github.com/arkanis/syscall-benchmark on an Intel i5-4200U. Of course this is only a microbenchmark. The real world slowdown is much less.

Kernel page-table isolation merged

joib — Tue, 02 Jan 2018 08:51:08 +0000

https://www.realworldtech.com/westmere/ claims PCID was introduced already in the Westmere generation.

Kernel page-table isolation merged

Lionel_Debroux — Tue, 02 Jan 2018 08:06:01 +0000

Ivy Bridge and Skylake, which were spender's ( https://twitter.com/grsecurity ) benchmark targets, saw 34% and 29% slowdowns on a simple `du -s` benchmark. These are newer than Sandy Bridge and feature PCID, so barring bugs, the patchset should have leveraged that feature, shouldn't it ?

Kernel page-table isolation merged

andresfreund — Tue, 02 Jan 2018 01:32:17 +0000

> The change isn’t in Linus’s tree yet, so I guess we’ll have to wait and see, but if the performance impact is significant on Intel processors and Ryzen and EPYC aren’t affected it could spell a serious problem for Intel.

I don't think it'll be that bad. The patchset has ASID / PCID support on capable hardware reducing the syscall overhead on capable hardware (apparently sandy bridge onwards).

Kernel page-table isolation merged

Felix_the_Mac — Tue, 02 Jan 2018 00:00:19 +0000

I guess now would be a good time to buy AMD stock

Kernel page-table isolation merged

GhePeU — Mon, 01 Jan 2018 23:28:41 +0000

AMD CPUs could be unaffected by whatever vulnerability KPTI is meant to protect from: https://lkml.org/lkml/2017/12/27/2

The change isn’t in Linus’s tree yet, so I guess we’ll have to wait and see, but if the performance impact is significant on Intel processors and Ryzen and EPYC aren’t affected it could spell a serious problem for Intel.

Kernel page-table isolation merged

arekm — Mon, 01 Jan 2018 13:02:09 +0000

Nice summary
http://pythonsweetness.tumblr.com/post/169166980422/the-m...

Kernel page-table isolation merged

amacater — Sun, 31 Dec 2017 19:41:01 +0000

Only if you're paying for extended lifecycle support - 5 was extended to 30/11/2020 as was 6 but only for RHEL. CentOS stopped supporting 5 this year, CentOS 6 will be 30/11/2020

Kernel page-table isolation merged

MarcB — Sun, 31 Dec 2017 17:44:35 +0000

It seems, Microsoft is making the same change in Windows: https://twitter.com/aionescu/status/930412525111296000

Perhaps they were inspired by the publication of KAISER (time frame fits, if they were really fast), perhaps they got the idea independently or perhaps there really is an upcoming publication. Speculation is fun :-)

Kernel page-table isolation merged

pbonzini — Sun, 31 Dec 2017 16:04:49 +0000

There are plenty of users still on RHEL 6. RHEL 6.9 was released last March, and I would be surprised if it's the last update.

The last RHEL 5 release was 5.11, and in fact 2020 will be when the last security updates come for RHEL 5, not 6.

Kernel page-table isolation merged

amacater — Sun, 31 Dec 2017 14:17:22 +0000

RHEL == Red Hat Enterprise Linux throughout. [I know Red Hat folk don't like the abbreviation but I'm lazy :) ]

If running RHEL/CentOS 6 - move your machines and code to version 7 quickly? There's no very good reason not to run 7 given that 6 is coming to the end of its supported life [November 2020 is less than two years away now] and 6.8 was likely the last major release.

Oh, and if you're not running RHEL 6.8 move to it now for your RHEL machines because there is a relatively supported upgrade script to take 6 machines forward to 7 [This no longer works well for CentOS machines because of version skew between 6 and 7 and the CentOS folk are looking for a new maintainer - but RHEL will presumably pay someone to hand fix it]

And 2.6.32 is well beyond support, realistically, for any but the very worst case: RHEL 7 has a 3.* kernel and I'd expect RHEL 8 betas any day now with a 4.* kernel (probably 4.4). I can't see anyone willingly supporting three supported versions of RHEL concurrently so 6 users are being encouraged to move to 7 now: there's no conceivable upgrade path to skip from 6 to 8 over 15 years worth of development.

Kernel page-table isolation merged

lkundrak — Sun, 31 Dec 2017 13:48:43 +0000

Well, no need to guess -- X86_BUG_CPU_INSECURE says that pretty clearly.