The impact of page-table isolation on I/O performance [LWN.net]

By Jonathan Corbet
April 24, 2018

Ever since kernel page-table isolation (PTI) was introduced as a mitigation for the Meltdown CPU vulnerability, users have worried about how it affects the performance of their systems. Most of that concern has been directed toward its impact on computing performance, but I/O performance also matters. At the 2018 Linux Storage, Filesystem, and Memory-Management Summit, Ming Lei presented some preliminary work he has done to try to quantify how severely PTI affects block I/O operations.

This work was done by running the fio benchmark on current hardware. The initial tests, running in a virtual machine, showed a significant impact: a system that could execute just over 1 million I/O operations per second (IOPS) without PTI was reduced to 726,000 IOPS with PTI turned on. The situation changes significantly when the test is run on bare metal on the same machine; in that case, a system that could achieve 1,706,000 IOPS dropped to 1,568,000 IOPS when PTI is turned on. At a little under 10%, that is a smaller impact, but still a significant one.

It's not clear why performance regresses so severely when the test is run under virtualization. There was some theorizing that clock_gettime(), which is called frequently by fio, is not implemented properly on the guest system, but no real answers.

Further tests were done using an NVMe-attached drive. In this case, the IOPS rates were about the same regardless of whether PTI was being used, but the system's CPU utilization was significantly higher in the PTI case.

Lei concluded from his tests that enabling PTI adds about 0.2µs to the execution time of every system call. Normal synchronous I/O operations can be performed with a single system call, so they slow down slightly as a result. Asynchronous I/O operations, as used in the benchmark, require two system calls — one each to io_submit() and io_getevents(). As a result, asynchronous I/O feels the PTI penalty more severely. Interrupts add a similar penalty to each operation as well.

Dave Hansen (who did much of the work to bring PTI to Linux) noted that there was nothing new in these results. There has always been a cost to both interrupts and system calls; PTI just makes those costs worse. He did note that it was nice to see that the IOPS don't drop when there is adequate CPU time available, though.

Block maintainer Jens Axboe said that fio performs three clock_gettime() calls for every I/O operation by default. So, to a great extent, Lei's tests were measuring the impact of PTI on system-call execution time. Bart Van Assche suggested using the options that reduce the number of clock_gettime() calls, just as the session wound down.

Index entries for this article
Kernel	Security/Meltdown and Spectre
Conference	Storage, Filesystem, and Memory-Management Summit/2018

to post comments

The impact of page-table isolation on I/O performance

Posted Apr 24, 2018 18:37 UTC (Tue) by luto (subscriber, #39314) [Link] (2 responses)

Historically, KVM’s vDSO clock handling was a catastrophe^Wmess. I don’t remember how much better it is now.

With some loss of correctness, a guest can force the ‘tsc’ clocksource to get performance back.

The impact of page-table isolation on I/O performance

Posted Apr 24, 2018 19:45 UTC (Tue) by aliguori (guest, #30636) [Link] (1 responses)

Xen doesn't have a vDSO for gettimeofday() so if this was Xen based testing, it will definitely be worse if pvclock is used and you are actually making system calls.

KVM's vDSO is a lot better these days but still measurably worse than TSC.

Is there more info on what the actual tests were? The bit about NVMe seems fishy...

The impact of page-table isolation on I/O performance

Posted Apr 25, 2018 5:59 UTC (Wed) by matthias (subscriber, #94967) [Link]

> Is there more info on what the actual tests were? The bit about NVMe seems fishy...

My first thought also was, NVMe should be faster, do more IOs, therefore do more syscalls, so there should be a bigger impact. The following is speculative, as I do not know the test setup, but my explanation is this: NVMe has less CPU overhead associated with each IO. Without NVMe, the bottleneck is the CPU. Adding more load (by slowing down syscalls) necessarily impacts the IO speed. Using NVMe, the bottleneck is the drive (even if its faster) and adding more load to the CPU will only increase the utilization of the CPU, but not decrease IO speed, which is still limited by the drive.

Probably this would be different, if the CPU would be used by other threads. Then IO speed once more would be limited by available CPU time.

The impact of page-table isolation on I/O performance

Posted Apr 25, 2018 10:36 UTC (Wed) by pbonzini (subscriber, #60935) [Link] (2 responses)

Are slides available to understand the benchmark setup? Some performance loss is expected from KVM since there are effectively three I/O stacks (guest, host QEMU, host kernel). The most recent release of QEMU has a userspace NVMe driver that uses VFIO and can get some more iops, and of course you can assign an NVMe device directly to the VM.

Another hidden cost is that injecting interrupts is more expensive and adds latency. Fam Zheng will present on storage performance at LinuxCon China.

The impact of page-table isolation on I/O performance

Posted Apr 25, 2018 21:23 UTC (Wed) by ming.lei (subscriber, #74703) [Link]

========nvme_perf===========
#!/bin/sh
fio --bs=4k --size=300G --rw=randread --norandommap --direct=1 --ioengine=libaio --iodepth=256 --numjobs=3 --runtime=$RUNTIME --group_reporting=1 --name=nvme --filename=$DEV

=============null_perf===========
#!/bin/sh
rmmod null_blk > /dev/null 2>&1
modprobe null_blk queue_mode=2 nr_devices=4 shared_tags=1 submit_queues=1 hw_queue_depth=64

fio --bs=4k --size=128G --rw=randread --norandommap --direct=1 --ioengine=libaio --iodepth=64 --numjobs=4 --runtime=40 --group_reporting=1 --name=nullb0 --filename=/dev/nullb0 --name=nullb1 --filename=/dev/nullb1 --name=nullb2 --filename=/dev/nullb2 --name=nullb3 --filename=/dev/nullb3

The impact of page-table isolation on I/O performance

Posted Apr 25, 2018 21:37 UTC (Wed) by ming.lei (subscriber, #74703) [Link]

> Another hidden cost is that injecting interrupts is more expensive and adds latency.

I only run fio test on null_blk, and there isn't interrupt involved.

It can be observed that clock_gettime() numbers is ~3X of IO syscall when running
the test in VM, and there are still clock_gettime() syscall observed when test is run
on real machine, but number is much less compared with IO syscall number.

The impact of page-table isolation on I/O performance

Posted Apr 25, 2018 12:17 UTC (Wed) by anton (subscriber, #25547) [Link]

For a benchmark that uses asynchronous I/O and is not CPU-limited, there may not be a slowdown from slower system calls at all, as pointed out by matthias. The question, as always, is: how representative is the benchmark of real applications. fio seems to allow different configurations in order to represent different applications, but this transforms the question to a question about the configuration used in the present work.

AIO, when used properly, should see less of a performance impact

Posted Apr 25, 2018 18:28 UTC (Wed) by phro (subscriber, #29295) [Link] (1 responses)

"Asynchronous I/O operations, as used in the benchmark, require two system calls — one each to io_submit() and io_getevents(). As a result, asynchronous I/O feels the PTI penalty more severely."

If you are only issuing one I/O per io_submit call, then that is true. However, that's not the way AIO is typically used.

AIO, when used properly, should see less of a performance impact

Posted Apr 25, 2018 21:33 UTC (Wed) by ming.lei (subscriber, #74703) [Link]

>If you are only issuing one I/O per io_submit call, then that is true. However, that's not the way > AIO is typically used.

That is what fio is doing in the setup I posted, and this way may get better latency I guess.

Actually this AIO test can cover a worst case wrt. the IO perf effect from kpti, and the
effect looks not big enough.