LWN: Comments on "The impact of page-table isolation on I/O performance"

The impact of page-table isolation on I/O performance

ming.lei — Wed, 25 Apr 2018 21:37:57 +0000

> Another hidden cost is that injecting interrupts is more expensive and adds latency.

I only run fio test on null_blk, and there isn't interrupt involved.

It can be observed that clock_gettime() numbers is ~3X of IO syscall when running
the test in VM, and there are still clock_gettime() syscall observed when test is run
on real machine, but number is much less compared with IO syscall number.

AIO, when used properly, should see less of a performance impact

ming.lei — Wed, 25 Apr 2018 21:33:51 +0000

>If you are only issuing one I/O per io_submit call, then that is true. However, that's not the way > AIO is typically used.

That is what fio is doing in the setup I posted, and this way may get better latency I guess.

Actually this AIO test can cover a worst case wrt. the IO perf effect from kpti, and the
effect looks not big enough.

The impact of page-table isolation on I/O performance

ming.lei — Wed, 25 Apr 2018 21:23:12 +0000

========nvme_perf===========
#!/bin/sh
fio --bs=4k --size=300G --rw=randread --norandommap --direct=1 --ioengine=libaio --iodepth=256 --numjobs=3 --runtime=$RUNTIME --group_reporting=1 --name=nvme --filename=$DEV

=============null_perf===========
#!/bin/sh
rmmod null_blk > /dev/null 2>&1
modprobe null_blk queue_mode=2 nr_devices=4 shared_tags=1 submit_queues=1 hw_queue_depth=64

fio --bs=4k --size=128G --rw=randread --norandommap --direct=1 --ioengine=libaio --iodepth=64 --numjobs=4 --runtime=40 --group_reporting=1 --name=nullb0 --filename=/dev/nullb0 --name=nullb1 --filename=/dev/nullb1 --name=nullb2 --filename=/dev/nullb2 --name=nullb3 --filename=/dev/nullb3

AIO, when used properly, should see less of a performance impact

phro — Wed, 25 Apr 2018 18:28:10 +0000

"Asynchronous I/O operations, as used in the benchmark, require two system calls — one each to io_submit() and io_getevents(). As a result, asynchronous I/O feels the PTI penalty more severely."

If you are only issuing one I/O per io_submit call, then that is true. However, that's not the way AIO is typically used.

The impact of page-table isolation on I/O performance

anton — Wed, 25 Apr 2018 12:17:48 +0000

For a benchmark that uses asynchronous I/O and is not CPU-limited, there may not be a slowdown from slower system calls at all, as pointed out by matthias. The question, as always, is: how representative is the benchmark of real applications. fio seems to allow different configurations in order to represent different applications, but this transforms the question to a question about the configuration used in the present work.

The impact of page-table isolation on I/O performance

pbonzini — Wed, 25 Apr 2018 10:36:43 +0000

Are slides available to understand the benchmark setup? Some performance loss is expected from KVM since there are effectively three I/O stacks (guest, host QEMU, host kernel). The most recent release of QEMU has a userspace NVMe driver that uses VFIO and can get some more iops, and of course you can assign an NVMe device directly to the VM.

Another hidden cost is that injecting interrupts is more expensive and adds latency. Fam Zheng will present on storage performance at LinuxCon China.

The impact of page-table isolation on I/O performance

matthias — Wed, 25 Apr 2018 05:59:42 +0000

> Is there more info on what the actual tests were? The bit about NVMe seems fishy...

My first thought also was, NVMe should be faster, do more IOs, therefore do more syscalls, so there should be a bigger impact. The following is speculative, as I do not know the test setup, but my explanation is this: NVMe has less CPU overhead associated with each IO. Without NVMe, the bottleneck is the CPU. Adding more load (by slowing down syscalls) necessarily impacts the IO speed. Using NVMe, the bottleneck is the drive (even if its faster) and adding more load to the CPU will only increase the utilization of the CPU, but not decrease IO speed, which is still limited by the drive.

Probably this would be different, if the CPU would be used by other threads. Then IO speed once more would be limited by available CPU time.

The impact of page-table isolation on I/O performance

aliguori — Tue, 24 Apr 2018 19:45:24 +0000

Xen doesn't have a vDSO for gettimeofday() so if this was Xen based testing, it will definitely be worse if pvclock is used and you are actually making system calls.

KVM's vDSO is a lot better these days but still measurably worse than TSC.

Is there more info on what the actual tests were? The bit about NVMe seems fishy...

The impact of page-table isolation on I/O performance

luto — Tue, 24 Apr 2018 18:37:38 +0000

Historically, KVM’s vDSO clock handling was a catastrophe^Wmess. I don’t remember how much better it is now.

With some loss of correctness, a guest can force the ‘tsc’ clocksource to get performance back.