Investigating a performance bottleneck

In a short plenary session near the end of day one of the 2015 LSFMM Summit, Chuck Lever and Peter Zijlstra led a discussion on performance bottlenecks. The original idea for the session was to look at various performance problems, one of which came from Lever and others that would be offered up by those in attendance. As it turned out, though, only Lever's problem was discussed, perhaps due to low energy after a long day.

Lever described a problem he is seeing in NFS on low-latency transports, which have latencies an order of magnitude less than Ethernet. For his test, the latency added by the RPC infrastructure is on the order of 20µs and the round-trip network time is around 25µs. On idle clients, the performance is much what he expects, but if he loads the client with, say, a kernel build, these RPC tests start taking 300µs.

Lever has narrowed the problem down to wake_up_bit(). That function is taking "too bloody long", Zijlstra said. There is some contention on waking, he continued, but it is not clear what that could be.

Dave Chinner suggested using the latency tracer in ftrace to help further narrow it down.

Chris Mason noted that he has started benchmarking newer kernels at Facebook and had not run into anything surprising yet.

Lever said that it is not just a spinlock that is being contended, as the resources are being held far longer than that. Zijlstra said that the wakeup itself should not be that expensive. Perhaps it is the runqueue locks that are being contended in that situation.

Andy Lutomirski wondered if inter-processor interrupts (IPIs) take longer to send in this case. There is a different path in the code when the system is under load, he said. Mel Gorman suggested testing with a maximum cstate value set to zero to ensure that power management wasn't affecting things. At the end, Zijlstra suggested gathering more data and said that he and others would have a look then.

[I would like to thank the Linux Foundation for travel support to Boston for the summit.]