Likely unlikely()s

The kernel has two macros to assist the compiler and CPU in doing branch prediction: likely() and unlikely(). As their names imply, they are meant to annotate tests in the code based on the likelihood that they will evaluate to true. But, getting it wrong such that something marked likely is actually unlikely—or vice versa—can impact performance as the CPU may prefetch the wrong instructions.

Steven Rostedt has been looking at the problem using the annotated branch profiler and found ten places "that really do not need to have an annotated branch, or the annotation is simply the opposite of what it should be". So, he created a series of patches that either switched the sense of the annotation or removed the likely()/unlikely() entirely.

As an example, page_mapping() had an unlikely() annotation that Rostedt reported as being "correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect". Those numbers come from his main workstation which runs a variety of standard programs (Firefox, XChat, etc.) as well as participating in his build farm, so it should represent a reasonably "normal" workload. Being wrong 39% of the time was pretty obviously too much and led to the removal of the annotation for that test.

The changes are various subsystems including the scheduler, memory management, and VFS. So far, there have been no complaints, though there have been several requests to completely remove annotations that had just been changed in order to allow the compiler's and CPU's branch prediction logic make the decision. That, and breaking the patches up into separate sets for each subsystem, caused Rostedt to respin them. It would seem likely() that we'll see them make their way into 2.6.38.

Note: you can avoid this step in the future by logging into your LWN account.