Yet another way to configure transparent huge pages

Transparent huge pages (THPs) are, theoretically, supposed to allow processes to benefit from larger page sizes without changes to their code. This does work, but the performance impacts from THPs are not always a benefit, so system administrators with specific knowledge of their workloads may want the ability to fine-tune THPs to the application. On May 15, Usama Arif shared a patch set that would add a prctl() option for setting THP defaults for a process; that patch set has sparked discussion about whether such a setting is a good fit for prctl(), and what alternative designs may work instead.

The patch set added three new prctl() flags. Two of them would globally enable or disable THPs for a process, while the third would restore the system default. All three of these flags would be persisted across calls to fork() and exec(). Being able to set separate policies for each process — and to have those policies configured by the parent process, such as a system manager — would help with systems where multiple different types of workloads are present on the same machine, Arif explained.

THPs can group many smaller pages of memory together; this eliminates the overhead of tracking as many pages, and removes indirections from the process of resolving a virtual memory address to a location in physical memory. This comes at the cost of greater potential for internal fragmentation, and therefore higher memory use. Whether that tradeoff is appropriate for a program depends on the amount of available memory, its memory-access patterns, and what other jobs are competing for memory on the system.

Lorenzo Stoakes was concerned that Arif's change would effectively make the current mechanism for controlling system-wide THP policy meaningless. On current kernels, there is a sysfs file that can be used to choose whether THPs are used for all processes, for no processes, or only for processes that request it via a call to madvise(). Arif attempted to reassure him that the existing mechanism would remain in place; the prctl() proposal would effectively take the place of madvise(). Stoakes wasn't reassured:

prctl() feels like it's literally never, ever the right choice.

It feels like we shove all the dark stuff we want to put under the rug there.

Reading the man page is genuinely frightening. [T]here's stuff about VMAs _I wasn't aware of_.

After an extensive back-and-forth about the merits of Arif's proposal with David Hildenbrand, Stoakes eventually proposed creating an entirely new system call that subsumes both the current madvise() interface and Arif's need for a configurable process-wide default. His proposed interface would be a "struct-configured function" that takes a new structure describing the version of the API in use, and one of several possible operations. An enumeration would be used to select between applying advice to a single range, multiple ranges, or an entire address space. In the latter case, it could be set as the default. An optional pidfd could apply the operation to a separate process, and a set of boolean flags would configure how the call reacts to errors or gaps in the requested range.

Arif thought Stoakes's proposal was unnecessary, and would "introduce a lot of code to solve something that can be done in a very very simple way". In Arif's view, the three use cases for modifying single ranges, multiple ranges, and whole-process changes are already served by madvise, process_madvise(), and prctl(), respectively. Stoakes didn't agree, pointing out several problems with the current interfaces and calling his own proposal a "fixed madvise()". One complaint was that the current madvise() interface stops on the first error, with no way for the user to discover how far it progressed or what the details of the error were beyond the error code. He also said that the process_madvise() interface was limited to eight separate ranges. SeongJae Park pointed out that this does not seem to be true, but Stoakes did not change his mind, in light of his other concerns.

Despite their disagreement, Stoakes provided substantial review comments on Arif's patch set, which were happily accepted. Arif published a new version of the patch set on May 19. That same day, Stoakes came back with an alternative patch set extending the existing madvise() interface to support setting process defaults.

Stoakes's patch set introduces four new madvise() flags. PMADV_SKIP_ERRORS will allow a call to madvise() to continue applying advise to subsequent ranges even after encountering an error. PMADV_NO_ERROR_ON_UNMAPPED is a less strict version; with this flag, madvise() stops on errors, but does not consider encountering an unmapped range in the middle of an area as an error. PMADV_ENTIRE_ADDRESS_SPACE applies the supplied advice to a process's entire address space, and should ordinarily be called with PMADV_SKIP_ERRORS, given that not every mapping in the address space will necessarily support a given piece of advice. Finally, PMADV_SET_FORK_EXEC_DEFAULT specifies that a given set of advice should become the default for the process, persisting across fork() and exec().

Hildenbrand found Stoakes's approach interesting, but thought that the patch should introduce the minimum number of needed flags. Specifically, he questioned whether PMADV_NO_ERROR_ON_UNMAPPED was really needed as a separate flag from PMADV_ENTIRE_ADDRESS_SPACE. Stoakes agreed to make the latter flag imply the former flag, but thought the former flag could still be useful in some circumstances.

Shakeel Butt didn't like the interface for a different reason; he pointed out that the eventual goal of the THP effort is to reach a state where they can be enabled by default for all workloads. In a world where that goal is reached, what is the use for these madvise() flags? He would rather see a temporary change to prctl(), lasting just long enough to finish the THP work.

That vision of the future wasn't one that Stoakes agreed with, though. He pointed out that MADV_HUGEPAGE and MADV_NOHUGEPAGE already existed and would need to be maintained in the madvise() interface for compatibility reasons. Butt clarified that he wasn't disputing that, he just thought that this option would likely go away in the future, and so it shouldn't be embedded in the madvise() interface.

Stoakes brought up an off-list suggestion from Liam Howlett: creating a new "beautifully named" mmadvise() system call for process-wide options (of which there are already a few), to remove them from the madvise() interface. Hildenbrand thought the idea was worth experimenting with. Johannes Weiner agreed, but was worried about introducing an over-broad solution if there was actually only one thing the new interface would be used for.

Arif wanted to reach some consensus on a design before people showed off more code, and thought that introducing another option into the conversation would just make things more complicated. Eventually, Arif and Stoakes both agreed to produce new versions of their patch sets for comparison.

While those patch sets have not yet been forthcoming, the discussion on the mailing list makes it clear that there is a use case for setting process-specific THP policies. The exact form of the API to enable this is still up in the air, but it's likely that the capability will soon be added to the kernel one way or the other. The nature of temporary solutions suggests that it might remain with us for some time.