The transparent huge page shrinker [LWN.net]

The transparent huge page shrinker

Posted Sep 9, 2022 7:30 UTC (Fri) by linoliumz (guest, #134676)
Parent article: The transparent huge page shrinker

In the README file of one of my HPC projects I have a section that describes how to enable transparent huge pages on Linux in order to get the best performance. Based on my experience, enabling transparent huge pages can provide a huge speed up on workloads that use a lot of RAM (> 50 GiB) e.g. one of my application's algorithms runs twice as fast if transparent huge pages are enabled. Any patch that brings us closer to enabling transparent huge pages by default is highly appreciated by me.

to post comments

The transparent huge page shrinker

Posted Sep 10, 2022 8:00 UTC (Sat) by WolfWings (subscriber, #56790) [Link] (4 responses)

Serious question, is there a reason your code doesn't check if it's running on Linux at compile time and if so just utilize madvise() to explicitly request THP?

Because all distro's I've encountered for a few years now leave it set to madvise not disabling it entirely at least. And at this point the madvise() with MADV_HUGEPAGE is by orders of magnitude the most reliable and least troublesome way to utilize huge pages on Linux since everything else requires mucking about with lots of individual knobs by comparison.

The transparent huge page shrinker

Posted Sep 10, 2022 9:35 UTC (Sat) by linoliumz (guest, #134676) [Link] (3 responses)

My application supports many different operating systems, not only Linux. I try to keep my code base portable, I am willing to add workarounds for specific operating systems to my application if it will provide a significant benefit to my users and the workaround is relatively small. However, in this case I don't think is worth the effort. In my opinion the current Linux support of transparent huge pages is a mess (to be fair, on Windows it is even worse) and I prefer to simply wait until the situation improves and hopefully transparent huge pages will eventually be enabled by default.

The transparent huge page shrinker

Posted Sep 10, 2022 18:23 UTC (Sat) by WolfWings (subscriber, #56790) [Link] (2 responses)

I guess that's my point: It is enabled by default and has been for almost a decade now, just behind madvise where the interface to use it becomes trivial.

#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h> // Also used for various posix functions, cross platform
const int alignment = 1 << 21;
const int size = 16 << 21;
int main( void ) {
    void *x = aligned_alloc( alignment, size );

#ifdef MADV_HUGEPAGE
    if ( x != NULL ) {
        madvise( x, size, MADV_HUGEPAGE  );
    }
#endif

    printf( "Go run: grep -i hugepage /proc/meminfo\nPausing for 60 seconds.\n" );
    sleep( 60 );

    return 0;
}

Beware fragmentation

Posted Sep 11, 2022 0:09 UTC (Sun) by jreiser (subscriber, #11027) [Link] (1 responses)

Often I get fewer huge pages than requested, even with 32GB of RAM on a (now) lightly-loaded system that has been up for a week or so:

    system("echo \"requested 32MB of anon huge pages:\";\
            grep -i hugepage /proc/meminfo");
    //printf( "Go run: grep -i hugepage /proc/meminfo\nPausing for 60 seconds.\n" );
    //sleep( 60 );

requested 32MB of anon huge pages:
AnonHugePages:      6144 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Also remember: #include <stdio.h> if printf remains in the source.

Beware fragmentation

Posted Sep 12, 2022 8:25 UTC (Mon) by WolfWings (subscriber, #56790) [Link]

Sorry, minor oversight on my part, since you do need to fault the RAM in before it exists. The program was literally typed from memory as an example, sorry if I missed an include. :P

#ifdef MADV_HUGEPAGE
    if ( x != NULL ) {
        madvise( x, size, MADV_HUGEPAGE )
        for ( int i = 0; i < size; i += alignment ) {
            ((char *)x)[i] = 0;
        }
    }
#endif

If you fault the pages in before the madvise then you have to wait for the THP scanner to circle back around and find things, but if you madvise before you fault the pages in then each fault is generated as a THP directly so you also need a lot less page faults to actually commit all the RAM you allocated, since Linux is (too) aggressive about overcommit until pages are actually accessed.