LWN: Comments on "Huge pages part 1 (Introduction)"

Huge pages part 1 (Introduction)

heguanjun — 2012-12-26T08:00:58+00:00

well, now there is another huge page implementation: Transparent huge pages. and with many benefits.

Huge pages part 1 (Introduction)

glennewton — 2010-07-07T04:48:06+00:00

I've collected a number of good resources for huge pages for Linux, Java, Solaris, MySql, AMD here: Java, MySql increased performance with Huge Pages.

Alan Cox on FreeBSD

nix — 2010-02-24T22:47:20+00:00

Yay, thank you!

Alan Cox on FreeBSD

biged — 2010-02-24T19:37:41+00:00

Try here: http://www.usenix.org/events/osdi02/tech/full_papers/nava...

Alan Cox on FreeBSD

nix — 2010-02-23T21:53:59+00:00

I wish I had been thinking of that paper, but I had no idea it existed.
Unfortunately all the links to it appear dead :( 10.1.1.14.2514 is not a
valid DOI as far as I can tell, and the source link throws an error page
at me...

Alan Cox on FreeBSD

viro — 2010-02-23T06:10:23+00:00

ac != alc... IOW, it's not who you probably think it is.

Alan Cox on FreeBSD

man_ls — 2010-02-23T05:22:57+00:00

You are probably thinking of this excellent LWN article from 2007 and the excellent Navarro paper (with a contribution by Alan Cox). The fact that they decided to bring transparent huge pages support to FreeBSD and not to Linux is funny, considering.

Huge pages part 1 (Introduction)

nix — 2010-02-20T16:33:52+00:00

This stuff could presumably autotune, kicking in only on CPUs for which it
is a net win.

Huge pages part 1 (Introduction)

nix — 2010-02-20T16:31:19+00:00

Oh, hell, I forgot about the split TLB. That makes the whole resource
allocation problem drastically harder :/ Still, does the kernel need more
than one or two entries? If you're not using hugepages currently, it seems
to me that some of those hugepage TLB entries are actually wasted...

Huge pages part 1 (Introduction)

farnz — 2010-02-19T14:19:03+00:00

It does. The work being done is for huge pages for userspace, which is a whole different ballgame, and could result in the kernel's hugepage mapping being pushed out of the TLB.

If/when someone does the work, it'll need benchmarking not just on the latest and greatest, but also on real-world older systems with more restrictive TLBs, to see if it's a net loss.

Huge pages part 1 (Introduction)

paulj — 2010-02-19T14:15:43+00:00

Doesn't Linux already use huge pages for the kernel? Or am I misremembering
something?

Huge pages part 1 (Introduction)

farnz — 2010-02-19T12:03:34+00:00

What you really want (but it's difficult to do sanely on x86) is transparent huge pages. Where possible, the kernel gives you continguous physical pages for continguous virtual pages, and it transparently converts suitable sets of continguous virtual pages to the next size of mapping up when it can do so, and splits large mappings into the next size down when they're not in use, or when there's memory pressure.

The pain on x86 is twofold: first, instead of getting to aggregate (e.g.) 16 4K pages into a 16K page, then 16 16K pages into a 256K page, you get to do things like aggregate 1024 4K pages into a 4M page, and 256 4M pages into a 1GB page. Second, typical x86 TLBs are split by page type; so it's not uncommon to have something like the Core 2 Duo, where you have 128 entries for 4K pages, and just 4 entries for 4M pages (Instruction TLB).

Given that split, most workloads gain more from having the kernel always in the TLB, than from evicting the kernel in favour of your own code (which would have been in the 4K page size TLB otherwise).

Huge pages part 1 (Introduction)

nix — 2010-02-19T10:57:46+00:00

I find myself looking at huge pages and thinking that huge pages are a feature that will be useful only for special-purpose single-use machines (basically just the two Mel mentions: simulation and databases, and perhaps here and there virtualization where the machine has *lots* of memory) until the damn things are swappable. Obviously we can't swap them as a unit (please wait while we write a gigabyte out), so we need to break the things up and swap bits of them, then perhaps reaggregate them into a Gb page once they're swapped back in. Yes, it's likely to be a complete sod to implement, but it would also give that nice TLB-hit speedup feeling without having to worry if you're about to throw the rest of your system into thrash hell as soon as the load on it increases. (Obviously once you *do* start swapping speed goes to hell anyway, so the overhead of taking TLB misses is lost in the noise. In the long run, defragmenting memory on swapin and trying to make bigger pages out of it without app intervention seems like a good idea, especially on platforms like PPC with a range of page sizes more useful than 4Kb/1Gb.)

IIRC something similar to this being discussed in the past, maybe as part of the defragmentation work? ... I also dimly remember Linus being violently against it but I can't remember why.

About NUMA

cma — 2010-02-18T18:17:55+00:00

Mel, thanks A LOT! It's all very clear now! Best regards!

About NUMA

mel — 2010-02-18T15:00:54+00:00

> I'm curious... If my app is NUMA aware (let's say it's an old fashioned threaded app), in this
> case woould it be interesting in enabling in BIOS (in concrete for a Dell R610 server) the
> memory option NODE INTERLEAVING? Or just let BIOS enable NUMA behavior (NODE
> INTERLEVING disabled)? Thanks and congrats for this great article!

s/NUMA aware/not NUMA aware/

It depends on whether your application fits in one node or not. If it fits in one node, then leave
NODE_INTERLEAVING off and use taskset to bind the application to one nodes worth of CPUs.
The memory will be allocated locally and performance will be decent.

If the application needs the whole machine and one thread faults all of the memory before
spawning other threads (a common anti-pattern for NUMA), then NODE_INTERLEAVING will give
good average performance. Without interleaving, your performance will sometimes be great
and other times really poor depending on if the thread is running on the node that faulted the
data or not. You don't need to go to the bios to test it out, launch the
application with

numactl --interleave=all your-application

About NUMA - corrigenda

cma — 2010-02-18T14:36:34+00:00

Sorry about the typo: I was meaning: if my app WAS NOT NUMA aware...

About NUMA

cma — 2010-02-18T14:35:10+00:00

I'm curious... If my app is NUMA aware (let's say it's an old fashioned threaded app), in this case woould it be interesting in enabling in BIOS (in concrete for a Dell R610 server) the memory option NODE INTERLEAVING? Or just let BIOS enable NUMA behavior (NODE INTERLEVING disabled)? Thanks and congrats for this great article! Regards

Well-written article

sdalley — 2010-02-18T11:12:11+00:00

Well done Mel Gorman for writing about a deeply technical topic in such a comprehensible way! I actually understood it 80% on the first pass.

It's a real art to be able to define unfamiliar concepts as you go without over-simplifying on the one hand or being lost in nested interrupts on the other.

The only things I had to look up were MPI and OpenMP but that was easily done.