LWN: Comments on "MAP_FIXED_SAFE"

the problem isn't MAP_FIXED...

HelloWorld — Thu, 21 Dec 2017 05:42:58 +0000

...but its weird replacing behaviour when a mapping already exists. The question is: is there any program that relies on this behaviour to function? I'd be surprised if that were the case.

How about this?

mdenton — Wed, 20 Dec 2017 22:24:14 +0000

I have previously worked on a filesystem that provided transactional memory; we needed MAP_FIXED to keep internal addresses the same when a transactional "file" was mapped in multiple processes. We could've of course used relative addresses, but that comes with a not insignificant performance cost, and would've required some C++ magic we didn't really want to deal with.

Terminology wisdom

ortalo — Tue, 19 Dec 2017 09:56:32 +0000

Yeah! +1 - love it (the last one) ..._AND_SECURE2

Terminology wisdom

ortalo — Tue, 19 Dec 2017 09:53:37 +0000

In the long term, okay. But let's not let economics discussions distract us from the security concerns...

Terminology wisdom

ortalo — Tue, 19 Dec 2017 09:50:46 +0000

Well, in my humble opinion, the plot end is more due to the environment than to terminological issues. Let's hope there are not too many well-meaning advisors in the development team and historical partisan disputes have been banned.
That's the appropriate time of year for dreaming, no?

Terminology wisdom

shane — Mon, 18 Dec 2017 13:46:35 +0000

Fair enough, although I do think that it should be called MAP_FIXED_SAFER, because calling it "SAFE" is almost going to guarantee an exploit at some point, and then we will need MAP_FIXED_REALLY_SAFE, and probably MAP_FIXED_SAFE_FOR_REALZ_THIS_TIME...

How about this?

dtalen — Sat, 16 Dec 2017 14:47:54 +0000

FYI, I believe position independent builds have only just become default for the Gentoo distribution.

How about this?

epa — Sat, 16 Dec 2017 13:01:05 +0000

The idea was that if you see MAP_FIXED then you know that this particular bit of code hasn't yet been audited and changed to either SAFE or UNSAFE. So checking your codebase is a simple grep. I guess as you say you can "#define MAP_FIXED_UNSAFE MAP_FIXED" and then use the new keyword.

Terminology wisdom

cpitrat — Sat, 16 Dec 2017 10:58:44 +0000

Unfortunately I'm pretty sure they'll all end-up dead ...

How about this?

cpitrat — Sat, 16 Dec 2017 10:57:19 +0000

I think the idea is to help people clarify their intent in their code. They can easily do that with a #define though so I don't see a real benefit in this proposition.

Terminology wisdom

mogendavido — Sat, 16 Dec 2017 00:56:56 +0000

Note though that in spite of Juliet's argument, both she and Romeo ended up dead. Hopefully the same won't happen to the participants in this fight ;-)

Terminology wisdom

ortalo — Fri, 15 Dec 2017 10:48:40 +0000

"What's in a name? that which we call a rose
by any other word would smell as sweet;"

How about this?

ballombe — Thu, 14 Dec 2017 19:30:40 +0000

This is true, however there is a difference if you use vm.overcommit=2.
Using MAP_FIXED allows the memory not to be counted as committed.
Sometime this is useful.

MAP_FIXED is necessary for almost all execve() and dlopen()

roc — Thu, 14 Dec 2017 19:14:24 +0000

rr is another example of a program-manipulating program that absolutely needs MAP_FIXED.

How about this?

zlynx — Thu, 14 Dec 2017 17:16:26 +0000

Some programmers prefer to mmap a large block and then mprotect the pieces of it instead of using MAP_FIXED.

How about this?

magfr — Thu, 14 Dec 2017 15:58:02 +0000

MMAP_FIXED is part of POSIX so if you change the semantics of it then I would expect libc to add a wrapper that works around the broken kernel implementation.

How about this?

Sesse — Thu, 14 Dec 2017 15:56:41 +0000

Who do you propose should go through all code that exists in the world (open and closed source alike) and do this audit?

How about this?

epa — Thu, 14 Dec 2017 08:53:16 +0000

A new name MAP_FIXED_UNSAFE should be added which is an alias of the existing MAP_FIXED. Then existing code can be audited and changed to the safe variant if possible, or explicitly marked as requiring the unsafe call.

MAP_FIXED is necessary for almost all execve() and dlopen()

jreiser — Thu, 14 Dec 2017 05:58:26 +0000

> In 2017 I don't think there is any good use for MAP_FIXED.

MAP_FIXED is necessary for the proper mapping by execve() of any native ET_EXEC file. The addresses are fixed, after all. There are ET_EXEC files which are several percent smaller and run several percent faster than the corresponding ET_DYN. Some environments have done a good risk-vs-benefit analysis, and for them ET_EXEC is worth it.

MAP_FIXED is necessary for the proper mapping by execve() and dlopen() of any native ET_DYN file which has more than one PT_LOAD, which is nearly all of them. The second and subsequent PT_LOADs must have fixed offsets from the first PT_LOAD. In most cases the best strategy is to use mmap(0, size, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, FD_ANON, 0) to obtain enough contiguous space to hold the convex hull of all PT_LOAD, then mmap() each PT_LOAD into that space using MAP_FIXED.

MAP_FIXED is essential for program-manipulating programs such as valgrind and upx.

How about this?

TheJH — Thu, 14 Dec 2017 01:49:35 +0000

> In 2017 I don't think there is any good use for MAP_FIXED.

MAP_FIXED is how you properly, safely allocate virtually contiguous VMAs. This includes:
- guard pages for thread stacks
- the different sections for library mappings (you need separate VMAs for code, readonly data, copy-on-write data and zero-initialized data)

Try running strace on any binary that is dynamically linked and/or uses threads, and you should see MAP_FIXED.

How about this?

TheJH — Thu, 14 Dec 2017 01:44:00 +0000

Because that would break existing, perfectly legitimate users of MAP_FIXED.

Look at, for example, how glibc allocates memory for threads:

$ cat thread.c
#include <pthread.h>
void *testfn(void *arg) { return (void*)0; }
int main(void) {
pthread_t thread;
pthread_create(&thread, NULL, testfn, (void*)0);
return 0;
}
$ gcc -o thread thread.c -Wall -pthread
$ strace ./thread
[...]
mmap(NULL, 3795360, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b24276000
mprotect(0x7f9b2440b000, 2097152, PROT_NONE) = 0
mmap(0x7f9b2460b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f9b2460b000
mmap(0x7f9b24611000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b24611000
[...]

MAP_FIXED is perfectly safe as long as you only pass in address ranges that you have allocated yourself - just like other APIs, like munmap() or mprotect(), which you also shouldn't call on address ranges that haven't been allocated to you. Unlike MAP_FIXED, the suggested MAP_FIXED_SAFE is for a very uncommon usecase and actually requires more care to get right. (Which is also why MAP_FIXED_SAFE isn't exactly a good name - it suggests that it is a "safer" alternative to MAP_FIXED, which it simply isn't in what I think is probably the majority of usecases.)

How about this?

wahern — Thu, 14 Dec 2017 01:35:12 +0000

s/hardware failures/filesystem corruption/. At various times hardware corruption was believed to be the primary culprit.

How about this?

wahern — Thu, 14 Dec 2017 01:33:14 +0000

In 2017 I don't think there is any good use for MAP_FIXED. Not only has position independence been best practice for decades, for both normal toolchains as well as bespoke applications, these days it's all but mandatory to be position independent to avoid being an exploit target.

I once wrote a Bayesian SPAM token database that self-repaired file corruption. (The devices it ran on were seeing substantial hardware failures, which years later was tracked down to XFS bugs.) It used a red-black tree and thus internal pointers, and I very briefly considered MAP_FIXED; but even circa 2003 it just wasn't worth baking in such an anachronism, and even when disregarding the headaches it would impose to utilize multiple databases concurrently.

But the reason the default semantics can't be changed is because of the Linux backwards compatibility guarantee. Also, while I've never used MAP_FIXED nor see myself ever using it, I don't find the semantics of silent replacement odd. dup2 has the same semantics, and for the same reasons the semantics seem quite natural and intuitive to me. It's just that back when people regularly mmap'd stuff to fixed addresses sbrk() was the primary (if not only) means of growing the heap, and static linking was the normal (if not only) method of linking.

How about this?

pr1268 — Thu, 14 Dec 2017 00:39:42 +0000

Some had suggested adding a separate flag to modify the behavior of MAP_FIXED, so that applications would pass something like MAP_FIXED|MAP_SAFE to mmap(). The problem with that approach is that mmap() is one of those system calls that never checked for unknown flags.

Why not do the converse of that, i.e. make MAP_FIXED default to the MAP_SAFE behavior as defined and implemented by Mr. Hocko's patch, and add the separate flag, e.g. MAP_RIGHT_HERE for the legacy behavior?

As an aside, I've used mmap(2) for years, and not once have I ever needed to use MAP_FIXED. (But I'm sure there's a good use for it somewhere...)