LWN: Comments on "Fun with NULL pointers, part 1"

Fun with NULL pointers, part 1

gvy — Sat, 15 Aug 2009 09:01:39 +0000

Yeah, previously one had to introduce a rootkit. Now it's half-done by prominent distributors. :(

Fun with NULL pointers, part 1

jschrod — Fri, 07 Aug 2009 12:21:44 +0000

I'll have to say that I don't find PolicyKit's documentation very good. I don't know how it is for a developer, but for me as a long-time Unix admin it was frustrating when I had to delve into it a few weeks ago for the first time.

The document linked by you was one of the 1st that I read, btw. It's not that I didn't understand what PolicyKit was supposed to be doing, that was quite clear from the start. And not even that I couldn't read the config files -- while being rather chatty, they were understandable.

Where I didn't succeed, is finding concise information about the overall architecture: how PolicyKit / ConsoleKit / HAL / D-Bus / pam / login processes, both X and console, are *supposed* to work together. That would have delivered pointers to information sources that could have answered questions like Where are the names in the PolicyKit XML files from? Which possible names exist on a given system? Which daemons are started and by whom? What are common PolicyKit authorizations, since it is all about policy and nothing about content? The output of polkit-auth --show-obtainable is not sufficient, IMHO. E.g., I'd need to know what org.freedesktop.hal.lock is, and googling it on site:freedesktop.org does _not_ provide an adequate answer, at least not for me. (And yes, I've read http://people.freedesktop.org/~david/hal-spec/hal-spec.html.)

It was not easy to find the answers to such questions in reasonable time frames because the overall structure had to be laborously reverse engineered by yours humbly. Well, maybe my Google foo may simply be not good enough.

Just my 0.02 EUR. :-)

Joachim

Optimizations and undefined behavior

hozelda — Fri, 31 Jul 2009 14:28:03 +0000

I know the conversation is partly about what developers should depend on or not, but keep in mind that "undefined behavior" and "implementation-defined behavior" mean something precise and completely different from each other in C99. See Chapter 3 (and 4) and informative Appendix J http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1336.pdf

Optimizations and undefined behavior

nix — Thu, 30 Jul 2009 21:31:23 +0000

fsync(), brought to you by the same people who thought up 'volatile',
another equally impossible-to-define-except-by-reference-to-implementation
feature.

Optimizations and undefined behavior

foom — Thu, 30 Jul 2009 15:04:31 +0000

> I remember GCC taking out tests like if(x+n>x)

Still does. You can use -fwrapv if you want to tell it that you want signed number overflow to be
defined as wrapping.

Optimizations and undefined behavior

forthy — Thu, 30 Jul 2009 13:38:30 +0000

The problem with fsync() is that it's semantics resemble whery much the one of "PLEASE" in INTERCAL, which means it is a joke. fsync() basically has no semantics, except "make it so" (make it persistent now). Now, all file system operations are persistent, anyway, just not made persistent now. You can't properly test (that is: automated), because to test if there's a missing fsync(), you have to force an unexpected reboot, and then check if there's any missing data. What's worse: A number of popular Unix programming languages don't even have fsync(), starting with all kinds of shell scripts. fsync() is a dirty hack introduced into Unix because of broken (but extremely fast) file system implementations.

We know that Unix is a joke for quite some time, but parts of the API like fsync() show that this is not so far away from the truth ;-). From the kernel development side it is always "easier" to maintain a sloppy specification and blame the loser, but that's the wrong thinking. You are providing a service. Same thing for GCC: Compiler writers provide a service. Using a sloppy specification for questionable "optimizations" is wrong, as well. If the compiler writer can't know that the code really will break when accessing the NULL pointer, then he can't take the test out after having accessed an object. I remember GCC taking out tests like if(x+n>x), because overflows are said to be unspecified in the C language, but compiling code to a machine where overflows were very specifically handled as wraparounds in two's complement representation. This is all wrong thinking.

Optimizations and undefined behavior

lysse — Thu, 30 Jul 2009 13:31:05 +0000

> "Implementation dependent" != `cat /dev/urandom`

Wasn't that where Bruce came in...? ;)

A zero pointer is not a null pointer

nix — Sun, 26 Jul 2009 18:27:32 +0000

Well, you can't access a structure at address zero on such a platform
without either disabling all optimizations that involve knowing which
pointers are null (as the kernel now is) and taking great care to ensure
that you never need anything that can point to said structure to be NULL
at any time, or defining the null pointer to be other than all-bits-zero
(allowed, but weird, about as rare as platforms with strange word sizes).

I'd say that trying to access structures at address zero, MMU or no MMU,
is extremely unusual and not really sane to handle in a general-purpose
compiler. (GCC goes further than I would expect in actually having a
switch that makes it possible to use such a barmy thing.)

A zero pointer is not a null pointer

PaXTeam — Sat, 25 Jul 2009 23:18:38 +0000

> Data structures at address zero do not exist on any sane C platform.

so platforms without an MMU are not sane?

A zero pointer is not a null pointer

nix — Sat, 25 Jul 2009 12:49:12 +0000

GCC doesn't ignore the standard: you simply can't have data structures
that reside at address zero. Shaving off 1/2^32 or less of the address
space, and disabling an optimization in the one place that cares about
this (the kernel) does not seem like a terrible cost to me.

Data structures at address zero do not exist on any sane C platform.

A zero pointer is not a null pointer

spitzak — Sat, 25 Jul 2009 09:37:55 +0000

That won't work, plenty of code assumes that comparing two NULL pointers for equality will return
true.

A zero pointer is not a null pointer

giraffedata — Sat, 25 Jul 2009 02:51:47 +0000

GCC certainly doesn't insert code checking if a pointer is NULL before every pointer dereference.

Sure, but that wouldn't improve standards compliance anyway. I asked if GCC generates extra code to comply with the C99 requirement that a null pointer not be equal to any non-null one (while still allowing the existence of pointers to a data structure that resides at address 0). Thinking about it now, though, I don't see how any such code is possible since a null pointer still has to compare equal to another null pointer.

That kernel space has to work when the lower part of its address space is effectively under the control of a hostile attacker is a unique problem which it is really not worth changing the C standard for,

There has been no proposal to deal with this by changing the standard, which GCC apparently ignores anyhow. And objection to GCC's conflation of null pointers and zero-address pointers wasn't that it's a security problem but that it's a basic correctness problem. Even without a hostile page 0, unless you proclaim data structures at address 0 don't exist, this optimization breaks code.

Fun with NULL pointers, part 1

mikov — Sat, 25 Jul 2009 02:13:23 +0000

I am with you.
In my mind the key to reasonable security is simplicity. SUID is extremely simple. Compare changing a user's password using passwd to what has to happen in Windows.

A zero pointer is not a null pointer

nix — Fri, 24 Jul 2009 22:06:06 +0000

GCC certainly doesn't insert code checking if a pointer is NULL before
every pointer dereference. Considering how common pointer dereferencing is
in C and related languages, this would be a substantial slowdown for no
real gain, given that multiuser OSes invariably trap such things, and
non-multiuser OSes are specialist environments in which attacks by hostile
local users are not so common (yet).

That kernel space has to work when the lower part of its address space is
effectively under the control of a hostile attacker is a unique problem
which it is really not worth changing the C standard for, nor imposing
vast overheads on all userspace code. -fno-delete-null-pointer-checks does
the job.

A zero pointer is not a null pointer

giraffedata — Fri, 24 Jul 2009 20:38:29 +0000

It would be possible to keep 0x0 for the null pointer while respecting the C99 standard: the compiler would need to put in a couple of extra instructions for every pointer comparison making sure that 0x0 != P for any value of P.

Do we know Gcc doesn't do this? Seems like it would have to, to be C99 compliant.

Optionally, it could also put in a check before every pointer dereference making sure the pointer is not 0x0 and causing a SIGSEGV if it is (since the memory layout can no longer be relied on to guarantee that).

But then how would you represent a pointer to a data structure that resides at address 0? A pointer should be able to do that.

It would of course be unrealistically expensive on typical machines to represent a pointer with anything but a simple address, but pointer comparisons are rare enough that a few extra instructions for them seems worthwhile to maintain the null pointer concept.

Regardless of how the compiler chooses to represent pointers (null or otherwise), the optimization in question is logically sound. C99 says a dereference of a null pointer causes undefined behavior, so either a) tun is non-null and !tun must be false or b) tun is null and !tun can be anything, including false.

Fun with NULL pointers, part 1

rogue@autistici.org — Fri, 24 Jul 2009 16:06:49 +0000

And, may you be glorified for your find as much as spender is for his exploit.

Optimizations and undefined behavior

johnflux — Thu, 23 Jul 2009 05:07:21 +0000

I hope not. Good developers will realize that fsync is a complete overkill there. They don't need wait for the changes to actually be made to disk before continuing - they only need to make sure that the changes happen in the right order.

Fun with NULL pointers, part 1

johnflux — Thu, 23 Jul 2009 05:01:18 +0000

You, sir, are a coding god.

Fun with NULL pointers, part 1

nix — Wed, 22 Jul 2009 22:00:39 +0000

Ew, no. Utterly un-awklike and doing awk-like transformations with XSLT is
really quite painful. (And yes, you can do awklike languages for things
other than text streams: see gvpr(1) for example.)

(One of many problems is XSLT's heavy use of <>, which makes it very
annoying to use from the shell prompt. Another is its astonishing
verbosity. Another is its total lack of good taste in design... also the
functional nature of it, while one of its nicer aspects, fits very badly
with the shell in my experience.)

Fun with NULL pointers, part 1

nix — Wed, 22 Jul 2009 21:59:08 +0000

I think one is essential if PolicyKit can be considered safe to use: we
have to be able to do automated audits and yell at unexpected quiet
changes, lest we miss attacks or accidental fumbles opening up holes.

You're quite right on the fugliness of the UID-setting syscalls in Unix,
but this is not helped by introducing *more* :/ again, userv did all this
better (IMNSHO) many years ago. Why more people don't use it I have no
idea.

RISC can do that

klossner — Wed, 22 Jul 2009 18:13:58 +0000

PowerPC silently drops the two low bits when loading an address into the PC, so a branch to 1 becomes a branch to 0. The misaligned-address exception occurs only for load/store instructions.

#pragma and GCC

zeekec — Wed, 22 Jul 2009 15:46:25 +0000

Wouldn't that be a great response to a NULL dereference in the kernel.

Fun with NULL pointers, part 1

mezcalero — Wed, 22 Jul 2009 14:24:48 +0000

pa 0.9.16 is not setuid anymore, because the introduction of rtkit makes that unnecessary.

Fun with NULL pointers, part 1

cortana — Wed, 22 Jul 2009 12:49:10 +0000

It's not GNOME-specific. It just uses GTK+. Yes, it's unfortunate that there is no command-line equivalent, but it shouldn't be too difficult for one to be written--you are just querying the PolicyKit object after all.

Fun with NULL pointers, part 1

nix — Wed, 22 Jul 2009 11:19:46 +0000

So the only way you can see what actions can be run on a system with privilege is... to run a GNOME-specific GUI application?

*sigh*

xml awk

nix — Wed, 22 Jul 2009 11:18:59 +0000

That looks nice. Not very awkish though...

A zero pointer is not a null pointer

epa — Wed, 22 Jul 2009 09:02:41 +0000

It would be possible to keep 0x0 for the null pointer while respecting the C99 standard: the compiler would need to put in a couple of extra instructions for every pointer comparison making sure that 0x0 != P for any value of P. Optionally, it could also put in a check before every pointer dereference making sure the pointer is not 0x0 and causing a SIGSEGV if it is (since the memory layout can no longer be relied on to guarantee that).

xml awk

Frej — Wed, 22 Jul 2009 07:08:00 +0000

xml cli tool
http://xmlstar.sourceforge.net/

Optimizations and undefined behavior

mjg59 — Wed, 22 Jul 2009 04:16:05 +0000

Good developers aren't going to have to - good operating systems will provide guarantees above and beyond POSIX. Operating systems that don't will end up poorly supported and gradually become irrelevant.

Optimizations and undefined behavior

BrucePerens — Wed, 22 Jul 2009 03:28:40 +0000

Sigh. Good developers are still going to do create temp, write, fsync file, link to permanent name, unlink temp. Fsync on the directory, though, shouldn't be necessary.

Fun with NULL pointers, part 1

Baylink — Wed, 22 Jul 2009 02:18:52 +0000

Am I the only person who read "has checks for this" and thought "a wart is a crocky feature"?

Fun with NULL pointers, part 1

Baylink — Wed, 22 Jul 2009 02:14:12 +0000

"vulnerability surface".

Wow. That's an altogether neat concept. Thanks.

Optimizations and undefined behavior

mjg59 — Wed, 22 Jul 2009 01:39:50 +0000

If a system left truly random numbers in uninitialised variables and applications started making use of that functionality (even if undocumented), then I think you'd need a very good reason to change that behaviour. But since I can't imagine any sane system ever doing that, no, I'm not proposing that we need a user contract on that point.

Contrast that to the ext3 behaviour. While, yes, the behaviour of fsync() on ext3 did result in people being even less likely to use it, the fact that ext3 also made it possible to overwrite a file without having to go through a cumbersome sequence of fsync()ing both the data and the directory made it attractive to application writers. That behaviour dates back far beyond Firefox 3, as demonstrated by people's long-term complaints about XFS leaving their files full of zeros after a crash. ext4 now provides the same ease of use because people made it clear that they weren't going to use it otherwise. Future Linux filesystems are effectively forced to provide the same semantics, which is a good thing.

Optimizations and undefined behavior

BrucePerens — Wed, 22 Jul 2009 01:28:41 +0000

Matt, you aren't seriously proposing that we provide some sort of user contract regarding the contents of uninitialized variables being reliable sources of entropy.

Regarding ext3, this problem came up because fsync() was implemented as a performance pig, at least in ext3, and rather than fix it we trained application developers that they'd be safe without it. Had fsync() been repaired when the mozilla problem came up, nobody would be arguing about it today.

Optimizations and undefined behavior

mjg59 — Tue, 21 Jul 2009 22:39:48 +0000

The role of software is to be useful to its consumers. Software that fails in this will tend to end up being ignored in the long run.

#pragma and GCC

BrucePerens — Tue, 21 Jul 2009 21:48:48 +0000

Some time in the 80's, when RMS was still coding GCC, the response to #pragma in C was explicitly stated to be undefined, so that your compiler could have whatever magic happen that it wished.

So, RMS coded GCC to start the game "rogue" when it saw #pragma.

Optimizations and undefined behavior

BrucePerens — Tue, 21 Jul 2009 20:55:17 +0000

The recent ext3 fsync() snafu is a good example of implementation dependent behavior becoming taken as an implicit guarantee. And then the developer had to reduce the scope of the promise for performance reasons. He ended up regretting that he had ever made that feature visible.

Anyway, there was no such guarantee in this case.

Optimizations and undefined behavior

martinfick — Tue, 21 Jul 2009 20:11:02 +0000

"Implementation dependent" means that the folks who change the compiler, the C library, the kernel, and any stack-smashing detection/prevention code that you are using have the right to change the behavior at any time without documenting the particular side-effect that your code is depending upon.

No, it does not always mean this. In this particular case it may, but it is perfectly reasonable for a specific C compiler to specify its behavior when the standard says that it is undefined, that is the point of "undefined" and "Implementation dependent".

"Implementation dependent" != `cat /dev/urandom`

Optimizations and undefined behavior

BrucePerens — Tue, 21 Jul 2009 20:04:38 +0000

Especially in the context of the various stack-smashing prevention hacks going around, you have no reason to believe that an uninitialized variable would not actually be initialized to a fixed value.

Undefined stuff is not a service that you can count on. Ever.

Optimizations and undefined behavior

martinfick — Tue, 21 Jul 2009 19:42:21 +0000

it means "implementation dependent".

which as others have pointed out, is the purpose of undefined in the C spec, which does not mean that it has "No correct cases.", but rather, that you'd better be familiar with the implementation you are using. It makes no sense to say: "Here is a feature not available elsewhere, that we think it valuable, but you should never use it!" does it?