Betrayed by a bitfield

Posted Feb 4, 2012 21:47 UTC (Sat) by giraffedata (guest, #1954)
In reply to: Betrayed by a bitfield by nix
Parent article: Betrayed by a bitfield

That's very interesting, because it seems to say the provision for "volatile" is so incomplete as to be a pointless language feature. "volatile" was supposed to deal with C programs running in an address space that includes memory mapped I/O regions. But if the compiler is allowed to write whatever it wants whenever it wants into I/O regions, just as long as it doesn't expect it to stay there, what's the point? You can't confidently run a C program in an address space that contains memory mapped I/O regions.

Maybe it's useful in read-only memory mapped I/O regions that just ignore writes.

Betrayed by a bitfield

Posted Feb 4, 2012 22:25 UTC (Sat) by nix (subscriber, #2304) [Link] (2 responses)

That's very interesting, because it seems to say the provision for "volatile" is so incomplete as to be a pointless language feature.

I dunno. As long as compiler vendors cooperate, it's useful for its intended purpose, which was pretty much entirely *reading* from memory-mapped I/O regions and writing into them very carefully, in a serial environment. It wasn't particularly intended for parallel processing environments or multithreading, AIUI (though since I was only 13 when C89 was finalized, I obviously wasn't there and don't know for sure).

In general you can't (and never could) rely on 'volatile' without knowing what the compiler would do when you asked it: implementations have had horrible bugs in their treatment of volatile often enough that care is warranted. But at the very least, it provides a way to flag that 'something tricky is happening here, look out'. It's proved more useful than 'register', low bar though that is. :)

Betrayed by a bitfield

Posted Feb 4, 2012 22:37 UTC (Sat) by giraffedata (guest, #1954) [Link] (1 responses)

Right, it's a useful feature of de facto C, but maybe not of standard C. This is just an academic discussion about the standard, since it seems everyone agrees GCC needs to change regardless of whether it presently implements the standard.

As long as you know that storing to your memory mapped I/O regions is a no-op, you can use standard C with volatile to usefully fetch from them, but otherwise you need more than standard C. You need common sense C. It just seems strange to me because writable I/O regions did exist at the time this accomodation for reading them was added to the spec.

Betrayed by a bitfield

Posted Feb 6, 2012 19:21 UTC (Mon) by chrisV (guest, #43417) [Link]

"This is just an academic discussion about the standard, since it seems everyone agrees GCC needs to change regardless of whether it presently implements the standard."

I absolutely don't agree with this. Forbidding 64-bit read/writes for 32-bit scalars within an aligned 64-bit boundary is a pessimization which should be opted into in C99, such as with the -pthread flag as in current gcc implementations (-pthread does require this pessimization if adjacent memory locations would otherwise be corrupted).

Presumably when C11 is implemented in gcc, opting out of this pessimization will also be available. Or possibly gcc will require some specific flag to be set where a multi-threaded program is being compiled, who knows.

Betrayed by a bitfield

Posted Feb 6, 2012 18:08 UTC (Mon) by daglwn (guest, #65432) [Link] (20 responses)

> That's very interesting, because it seems to say the provision for
> "volatile" is so incomplete as to be a pointless language feature.

Not totally true. It's true that volatile doesn't do what most people expect. Basically, volatile says the data can't be cached in a register (because the unerlying value might change unexpectedly), and little else. But the prohibition on register allocation is a pretty big deal to the compiler.

Betrayed by a bitfield

Posted Feb 6, 2012 19:27 UTC (Mon) by dlang (guest, #313) [Link] (16 responses)

if it's not allowed to be cached in a register (because it could change unexpectedly), why are you allowed to do a read-write cycle on it? isn't that effectively the same thing?

Betrayed by a bitfield

Posted Feb 6, 2012 20:01 UTC (Mon) by chrisV (guest, #43417) [Link] (4 responses)

I am not sure if it answers your question, but gcc would only be non-conforming if it carried out a corrupting read-write cycle on the memory location of a volatile variable as also read by an interrupt in the same thread attempting to read the same variable (or vice-versa), where the volatile variable is of type sig_atomic_t.

This is orthogonal to the original problem, which (as far as I can work out) is concerned with concurrent access by different threads.

Betrayed by a bitfield

Posted Feb 6, 2012 20:06 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

the thing is, the current GCC behaviour _would_ overwrite the value of a volatile variable that was modified by an interrupt in the middle of the read-modify-write cycle of a different (but adjacent) variable.

As Linux says, it's possible for someone to fix the bug with respect to volatile, and explicitly leave the bug in for all other accesses (with the claim that the spec allows the compiler to do that), but it would probably be more work to do that then to just change the behaviour across the board.

Betrayed by a bitfield

Posted Feb 6, 2012 20:51 UTC (Mon) by chrisV (guest, #43417) [Link] (2 responses)

"the thing is, the current GCC behaviour _would_ overwrite the value of a volatile variable that was modified by an interrupt in the middle of the read-modify-write cycle of a different (but adjacent) variable."

I don't believe that has been tested and I don't believe it to be true. On hardware where 64 bit boundaries are important, sig_atomic_t would be likely to be 64-bits wide. (Someone with a ia64 architecture might be able to confirm this.) In that case alignment boundaries would be fully respected.

Also, I can't see that the original problem had anything to do with interrupts. It seems to be concerned with either multi-threading, or shared access by multiple processes.

Lastly, when you refer to "the bug with respect to volatile", you impute that gcc has the bug. If there is a bug, it is a bug in the kernel.

Betrayed by a bitfield

Posted Feb 6, 2012 21:04 UTC (Mon) by dlang (guest, #313) [Link] (1 responses)

one of the posts I saw on this topic (I think from Linus) tested the code snippet

volitile int a;
int b;

b++;

and the resulting assembler did the exact same read-write on a that caused the problem with the case in question. If an interrupt were to happen between the read and the write that changed the value of a, the write would cause that value to be lost.

no, the item in question wasn't related to interrupts, but the cause of the problem causes the same problem if interrupts were involved.

If the 32 bit loads and stores were significantly more expensive to use than 64 bit loads and stores, there would be at least some reason to have this behaviour available, but in the case of the architectures in question there isn't a performance benefit to doing it this way.

Betrayed by a bitfield

Posted Feb 6, 2012 21:16 UTC (Mon) by chrisV (guest, #43417) [Link]

"no, the item in question wasn't related to interrupts, but the cause of the problem causes the same problem if interrupts were involved."

Not it doesn't. Memory access issues relating to multiple threads (or multiple processes in the case of shared memory) have nothing to do with interrupts, and nothing to do with volatile.

This really is beating a dead horse. Compiler switches are available to ensure memory consistency for the case in question, notably the -pthread switch. I can see why the kernel doesn't want to, or can't, use that. In that case the kernel authors need to write some assembler or use 64-bit values on architectures where it is important, or persuade the gcc developers to provide another compiler switch dealing with this particular problem.

Betrayed by a bitfield

Posted Feb 6, 2012 20:05 UTC (Mon) by daglwn (guest, #65432) [Link] (10 responses)

The key work is "cached." On any load/store machine the data must be first loaded into a register to do anything with it. volatile says it must be written back out by the next sequence point. I'm sure language lawyers will find some detail I've missed but that's the way I think about the guarantee.

Betrayed by a bitfield

Posted Feb 6, 2012 20:13 UTC (Mon) by dlang (guest, #313) [Link] (9 responses)

right, but the programmer is not attempting to do anything with it. The programmer is attempting to do something with another variable, one that just happens to be adjacent to the one in question.

again the code snippit is

volitile int a;
int b;

b++;

if modifying b causes a read/write of a, this is wrong.

the programmer has not made any attempt to specify alignment here.

Betrayed by a bitfield

Posted Feb 7, 2012 0:48 UTC (Tue) by daglwn (guest, #65432) [Link] (8 responses)

> if modifying b causes a read/write of a, this is wrong.

No, it's not. Believe me.

The volatile keyword doesn't say anything about when it will change value, be read/written etc. It says simply that it will not be cached in a register such that every read will get the "latest" value expected when executed under the Abstract Machine.

It says nothing about threading.

It says nothing about interrupts.

Simply remember that volatile is not magic. Think of it as the opposite of "register."

Betrayed by a bitfield

Posted Feb 7, 2012 1:03 UTC (Tue) by dlang (guest, #313) [Link]

in that case I have to agree with the other poster who said that if the compiler considers it Ok to write over any arbitrary memory locations, as long as what it's writing matches what the compiler thinks is already there, then that compiler is unsuitable for use with any memory mapped I/O as it will feel free to clobber the new data that is waiting to be read.

since this sort of thing has been part of C's traditional strength, this doesn't seem like a sane interpretation to me.

Betrayed by a bitfield

Posted Feb 7, 2012 16:54 UTC (Tue) by chrisV (guest, #43417) [Link] (6 responses)

"It says nothing about interrupts."

I think it does. See §5.1.2.3/5 and /10, which are normative. The principal purpose of volatile is to deal with arbitrary changes of data values outside the program context of the process in which the code is running (ie asynchronous interrupts). (This does not include threads, which are within the program context and which, because they can run on more than one core, require quite different synchronizations, some of which are not async-signal-safe.)

See also the footnote 134 of §6.7.3/8 (which is non-normative despite the "shall not"): "A volatile declaration may be used to describe an object corresponding to a memory-mapped input/output port or an object accessed by an asynchronously interrupting function. Actions on objects so declared shall not be 'optimized out' by an implementation or reordered except as permitted by the rules for evaluating expressions." This is a curious note as, as far as I am aware, it is the one and only reference to memory mapping (and about which I mis-spoke in an earlier posting on this article because it is not in C99 which contains no reference to memory mapping).

Betrayed by a bitfield

Posted Feb 7, 2012 18:38 UTC (Tue) by daglwn (guest, #65432) [Link] (5 responses)

> "It says nothing about interrupts."

Thanks for the correction. But the compiler is still correct here. Volatile doesn't say anything about restricting _when_ it is read or written, only that it will get the "latest" value in a single-thread context.

It's perfectly fine for I/O as long as you can guarantee alignment such that there is no "false sharing."

Betrayed by a bitfield

Posted Feb 7, 2012 19:07 UTC (Tue) by chrisV (guest, #43417) [Link] (4 responses)

I agree. And if you get false sharing between two free-standing variables where the one being operated on is marked volatile, or between fields of a struct where the struct is marked volatile, there is a compiler bug. It is still not clear whether that is the case with the kernel test case (first, the struct was not marked volatile, only one of its fields; and secondly, we don't know whether the test case involved an asynchronous test (as opposed to threads) or not.

Betrayed by a bitfield

Posted Feb 7, 2012 20:01 UTC (Tue) by dlang (guest, #313) [Link]

the kernel did not have one field marked volatile, but in the research into the problem, someone (I think it was Linus) tested with volatile and the false sharing was happening there as well.

Betrayed by a bitfield

Posted Feb 7, 2012 23:32 UTC (Tue) by daglwn (guest, #65432) [Link] (1 responses)

> And if you get false sharing between two free-standing variables where the > one being operated on is marked volatile, or between fields of a struct
> where the struct is marked volatile, there is a compiler bug.

No, there isn't.

There isn't. Really.

Volatile does not mean what you think it means.

It's a bit like sequential consistency. Just when you think you understand it, something unexpected happens that is both non-intuitive and perfectly legal.

Betrayed by a bitfield

Posted Feb 8, 2012 15:24 UTC (Wed) by daglwn (guest, #65432) [Link]

Seeing some of your other posts about -pthread, I think we are in agreement. Apologies if I mischaracterized your understanding.

Betrayed by a bitfield

Posted Feb 8, 2012 13:51 UTC (Wed) by nix (subscriber, #2304) [Link]

And, thirdly, the kernel is not C11 code -- yet.

Betrayed by a bitfield

Posted Feb 6, 2012 22:22 UTC (Mon) by giraffedata (guest, #1954) [Link] (2 responses)

That's very interesting, because it seems to say the provision for "volatile" is so incomplete as to be a pointless language feature.
Not totally true. It's true that volatile doesn't do what most people expect. Basically, volatile says the data can't be cached in a register ...

I was commenting at a higher level. Never mind what the compiler can and can't do. The question is, what programs can you write in C because it has the "volatile" feature that you couldn't otherwise? Relying on nothing but that the compiler implements the C standard.

I've always understood the originally intended answer to be, "you can write a program that uses memory mapped I/O." But comments in this thread say regardless of how one uses the "volatile" keyword in a program, the compiler may always generate code that arbitrarily writes to memory mapped I/O addresses. There's nothing in the standard to stop it. If so, then you not only can't use memory mapped I/O in a C program, you can't even let a C program — any C program — run in an address space that includes memory mapped I/O regions.

Unless all memory-mapped I/O regions ignore writes.

And it's hard to believe that the definers of "volatile" actually had such a useless thing in mind.

Betrayed by a bitfield

Posted Feb 7, 2012 0:52 UTC (Tue) by daglwn (guest, #65432) [Link] (1 responses)

Again, this is implementation-defined behavior. The standard you're looking for is the ABI for your target. That's where all the memory layout, access size, calling convention and other low-level stuff is specified.

Volatile is perfectly fine for I/O as long as you know the address being accessed is suitably aligned to avoid problems, as the ABI should indicate.

This is why volatile is non-portable. Unfortunately, C99 has no standard way to force alignment of any object.

Betrayed by a bitfield

Posted Feb 7, 2012 8:46 UTC (Tue) by khim (subscriber, #9252) [Link]

This is why volatile is non-portable. Unfortunately, C99 has no standard way to force alignment of any object.

GCC, MSCV and other compilers include such an ability and C11 finally adds it to standard so it's all is not so bad...