LWN: Comments on "Quotes of the week"

Quotes of the week

excors — Sun, 06 Jun 2021 08:42:39 +0000

> Anyway, it works in both compilers when the executable is not position-independent, which is fairly common in the embedded systems where this sort of trick is most commonly used. When you are referring to objects at fixed memory locations you usually don't want your code to be moved around in memory arbitrarily.

One exception is when you're doing dual-slot firmware upgrades, which seems fairly common in embedded devices (e.g. https://mcuboot.com/design.html in direct-xip mode). Boot the application from partition 0, download an updated application image into partition 1, boot from partition 1; and if the new application fails then the bootloader can revert to partition 0, else continue running from partition 1 until the next update (which gets installed into partition 0).

In that case it's pretty helpful to have position-independent code that can run the same from either partition. (Otherwise you have to compile the application twice at different base addresses, and figure out which one to download based on which slot is currently in use, and if you're 'downloading' directly from a second chip (instead of from the internet) then it doubles the flash usage on the second chip, etc, so that's a pain). But only references to the .text and .rodata sections should be PC-relative - any references to flash outside of the application image (e.g. if you have a filesystem that's persistent across firmware upgrades), and any references to RAM or to hardware registers, need to remain PC-independent.

It appears that's impossible in GCC (https://answers.launchpad.net/gcc-arm-embedded/+question/...), though apparently it's supported by -fropi in Clang on ARM (only).

It's a shame that C (the language and the compilers) has such inadequate and inconsistent support for relatively basic systems programming stuff like this.

Quotes of the week

nybble41 — Sun, 06 Jun 2021 03:06:48 +0000

> Your example works because you don't compile the executable PIE.

I stand corrected. I switched from GCC to Clang to use the -mpie-copy-relocations option, which seemed like it was intended to address this issue, but didn't realize that GCC and Clang have different defaults for the PIE settings and attributed the different result to the option rather than the compiler. Anyway, it works in both compilers when the executable is not position-independent, which is fairly common in the embedded systems where this sort of trick is most commonly used. When you are referring to objects at fixed memory locations you usually don't want your code to be moved around in memory arbitrarily.

One thing that did work was linking the entire program as a shared library (including main()) with a trivial front-end that just links the shared library, the CRT entry point, and the script defining the absolute symbols. The symbol is undefined in the library, which I suppose forces the access to occur via the GOT and prevents "optimization" by the linker. ASLR was enabled for both parts (confirmed with gdb with "set disable-randomization off") and it still printed the correct address for &SPECIAL_OBJECT. However, this requires the program to be split into two separate files.

> …but then the issue is that the linker will optimise the SPECIAL_OBJECT reference from a GOT load into "lea SPECIAL_OBJECT(%rip),..."…

Frankly this seems like a bug to me. The linker knows that the symbol has an absolute address since it's in the special SHN_ABS section, which is defined to be non-relocatable, and ought to refrain from rewriting a valid GOT load into something that can never give the correct address. There should at least be a setting to disable this "optimization", but I wasn't able to find any mention of it in the documentation.

Quotes of the week

Vipketsh — Fri, 04 Jun 2021 17:32:01 +0000

I don't think you can do that with pie. Your example works because you don't compile the executable PIE. Try this:

$ clang -mpie-copy-relocations -o ldtest ldtest.c ldtest.ld -fpie -pie
$ ./ldtest
&SPECIAL_OBJECT = 0x55cb19531000

As you say the compiler assumes that SPECIAL_OBJECT will be a normal data variable that can be referenced pc-relative.
You can get the compiler to do a semi-sane thing that would work by getting it to generate a GOT load to get the symbol value by using the -fpic option, but then the issue is that the linker will optimise the SPECIAL_OBJECT reference from a GOT load into "lea SPECIAL_OBJECT(%rip),..." since SPECIAL_OBJECT is guaranteed to bind locally. Few, if any, architectures other than x86 do these sort of optimisations in the linker.

Since all this is due to the deep dark voodoo of linking gcc behaves exactly the same.

Quotes of the week

nybble41 — Fri, 04 Jun 2021 16:27:51 +0000

> [excors] I think (intptr_t)(T*)0 is implementation-defined, you won't necessarily get 0 back.

This is correct. (_Bool)(T*)0 is guaranteed to evaluate to 0 due to special rules for _Bool, but all other integer conversions are implementation-defined.

> [excors] But (T*)(intptr_t)(T*)0 will give you a null pointer.

Yes, because (T*)0 is a null pointer and conversion to (only) intptr_t or uintptr_t and back without modification is defined to give back the original pointer.

>> [mpr22] Does the standard require that a pointer set to NULL contains the same bit pattern as an intptr_t set to 0, or just that if you compare a pointer set to NULL to an intptr_t set to zero, they are equal?

> [excors] You can't directly compare a pointer and an intptr_t, you have to convert them both to pointers or both to arithmetic types, in which case it depends on those conversion rules.

The result would be implementation-defined since "an intptr_t set to zero" is not "integer constant expression with the value 0" and thus does not qualify for the special treatment of (T*)0 being defined as a null pointer. In other words, "T *p = NULL; p == (T*)(intptr_t)0" should evaluate to 1, but "T *p = NULL; intptr_t n = 0; p == n" is implementation-defined since "n", as a variable rather than a constant expression, does not necessarily convert to a null pointer even if its integer value is 0 at runtime.

> …everyone just uses e.g. "(void *)0x12340000" to create a pointer to memory at 0x12340000…

Another option, relying on the ABI rather than integer-to-pointer conversions, is to declare the memory as an external object and define the address in a linker script. A useful trick here is that you can pass plain text files with an unrecognized extension to GCC or Clang (or ld/lld) and they will be treated as supplementary linker scripts, which allows for an easy way to define extra symbols:

$ cat > ldtest.ld
SPECIAL_OBJECT = 0x12340000;

$ cat > ldtest.c
#include <stdint.h>
#include <stdio.h>
extern volatile uint32_t SPECIAL_OBJECT;
int main(void) { printf("&SPECIAL_OBJECT = %p\n", (void*)&SPECIAL_OBJECT); return 0; }

$ clang -mpie-copy-relocations -o ldtest ldtest.c ldtest.ld

$ ./ldtest
&SPECIAL_OBJECT = 0x12340000

Assuming the address is valid, SPECIAL_OBJECT can be treated as an ordinary variable and used in expressions, assignments, etc.

The Clang-specific -mpie-copy-relocations option is needed when compiling position-independent executables (for ASLR) to make the compiler generate correct code for accessing an absolute, non-relocatable external symbol; otherwise an offset would be added to the address of SPECIAL_OBJECT at runtime under the assumption that it is relative to the address of the executable code. If using GCC you can disable PIE mode with "gcc -fno-pie -no-pie" to get the correct result, but this will also disable ASLR.

Quotes of the week

khim — Fri, 04 Jun 2021 08:58:40 +0000

> (Alternatively: Does the standard require that a pointer set to NULL contains the same bit pattern as an intptr_t set to 0, or just that if you compare a pointer set to NULL to an intptr_t set to zero, they are equal?)

But neither of your examples looked on bit pattern! And yes, standard guarantees that both would print 0.

What is doesn't guarantee is that the following would print 0:

void * myptr = NULL;
intptr_t myintptr;
memcpy(&myintptr, &myintptr, sizeof(myptr));
printf("%ld", (long) myintptr);

This makes it possible to implement NULL as “shifted pointer” (where it contains not address of memory, but, e.g., address of memory with one bit flipped). But this is rarely a good idea since then accessing memory becomes problematic. Especially on underpowered microcontrollers.

Quotes of the week

excors — Thu, 03 Jun 2021 19:03:34 +0000

I believe your first example is an aliasing violation (therefore undefined behaviour). If you fixed it by doing "memcpy(&myintptr, &myptr, sizeof(void*));", then the byte representation of pointers is unspecified so you could get anything.

What C99 specifically requires is that (T*)0 is a null pointer, which is equal to every other null pointer (of any type), and unequal to a pointer to any object or function. I think (intptr_t)(T*)0 is implementation-defined, you won't necessarily get 0 back. But (T*)(intptr_t)(T*)0 will give you a null pointer.

(I think I was wrong earlier when I said they "convert to 0". They only convert *from* 0.)

You can't directly compare a pointer and an intptr_t, you have to convert them both to pointers or both to arithmetic types, in which case it depends on those conversion rules.

So technically you could have a compiler which has different representations for null pointers vs pointers to memory at 0x00000000. And there's no standard way to construct a pointer to memory at a fixed location, that's always an implementation-defined thing, so the implementation could provide a way that's different to how you construct a null pointer.

But I'm not aware of any mainstream compilers/architectures that do that - everyone just uses e.g. "(void *)0x12340000" to create a pointer to memory at 0x12340000, and if you do the same for 0x0 then you'll get a null pointer.

Quotes of the week

khim — Thu, 03 Jun 2021 18:56:53 +0000

> Would you expect

int *p = malloc(sizeof(int));
free(p);
int *q = malloc(sizeof(int));
if (p == q) {
  *p = 1;
  *q = 2;
  printf("%d %d\n", *p, *q);
}

to have well-defined behaviour?

Good eye! You have caught the very important mistake: I have used pointer which I wasn't supposed to use. The correct way to write that code would be this:

int *p = malloc(sizeof(int));
free(p);
int *q = malloc(sizeof(int));
if (memcmp(&p, &q, sizeof(p)) == 0) {
  *p = 1;
  *q = 2;
  printf("%d %d\n", *p, *q);
}

Unfortunately that doesn't change anything: you still get the same “1 2” answer from the usual culprits.

> ...except if the second malloc returns the same "space" as the first malloc, is 'p' still indeterminate now that the space it was referring to is no longer free? Maybe the 'p == q' is okay (though the '*p' is still undefined because it lost its ability to access the object after the space was first freed).

Actually it's the other way around: p == q is not Ok (but you can use memcmp instead), while *p is fine (if you used memcmp). It wouldn't be fine only if you introduce “pointers provenance”. Which is not part of any existing C and/or C++ standard and most definitely not part of C89 (as I have noted elsewhere C++ have optional feature similar to “pointers provenance”, but only in “strict pointer safety” mode which none of the existing compilers support). Otherwise the only way for two different pointers to be equal is to have them point to the same object or, as special corner-case, to have one of them point to one-past-the-end-of-an-array (and there are no arrays in this example).

C committee was actually asked about more-or-less that issue (what should happen when one reads value of a pointer passed to free) and the result: after much discussion, the UK C Panel came to a number of conclusions as to what it would be desirable for the Standard to mean. Note: they haven't said that standard makes it invalid now, no. They said that the fact that this program is well-defined is a problem and standard needs to be changed. And make currently valid programs invalid.

> I don't think it's fair to blame C99 for changing the semantics here - C89 seems very unclear and ambiguous, and C99 was the first time it was actually specified semi-properly (by having a more precise definition of the lifetime of an object with the term "deallocated" (instead of "freed or reallocated"), and saying realloc() always "deallocates" the old object).

I wouldn't call that “semi-properly”. The history here is the following: when C89 was developed there was already a strong push from compiler writers about the need to ~~screw the programmer~~ enable optimizations. And nice (for a compiler writers) scheme was added to the standard draft. Unfortunately it turned up almost impossible to use it for writing real program (as Dennis Ritchie noticed). And language which you couldn't use to actually write programs in is not very useful. Thus C committee ripped out the most egregious parts and only left some small remnants of that attempt in C89.

In C99, C++11 and all subsequent standards these attempts were repeated. It's unclear how many useful optimizations these attempts enabled, but they made it almost impossible to write a correct programs in C/C++. Sooner or later you end up with some kind of security check which compiler would happily remove to “optimize” your code.

This quest is not yet finished, we still have no adequate set of rules there (apparently transformations which are derived from these rules and used in LLVM can easily turn correct program into incorrect one and that doesn't usually happen simply because these transformation are applied in certain order… that would have been really hilarious it weren't so sad), yet programmers were supposed to write programs, back in 1990, which adhere to rules which would be finalized, maybe (there is hope, but no promises right now) around 2030. Does that sound realistic to you?

Quotes of the week

mpr22 — Thu, 03 Jun 2021 17:34:30 +0000

Does the standard require that

void * myptr = NULL; intptr_t myintptr = *(intptr_t *) &myptr; printf("%ld", (long) myintptr);

prints "0", or just that

void *myptr = NULL; if(myptr == 0) { printf("myptr is equal to zero.\n"); }

prints "myptr is equal to zero"?

(Alternatively: Does the standard require that a pointer set to NULL contains the same bit pattern as an intptr_t set to 0, or just that if you compare a pointer set to NULL to an intptr_t set to zero, they are equal?)

Quotes of the week

excors — Thu, 03 Jun 2021 17:09:39 +0000

> My favorite example: dereferencing a NULL pointer is undefined behaviour. It can not be described in any other way since there exist many microcontrollers without an MMU where address 0 is mapped to some memory which may contain a data structure and the behaviour of overwriting some unknown state can not be defined.

Somewhat related to that, I once found a vulnerability in some microcontroller bootloader code. (Well, I found several, but only one is relevant). It tried to compute the hash of the application image that was stored in flash, then verify the signature of that hash, before booting the application. The hashing function returned a pointer to a buffer containing the computed hash - but on error (e.g. invalid fields in a header structure) it returned NULL. The rest of the code didn't bother checking for NULL, so it happily verified the signature over the 'hash' stored at location 0x00000000 (which was simply the start of flash), then booted the application. An attacker could extract the hash and signature from a valid signed application, write the hash to 0x00000000, copy the signature, and then write an arbitrary unsigned application image with an invalid header (to trigger the NULL return) and the bootloader would think it's valid and boot it.

Null pointers are particularly dangerous on systems like that.

On the other hand, sometimes those microcontrollers *want* to access the data stored at location 0x0. I'm not sure that's technically allowed by C - if I'm reading it right, null pointers are required to be distinct from any pointer to an actual object, but also they're required to convert to 0, which implies you can't have an object in memory at 0x0. That's not ideal when your hardware is storing useful information there but you can't trust your C compiler to let you read it.

Quotes of the week

excors — Thu, 03 Jun 2021 16:30:04 +0000

Would you expect

int *p = malloc(sizeof(int));
free(p);
int *q = malloc(sizeof(int));
if (p == q) {
  *p = 1;
  *q = 2;
  printf("%d %d\n", *p, *q);
}

to have well-defined behaviour?

I'd expect no - 'p' and 'q' are intuitively pointers to different objects, regardless of whether they happen to be numerically equal, and one of those objects is being accessed after it was freed.

C89 (or at least a version I can find online) says "The pointer returned [by calloc/malloc/realloc] [...] may be assigned to a pointer to any type of object and then used to access such an object in the space allocated (until the space is explicitly freed or reallocated). [...] The value of a pointer that refers to freed space is indeterminate". So I think that largely matches my expectation - the value of 'p' is indeterminate after the 'free', so the 'p == q' is undefined behaviour, and the '*p' is undefined behaviour (because the space has been explicitly freed so 'p' can't be used to access the object any more).

...except if the second malloc returns the same "space" as the first malloc, is 'p' still indeterminate now that the space it was referring to is no longer free? Maybe the 'p == q' is okay (though the '*p' is still undefined because it lost its ability to access the object after the space was first freed).

Anyway, when you replace the free+malloc with realloc, it sounds like you expect that to be well-defined behaviour in C89?

I think the problem is ambiguity with terms like "object" and "space" and "reallocated". C89 indicates realloc() returns the same object with a new size (though in other places it seems to directly contradict that and says it's a new object). But can that be the same object in the same space as the original malloc(), or different space, or logically different space but the pointers might happen to be equal? If you call realloc() with the object's current size, has it been reallocated for the purposes of "until the space is explicitly freed or reallocated"?

I don't think it's fair to blame C99 for changing the semantics here - C89 seems very unclear and ambiguous, and C99 was the first time it was actually specified semi-properly (by having a more precise definition of the lifetime of an object with the term "deallocated" (instead of "freed or reallocated"), and saying realloc() always "deallocates" the old object).

Quotes of the week

Vipketsh — Thu, 03 Jun 2021 16:08:34 +0000

To take your print_digit(int i) analogy further, if all implementations print, say, "?" on i>=10 and tons of code starts to rely on that, the de-facto standard is that print_digit(int i) prints "?" for i>=10. It doesn't matter at all what some document says. If the implementation is then changed to print "HelloWorld!" in the case of i>=10 all the users who's code now breaks are rightfully annoyed. No amount of lawyering about standards, documentation and such matters. The situation is not much different to C and undefined behaviours. As an example: for a long time the 2s complement wrapping of signed integers was the norm for compilers and code relied on this until at some point they started to lawyer about the standard and undefined behaviour.

This isn't a new situation either. It has happened countless times in the past and is occurring today: The linux user interface takes this stance (relied on behaviour is de-facto standard) and the same thing happened (to an extent) with the glibc memcpy and aliasing pointers.

I don't agree with khim's stance that the only correct way forward is to go and explicitly change the standard. If the predominant implementations become sane instead of doing all the stupid things supposedly allowed by the standard, the standards body will have two options: either become irrelevant and let others define what C is or to throw out their garbage and document the de-facto behaviour, as has happened with HTML for example. It also works the other way too: if the predominant implementations implement crazy things while the standard becomes sane, it doesn't actually help anyone.

This whole dancing around undefined behaviour started because compiler writers forgot the reasons behind C being the way it is and started to ignore de-facto user expectations. Instead the compiler authors decided that if a program is not in 100% conformance with the strictest reading of the standard any insane output is valid. Yodaiken's articles explain this pretty well.

My favorite example: dereferencing a NULL pointer is undefined behaviour. It can not be described in any other way since there exist many microcontrollers without an MMU where address 0 is mapped to some memory which may contain a data structure and the behaviour of overwriting some unknown state can not be defined. Henceforth any definition for dereferencing a NULL pointer would require a compiler to insert if(<is null pointer>) do_something() before every single load and store on these machines. Clearly this is not reasonable. On the other hand, on anything with an MMU, the user expectation is that a NULL pointer dereference turns into a SIGSEGV. Instead of understanding this reason compiler writers decided that undefined behaviour in this case means that the compiler can do whatever it wants to the program after the dereference.

Quotes of the week

rschroev — Thu, 03 Jun 2021 15:13:59 +0000

OK, thanks, that makes it clear what you're talking about. I agree this is bad.

And now I see that you made that clarification already elsewhere in the thread. Sorry, my bad.

Quotes of the week

khim — Thu, 03 Jun 2021 15:00:05 +0000

After reading comment from others I realized that I haven't actually explained what's the issue with that code.

The issue here is not that realloc may return different pointer and then program prints nothing. Sure, it's allowed to do that according to C89, K&R or any other standard.

No, the issues here is that existing compilers make it print 1 2 instead of 2 2 — and that may never happen according to C89.

Quotes of the week

khim — Thu, 03 Jun 2021 14:39:42 +0000

> I'm asking because I'm not sure how to interpret 'the only possible output is "2 2"'.

Ah, sorry. I actually forgot about the fact that realloc can, according to the standard, actually return a different address here. In practice all existing implementations return the same one.

Yes, correctly compiled program may return empty output here if, e.g. you realloc just always calls malloc and copies the content. That's not an issue.

The issue is: actual output (that is: 1 2) is clearly invalid.

> I agree, the program looks legal to me and should output either nothing or "2 2\n". Are you saying that behavior is under threat somehow?

Clang, MSVC and ICC all produce "1 2\n" output and they all claim that they can do that because they plan to add new set of undefined hehaviors to the C2x standard.

Once again: they miscompile perfectly valid C89 program in a strict C89 mode today and claim that it's fine because it, apparently, violates set of rules which they plan to add to C2x (and which is not yet even finalized).

And the explicitly refuse to provide any flags which can make it work (although -fno-builtin-realloc works, but it's extremely unituitive and non-obvious

Quotes of the week

rschroev — Thu, 03 Jun 2021 14:16:20 +0000

> Note that C89 quite explicitly allows that code and says the only possible output is “2 2”. This is because realloc there does the following: The realloc function changes the size of the object pointed to by ptr to the size specified by size. It's still the same object, both pointers point to it, so why would they behave differently?

Do you mean that the example program should always output "2 2\n"? Or do you mean that it should either generate output or not, and if it does output something it should output "2 2\n"? I'm asking because I'm not sure how to interpret 'the only possible output is "2 2"'.

In the first case:

I think you're wrong: C89 doesn't guarantee that p == q. The realloc function can legally move the object, even in C89. Look what it says about realloc's return value (at least in the draft of the standard -- I don't have access to the officially released version):

> The realloc function returns either a null pointer or a pointer to the possibly moved allocated space.

In the second case:

I agree, the program looks legal to me and should output either nothing or "2 2\n". Are you saying that behavior is under threat somehow?

Quotes of the week

khim — Thu, 03 Jun 2021 13:43:45 +0000

According to all existing C standards it's well-defined program. If you don't subscribe to the ridiculous notion that somehow realloc turns existing pointer to existing object (which is not even part of any array) pointer one-past-the-end of array (specifically invented to ~~screw the programmer~~ optimize code better). In fact in that same provenance proposal this is noted quite explicitly (page 17, “pointer equality comparison and provenance” where they talk about how that part should be changed to make existing programs which are fully standard-compliant invalid — all to make language “better”, of course).

Situation with C++ is a bit more complicated. C++ have the notion of “pointer safety” which could have invalidated that program on some compilers… except all existing compilers use relaxed pointer safety. So that part couldn't justify what they do.

If you want to say that program is not guaranteed to print "2 2" then you are absolutely correct: empty output it's also a valid possibility (if realloc would actually move object). But this program, when compiled with a compiler which correctly implements existing standards couldn't print "1 2" — which it does on most compilers (clang/msvc/icc, only gcc produces correct result). And problem here is not with the fact that compilers are buggy (all software may have bugs) but with the fact that prevailing attitude here is “we screwed you up?… and we haven't yet added enough “undefined behaviors” to the standard to justify miscompilation of a perfectly valid program?… oh, that's too bad, let us add more “undefined behaviors” to the standard and apply them retroactively to make it possible to ~~screw you up more~~ optimize better… and no, we wouldn't give you the flag to make it possible to correctly compile correct programs”.

Quotes of the week

HelloWorld — Thu, 03 Jun 2021 12:55:46 +0000

Why do you say the rules were retroactively changed? I don't think there ever was a C standard that defined the behaviour of that program.

Quotes of the week

khim — Wed, 02 Jun 2021 20:39:47 +0000

> But anything that relies on your interpretation is inherently broken ...

How and why? from the same C89 standard: the realloc function returns either a null pointer or a pointer to the possibly moved allocated space.

Yes, object can be moved by realloc and, as you have correctly noted sometimes it have to be moved, sure. But we have established that it haven't happened in our program with a simple check if (p == q). If that's the same object and if it wasn't moved — what makes it possible to treat pointer to it as “invalid”?

> Either your definition of realloc is correct and it has to return a failure (q == null), or it has to allocate a larger amount of space elsewhere and move the contents.

Of course realloc can move the content. But all standards quite explicitly say that object can be left in place. Even C18 says the following: the realloc function returns a pointer to the new object (which may have the same value as a pointer to the old object), or a null pointer if the new object has not been allocated.

This note (it's part of standard and was there since the invention in UNIX, although phrasing changed over time) just begs one to special-case that situation, right? But nope: apparently “provenance rules” (which are, once again, not part of C99, not part of C11, not part of C18, not part of any existing C++ standard and, apparently, are not yet even finalized) give the compiler the right to ~~screw the programmer~~ optimize that code.

One can, of course, play weasel-words with phrase “the same value as a pointer to the old object” because standard defines “the same value” in the following way: Two pointers compare equal if and only if both are null pointers, both are pointers to the same object (including a pointer to an object and a subobject at its beginning) or function, both are pointers to one past the last element of the same array object, or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space.

And yes, I have heard from one clang-developers that, indeed, after call to realloc old pointer becomes pointer to one past the end of zero-sized array object! And using it is thus “undefined behavior”.

I couldn't even properly answer anything to such reading of the standard! Since I'm not Linus and I don't know enough English swear words.

I guess at some point these rules would be finalized and would become, retroactively, part of C89/C99/C11/C18/C++98/C++11/C++14/C++17/C++20… but I can not see how someone can write any code in a language whose rules can be retroactively changed half-century after they were written.

Quotes of the week

Wol — Wed, 02 Jun 2021 19:17:39 +0000

> C99 changed that. Now realloc works differently: The realloc function deallocates the old object pointed to by ptr and returns a pointer to a new object that has the size specified by size.

But anything that relies on your interpretation is inherently broken ...

int *p = (int*)malloc(sizeof(int));
int *q = (int*)realloc(p, 16 * sizeof(int));

If malloc has only allocated a block 64 bytes in size for p and all the metadata it needs to manage it, it is just not possible to resize it such that q == p. Either your definition of realloc is correct and it has to return a failure (q == null), or it has to allocate a larger amount of space elsewhere and move the contents.

So regardless of whether it's correct, using your interpretation, and using realloc to grow the allocated space, is a pretty stupid idea if you assume it's "just going to work". Most people assume that malloc/realloc won't return a failure. Under your interpretation, it would be a common event.

Cheers,
Wol

Quotes of the week

khim — Wed, 02 Jun 2021 13:59:04 +0000

> Well, to turn something that is "unhandled" into "handled", I assume you need some sort of specification how to handle it.

What about the case where you turn something that was “handled” into “unhandled”? Let's consider concrete example. This code:

#include <stdio.h>
#include <stdlib.h>

int main() {
    int *p = (int*)malloc(sizeof(int));
    int *q = (int*)realloc(p, sizeof(int));
    if (p == q) {
        *p = 1;
        *q = 2;
        printf("%d %d\n", *p, *q);
    }
}

Note that C89 quite explicitly allows that code and says the only possible output is “2 2”. This is because realloc there does the following: The realloc function changes the size of the object pointed to by ptr to the size specified by size. It's still the same object, both pointers point to it, so why would they behave differently?

C99 changed that. Now realloc works differently: The realloc function deallocates the old object pointed to by ptr and returns a pointer to a new object that has the size specified by size.

Why would that be important? We compared pointers, thus they should behave identically, right? No. There is a decision of WG14 committee which says literally the following: after much discussion, the UK C Panel came to a number of conclusions as to what it would be desirable for the Standard to mean — and then short explanation of how standard should be changed to make that program illegal.

Note: they haven't said that standard actually means that today. Nope. Provenance insanity is not yet part of any standard. Not C99, not C18 and not even C++20! Yet compiler writers think they are entitled to apply these rules (which are, apparently, area of research because compiler writers still couldn't invent usable set of rules which you can use to write correct programs) to old, C89 programs.

Nice, huh?

> Well, a C compiler basically implements C.

Except today it's not true. C compiler writers implement basically whatever they want to implement and reserve the right to retroactively change rules of language. Without providing options which may bring back old behavior (-fno-builtin-realloc works today, but apparently there are no guarantee that it would work in the future).

> That would be an option, but I assume the smoothest path forward is to continue proposing "C language extensions/options", like -fno-strict-aliasing and -fno-strict-overflow to GCC, for specific instances of undefined ("unhandled") behavior in the C language. If considered useful enough to be the default option for the whole C community, it can then be brought up to the C committee.

I think at this point it's, basically, pointless. When I explicitly asked some clang developers about something like -fno-provenance option the answer was: provenance is something LLVM *violently* believes in, at the level of alloca, malloc, and similar intrinsics scribbling provenance information all over LLVM's internal representions. Even if you could turn it off, I doubt it would fix all of your miscompilations, since this is a fundamental building block of LLVM's IR. Like I said before: although provenance is not defined by either standard, it is a real and valid emergent property that every compiler vendor ever agrees on.

Note that not defined by either standard yet real and valid emergent property part. I think after answer like that… it's, essentially, pointless, to bring anything to a C committee. What's the point if said committee would pick something they like, not something that makes, you know, possible to write anything in that language?

We are more-or-less stuck with GCC extensions for the foreseeable future and I think it's good idea to adopt Linus stance. Essentially: “I couldn't forbid you to use clang but I don't consider “clang miscompiles that code” a valid reasoning for any change in any project”.

This is unfortunate because the only language which tries to address these issues in practically usable way, Rust, is basically, tied to LLVM currently. gccrs looks quite active novadays, though, thus there's hope. But C and C++… they should be declared “unfit for any purpose”, sadly. Certain specific implementations probably can be used, maybe, but there are zero hope of getting sane cross-compiler treatment. That ship have sailed.

Quotes of the week

patha — Mon, 31 May 2021 15:36:15 +0000

> but it lies in a different place than you think. You can read that article to understand Yodaiken's position better.

Sorry, I think I wasn't able to understand much more by reading this article.

> He doesn't argue about definition of “undefined behavior” in standard as much as he laments the fact that compilers today tend to interpret undefined behavior as, well, undefined behavior. Always. Unconditionally. To be avoided. Unconditionally.

Well, to turn something that is "unhandled" into "handled", I assume you need some sort of specification how to handle it. This also needs to be specified separaterly for each instance (the devil is in the details).

> And that is the gist of the problem: compiler writers insist that C standard should be the only thing that defines semantic of the program.

Well, a C compiler basically implements C. If you want it to implement something else, you develop an optional change to the language, like for example the -fno-strict-aliasing and -fno-strict-overflow options in GCC.

> The way to reconcile these camps is not to complain and whine — but to change the standard.

That would be an option, but I assume the smoothest path forward is to continue proposing "C language extensions/options", like -fno-strict-aliasing and -fno-strict-overflow to GCC, for specific instances of undefined ("unhandled") behavior in the C language. If considered useful enough to be the default option for the whole C community, it can then be brought up to the C committee.

Quotes of the week

khim — Mon, 31 May 2021 14:12:40 +0000

> I assume the main issue here is a misunderstanding of what undefined behavior actually is.

Indeed - but it lies in a different place than you think. You can read that article to understand Yodaiken's position better.

He doesn't argue about definition of “undefined behavior” in standard as much as he laments the fact that compilers today tend to interpret undefined behavior as, well, undefined behavior. Always. Unconditionally. To be avoided. Unconditionally. And that drives him (and people like him… including Linus, unfortunatelly) nuts.

> In our case with the 'print_digit' function, we may remove the undefined behavior by for example change it to:

// Print a digit to stdout. For input outside the range 0 to 9, modulo 10 arithmetic is applied. void print_digit(int i)

Yup. You can do that. In fact people have done that: they defined SIGSEGV signal and made sure it's triggered when you try to access NULL pointer. Behavior have become defined now, right? Nope: compiler doesn't think so.

And that is the gist of the problem: compiler writers insist that C standard should be the only thing that defines semantic of the program. Other guys (Yodaiken… Linus… and lots of other people who, coincidentally, don't write compilers) argue that it's completely unrealistic idea.

So we have two positions:

C was explicitly designed to omit certain pieces thus they are declared as “undefined behavior” in standard — yet they are defined by execution environment thus we may expected “constructive interpretation” from the compiler.
If something is important enough for the compiler to actually care — it should be defined in the C standard itself. It's already hard to write compiler which can work with dozen of hardware platforms and two standards (each hundreds of pages long). Adding unknown (and not limited!) number of things to that list is just not realistic.

Guess which position wins when people with #1 stance don't write the compilers and people with #2 actually do?

The way to reconcile these camps is not to complain and whine — but to change the standard.

When 2-3 guys were developing a compiler — it made some sense not to include everything into definition of language. But today… when literally thousands of people are involved in developing these standards and compilers… If something is important enough for the compiler to treat it as anything but “undefined behavior” then said something is important enough to be explicitly written in the standard. End of discussion.

Apparently Yodaiken actually tried to change the standard — but not with proposals to turn some “undefined behaviors” into other kinds (unspecified, implementation-defined, simply well-defined) but with attempt to push “constructive interpretation”. No wonder committee was not impressed. C (and C++… and CPUs… and OSes…) have just become too complicated for #1 stance to make any sense.

Quotes of the week

patha — Mon, 31 May 2021 08:26:03 +0000

I assume the main issue here is a misunderstanding of what undefined behavior actually is. Basically, you may view it as "unhandled" cases in the C semantics. The blog post seems to assume that compiler writers currently actively need to figure out a semantics for undefined behavior, and a "constructive interpretation" is to just slightly change the definition of how compiler writers should figure out the semantics. I think this is a fundamental misunderstanding. Undefined cases are basically unhandled, rather than interpreted in any way. Then, going from "unhandled" to "handled" is a fundamental change, rather than a minor change in interpretation.

The notion of undefined behavior should be interpreted in the context of section 5.1.2.3, Program execution, of the standard (excerpt):

The semantic descriptions in this document describe the behavior of an abstract machine in which issues of optimization are irrelevant.
In the abstract machine, all expressions are evaluated as specified by the semantics. An actual implementation need not evaluate part of an expression if it can deduce that its value is not used and that no needed side effects are produced (including any caused by calling a function or through volatile access to an object).
The least requirements on a conforming implementation are:

Volatile accesses to objects are evaluated strictly according to the rules of the abstract machine.
At program termination, all data written into files shall be identical to the result that execution of the program according to the abstract semantics would have produced.
The input and output dynamics of interactive devices shall take place as specified in 7.21.3. The intent of these requirements is that unbuffered or line-buffered output appear as soon as possible, to ensure that prompting messages actually appear prior to a program waiting for input.

This is the observable behavior of the program.

If some C construct lack a "semantic description" – that is, is "unhandled" by the standard (but the C standard is actually currently trying to explicitly point out cases of undefined behavior) – also C language implementations (e.g. a compiler + hardware realization) can leave these C constructs "unhandled". According to my view, this is basically the origination of "undefined behavior".

Let's take a simple example (outside the domain of the C standard and compiler implementations) of a simple C function:

// Print a digit to stdout. Only 0 to 9 has defined behavior.
void print_digit(int i)

The comment documenting the function implies that values for i outside of the range 0..9 may be left unhandled by the implementation. That is, if 10 is passed as an argument, the behavior is undefined, and the function is allowed to for example crash or print "Hello world!". (But in practice the last implementation is unlikely.) This also implies that the actual behavior for out of range cases may change, when a new version of the implementation is released. The actual behavior is uninteresting from the implementors point of view and may even be unknown (requires a non-trivial amount of reverse engineering to infer).

The Linux kernel turn on some GCC options, such as -fno-strict-aliasing and -fno-strict-overflow to actually change the C semantics into something else for some C constructs. (In my opinion these options should have been listed at Options Controlling C Dialect rather than Options That Control Optimization and Options for Code Generation Conventions in the GCC documentation.) Similarly, we can also turn other cases of undefined behavior into defined behavior, by actually changing the C semantics. However, this needs to be done case-by-case, rather than trying to somehow turn all instances of undefined behavior into defined behavior, with some kind of default interpretation of undefined behavior (which I would say is impossible).

In our case with the 'print_digit' function, we may remove the undefined behavior by for example change it to:

// Print a digit to stdout. For input outside the range 0 to 9, modulo 10 arithmetic is applied.
void print_digit(int i)

But other changes are also possible. For example:

// Print a digit to stdout. For input outside the range 0 to 9, '?' is printed.
void print_digit(int i)

Both for our simple 'print_digit' function and the C standard or C compilers, the actual benefits of having something as "undefined behavior" are basically the same:

It may simplify the implementation.
It may make the execution more efficient.

The drawback is probably also basically the same:

It may make the usage more risky (the program may crash or behave more unpredictably).

Quotes of the week

khim — Fri, 28 May 2021 15:34:36 +0000

> It would be way more helpful if they just introduced (for example) a statement "these two pointers are never the same, it's undefined if they somehow are" or "this pointer does not point to this other variable".

It's doable, it was actually done and it works very-very well. The problem? Th resulting language is called Rust and it's not C-compatible in any way, shape, or form.

And that is fundamental. Because once you start going down that path you quickly discover that you need to somehow tell the compiler about lifetimes of different variables — and like const these annotations become viral: for them to help they need to be pervasive and used in the whole program.

But while that helps, it's completely tangential issue. The core issue with “undefined behavior” is fundamental misunderstanding. Look again on the blog post where it explained how “shift example” is “constructively interpreted” with three possible “interpretations” and try to explain how that “constructively interpretation” of “undefined behavior” would differ from unspecified behavior. I dare you.

The short answer: it wouldn't differ at all — yet all these articles which cry about bad-bad-bad compilers ignore that fact completely.

Undefined behavior exist for a reason and as, you have correctly noticed, it was supposed to cover cases which no sane person would try to ever used in their program — but some people (who know “too much” about underlying hardware, usually) like to pretend that they know the full range of possible outcomes (like with unspecified behavior) — and become extremely angry when compiler does something else.

The proper solution is reclassification. Certain behaviors should be moved from “undefined behavior” bucket into “unspecified behavior” bucket, “implementation-defined behavior” bucket or may be just defined outright.

But this is hard. Complaining about lack of common sense in a compiler is easier. Well… newsflash: compilers with common sense are not coming and if you have no formal definition of how your program should be interpreted then chances of it being interpreted in a correct way by a compiler which does literally hundreds of passes is close to zero.

Quotes of the week

iabervon — Thu, 27 May 2021 21:29:39 +0000

I think it comes down to not being willing to introduce a way to specifically tell the compiler statements that would help it optimize, and instead finding ways to notice that code wouldn't necessarily be portable if those statements weren't true. Early on, that let compilers get a lot better at compiling existing code, which didn't have any annotations and where cases where the optimization was not valid had already caused problems and been fixed. But they later got to cases where violating the assumption didn't actually cause problems, so existing code has to be changed to still compile correctly. And they miss optimizing cases where the programmer knows something is true, but hasn't written the program in such a way that it being false would cause undefined behavior.

It would be way more helpful if they just introduced (for example) a statement "these two pointers are never the same, it's undefined if they somehow are" or "this pointer does not point to this other variable".

Quotes of the week

khim — Thu, 27 May 2021 20:21:56 +0000

The people who fight against “evil compiler writers” miss the point because they just don't think about how modern compilers work (actually not-so-modern, it was a problem quarter-century ago already).

The change in C99 wasn't arbitrary and, in fact, it was unavoidable. The reason is simple: phrase about “permissible undefined behavior” only makes sense for a very simple, basically trivial compiler. Which does not optimizations at all. Once you start doing optimizations you quickly find out that before you can argue if certain optimizations are valid or not and when… you need to know what “expected behavior” is. Because all optimizations, basically, depend on as if rule — but how can you apply it if you have no idea what happens in code at all? For many types of undefined behavior it's definitely not trivial… think an invalid array reference, null pointer reference. or reference to an object declared with automatic storage duration in a terminated block occurs: what kind of optimizations can you offer which would keep observable behavior the same for such a program?

Consider the following program:

#include <stdio.h>

int foo() {
  int i;
  i = 42;
}

int bar() {
  int i;
  return i;
}

int main() {
  foo();
  printf("%d\n", bar());
}

It works in GCC -O0. GCC -O2 breaks it but not in a way permitted by vyodaken's reading of C89. And Clang breaks it completely - with -O0, too yet differently with -O2.

Worse: when people start talking about examples of undefined behavior which happen in normal program they often miss the point entirely.

> Returning a pointer to indeterminate value data, surely a “use”, is not undefined behavior because the standard mandates that malloc will do that. Use of “asm” does not cause undefined behavior although it is nonportable because the standard (glancingly) mentions using asm.

See? Standard writers are idiots since noone can write a reasonable program without triggering undefined behavior! Except… returning a pointer to indeterminate value data is not use and it is allowed. It's not an undefined behavior. And asm is very explicitly permitted, it's not an “undefined behavior” either. And I'm yet to see anyone who will complain that compilers can do crazy things if something like this is violated: the format for the fprintf or fscanf function does not match the argument list.

The real problem is that when C89 was made lots of things were put in the “undefined behavior” bucket where they should have been put into “implementation-defined behavior” bucket. When C99 was made it become obvious that wording for “undefined behavior” just doesn't work - yet lots of stuff which should have been moved to “implementation-defined behavior” at this point was kept as “undefined behavior”.

But instead of trying to fix that, very real problem (and yes, when properly discussed such conversion does happen: read P0145 or P0593 for example), people attack compiler writers again and again and again.

As if that may ever change anything.

P.S. The mere fact that linux kernel or libsodium even have these switches shows that compiler writers do understand that lists of “undefined behavior”s in standard are not perfect. But demanding that compilers should stop treating “undefined behavior”s as, well… undefined and start and treating all of them as “implementation-defined” ones simply because someone doesn't have know standard well-enough… it just wouldn't fly, sorry.

Quotes of the week

smoogen — Thu, 27 May 2021 12:15:02 +0000

When I worked at an HPC center, I found that much of the people who spent time on these sorts of committees were fighting this 'war'. There was always some sort of 'why do we even have people who program in X, it can't do Y' because funding was always tight and well it is easier to say the lowlevel department should be axed since all our code is in highlevel.

I don't remember which of my 'elders' said it but it has stuck with me since: "It's like watching a playground fight in a toolbox where the screwdrivers say that no one needs hammers or wiresnips because you do that with a screwdriver."

Quotes of the week

ballombe — Thu, 27 May 2021 11:51:02 +0000

> It matters because over time the Standard and the common compilers have made C an unsuitable language for developing a range of applications, from memory allocators, to cryptography applications, to threading libraries and, especially operating systems.

Fully Agreed.

The C committee seems stuck optimizing 1970-era HPC benchmarks to get C to compete with FORTRAN (and failing) instead of recognizing what are the real use of C today: why a program is written in C rather than in a memory
managed language ? Usually because it is (in part) a memory manager, but the standard is still written in the spirit that all memory is always allocated with malloc. Apparently they did not hear about mmap. They even decided that the only way to copy data in a non aliasing way is byte-by-byte.

Instead of adding explicit keyword to specify a type behavior (wrapping/nonwrapping, aliasing/nonaliasing etc.)
they force arbitrary behaviour that are unpractical in most situation, break backward compatibility and pretending previous behaviour was undefined (which means: it is possible to write a compliant compiler that breaks this, even if no such compiler have ever been published).

And they do not address real issues than need update like the interface with the linker.