Footguns

Do you really hope anyone would treat your rants seriously when you don't do even a simple fact-checking?

How is it “trustable” when it happily removes your beloved security checks? Heck, even GCC 4.1 (oldest available on godbolt) does that. And if you would look on the appropriate bug you'll know GCC 2.95 did that as well!

And it was released last century when Rust wasn't even a thing! Graydon Hoare may have played with some ideas which eventually lead to creation of Rust back then, but it wasn't even presented to Mozilla employees, let alone general public.

In fact in the beginning Rust wasn't even about replacement of C or C++ (it still isn't… some Rust fanboys are trying to portray Rust as “modern C/C++ replacement” but it's developers are not aiming that high).

It was about moving low-level programming field from 70th to 90th (yes, Rust includes zero ideas from XXI century, all things which you see in it were “boring and old” when it was conceived…). Some of these ideas were quite “modern and unproven” at the end of XX century but today they are “old and boring”.

If you “don't know” then why do you write all these rants which one step away from “reptiolds are using Rust to destroy human civilization!” ?

Yes, compilers have become more and more clever at exploiting what they consider undefined behavior. But this “fall from grace” was very slow and gradual. And it wasn't caused by a switch to C++ or creation of clang, let alone Rust.

To claim that all that was done by some hidden beneficiary which did all that to, somehow, push modern language into system developer's community and bring them, kicking and screaming, to XXI century… that's new low, I think.

Sure, at first the breakage could be contained with the use of some simple flags… but as time marched on… it became harder and harder to do.

And these flags were introduced as a concession to developers, not as something compiler developer promised to always deliver.

The new thing that happened recently is situation where complexity of compiler optimizations have grown enough that not only offering such flags have become more-or-less impossible, but where they couldn't even compile correct programs reliably — and instead of thinking about how they can change the compiler to bring it back to standards-compliance they started thinking about way to move the goalposts in the standard text!

That is when C and C++ have become “languages unfit for any purpose” and Rust have started gaining popularity.

Yet execution time for benchmarks decreased which was the goal. The fact that kernel community was burned in the process is just a side effect.

#include <stdio.h>

void foo(int* i) {
  i[1] = 42;
}

int bar() {
  int i = 3, j = 3, k = 3;
  foo(&i);
  return j + k;
}

int main() {
  printf("%d\n", bar());
}

#include <stdio.h>

int main() {
  const int this_is_zero = 0;
  *(int*)(&this_is_zero) = 42;
  printf("Who said this is zero: %d\n", this_is_zero);
  return 0;
}

#include <stdio.h>

int foo() {
  int i;
  i = 42;
}

int bar() {
  int i;
  return i;
}

int main() {
  foo();
  printf("%d\n", bar());
}

void foo(int* i) {
  i[1] = 42;
}

int bar() {
  int i = 0, j = 0;
  foo(&i);
  return j;
}

int main() {
  printf("%d\n", bar());
}

void foo(int *i, int o) {
        i[o] = 42;
}
int bar() {
        int i = 0, j = 0;
        foo(&i, &j - &i);
        return j;
}
int main() {
        printf("%d\n", bar());
}

struct useful {
    int value;
    /* And a lot more bits */
};

void ub(void * forty) {
    struct useful * ptr = (useful*) forty;
    int value = ptr-> value;
    if (!forty) {
        return;
    }
    /* Do stuff with value */
}

/* In a header file, for config handling */
struct configs {
/* Lots of pointers to config members here, omitted */
};

#define GET_STR_CONFIG_OR_DEFAULT(config, member, default) \
    (config ? config->member ? config->member : default : default)

#define GET_INT_CONFIG_OR_DEFAULT(config, member, default)
    (config ? config->member ; default)

/* In a C file, using the header file */
void do_amazing_things(void *config) {
    int rating = GET_INT_CONFIG_OR_DEFAULT(config, rating, 0);
    char * expertise = GET_STR_CONFIG_OR_DEFAULT(config, expertise, "novice");

    /* Do the amazing things */
}

struct configs {
/* Lots of pointers to config members here, omitted */
};
/* Rest of headers etc */

void do_amazing_things(void *config) {
    int rating = (config ? config->member : 0);
    char * expertise = (config ? config->member ? "expertise" : "expertise");
}

/* In a C file, using the header file */
void do_amazing_things(void *config) {
    int rating = ((struct configs*) config)-gt;rating;
    char * expertise = GET_STR_CONFIG_OR_DEFAULT(config, expertise, "novice");

    /* Do the amazing things */
}

struct S {
    int n;
    // ...
};

template <typename T, typename Pred>
void reset_if(T &items, Pred pred) {
    for (auto &item : items)
        if (pred(item))
            item = nullptr;
}

// Precondition: all items are not nullptr
// Postcondition: all items are even-numbered or are nullptr
void remove_odd(std::vector<std::unique_ptr<S>> &items)
{
    reset_if(items, [](auto &item) { return item->n % 2 != 0; });
}

But they don't disable warnings because of malice! They disable warning because “they know better”! And, indeed, the original plan explicitly called for compilers writers to augment the language by providing a definition of the officially undefined behavior. To make life of such developers easier.

Instead C and C++ developers have painted themselves into the corner: by repeating mantra “nothing except the standard text matter to us” they made said standard text sacred. Now, when they have found out that some optimizations they actually perform are unsound and don't obey the letter of said standard… attempts to change the standard are rightfully and correctly perceived as an act of sabotage.

Maybe if for last couple of decades compiler developers have actually listened to the users and not used said standard as fig leaf… then maybe, just maybe reaction would have been different. Even then: it's not guaranteed. Backward incompatible changes, especially not detectable at compile time is not something which users would welcome.

But at this point it looks as if the only sane decision would be to switch to some other language. Where developers actually listen to user's complains.

Rust is one such language.

It's done in the compiler sources. The same way as with memcpy. You can even disable it the exact same way, by specifying -fno-builtin-realloc. And then, of course, everything works.

That's MSVC. Don't mix them. Clang, at least, looks on the function name…

Indeed. But you would be naive you C compiler developers agreed. When I talked about this I was assured that it's absolutely not possible to do that. And if object is actually moved you can not traverse it and update insider pointers, too.

Instead you have to convert all insider pointers into indexes, store these indexes somewhere (but you can use unions to save space so could just copy then into uintptr_t which resides in the exact same place) and then turn indexes back into pointers (properly adjusted now).

And no, it's not a joke.

Irrelevant (according to compiler developers). Existing code must be fixed as described above. The fact that it may force it to become slower (and much slower in some cases) is irrelevant, again. Unless it's part of some well-known benchmark, of course.

C standard says something which is, as Linus would would say, is “total and utter crap”. On one hand it says this:

Two pointers compare equal if and only if both are null pointers, both are pointers to the same object (including a pointer to an object and a subobject at its beginning) or function,both are pointers to one past the last element of the same array object, or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space¹⁰⁹).

But if you notice there's a footnote ¹⁰⁹). It says this:

Two objects may be adjacent in memory because they are adjacent elements of a larger array oradjacent members of a structure with no padding between them, or because the implementation choseto place them so, even though they are unrelated. If prior invalid pointer operations (such as accesses outside array bounds) produced undefined behavior, subsequent comparisons also produce undefined behavior.

If you read it carefully you will realize that since accessing one-past-end-element is forbidden it also means that what the main body of the standard says is just not possible… but then why is it there?

The answer: C compiler developers got an approval from C standards committee to add what they call “pointer provenance” to the C standard but since in 15 years they still were unable to invent sane rules which people would be able to understand… they only managed to add that one tiny footnote.

And they were unable to put anything else to the standard because C committee (unlike C++ committee) actually cares about correctness and just doesn't want to add complete an utter garbage to the standard (you can read about why it's complete and utter garbage here). > The claim that comparing these two pointers can possibly be illegal makes no sense whatsoever IMHO, and I very much doubt that that was the intent of the C standards people. They might be misguided but they're not *that* stupid.

True. They resisted more-or-less succesfully with both C11 and C18. But C/C++ compiler developers are that stupid. This now they are trying again.

And what is their reasoning, pray tell? Why, of course: existing compilers already rely on the properties and thus standard have to be changed!

I would have been somewhat sympathetic if the exact same prompt by C/C++ users WRT other controversial parts of the standard would not have been meet with derision and prompts to burn the heretics rewrite all the programs.

No. It was neither consistent nor complete. That's why they weren't able to add pointer provenance neither to C11 nor to C18. The latest attempt is here. Please note the dates.

Nope. Not even close. Kinda-sorta coherent models were developed (here is one attempt) — but they don't match what the compilers are actually doing (the new model requires a handful of problematic IR-level optimizations to be removed, but it also supports the addition of new optimizations that were not previously legal is pretty telling, isn't it?) and, more importantly, it's not clear why they are supposed to be applicable to C89 or even C99 programs.

It haven't done that yet. Pointer provenance is not a thing — even in C18 version of C standard.

Good for whom? Compiler writers? Maybe. It's bad for C users, definitely, but since they couldn't present anything better they had only one option: accept the standard or live without a compiler.

But now it looks as if accepting standard is not enough. That is the problem.

When the story was presented as “Semi-portable C” vs “Standard C” this sounded like a sensible compromise: sure standard maybe harsh is some places, but hey, it's a standard!

Now, then we know that compiler developers don't feel themselves restricted by requirements of the standard the question arises: why then C users demands to change the standard are ignored? Hmm?

What kind of reality?

Yes. Compilers are broken. And now we even know there broken not as an accident. So?

Indeed. Instead of throwing away premature and unfinished ideas about how the so-called “pointer provenance” have to work they acquiesced to compiler writers demands (or, more likely, an ultimatum, but I wasn't there and couldn't say for sure). Yet instead of saying that standard is wrong they just “came to a number of conclusions as to what it would be desirable for the Standard to mean”. IOW: they proposed that compiler developer would start developing some sane set of rules for the C users to follow.

Instead of doing that compiler developers went on to invent unproven, buggy and half-specified schemes which were, as they claim, permitted by DR260. Indeed even latest proposal from last year explicitly says DR260 CR has never been incorporated in the standard text and lament about the sad fact that DR260 only explicitly permits track the origins of a bit-pattern and [...] may also treat pointers based on different origins as distinct even though they are bitwise identical

Which basically means: output from that realloc example is still buggy. Even if you would incorporate DR260 resolution into the standard — you still couldn't get output 1 2 in question. The most you can get is elimination of comparison and the whole code which does operations after that branch.

Now, you may try to argue that compiler was clever enough to notice that the only situation where p is equal to q and both are valid… nope, no cigar

Yes, but at least there some memory model was actually accepted and added to the text of standard. Nothing like that happened with DR260.

And I would argue that “pointer provenance” was a very bad idea and shouldn't brought to the realms of C as unconditional property. Previous proposal at least tried to acknowledged that and proposed two modes, test macros and so on.

But it was rejected. Most likely because it was “too much work” to implement.

So… it's too much work for the compiler writers to implement -fno-provenance switch yet reviewing and rewriting billions lines of code to make them compliant with new “optimizations” (which break code which was acceptable for decades) is not “too much work”? WTH?

Why is not not the same? On the contrary: it is the same. I have verified that.

And I'm talking about explicit mention about the possibility of in pointer being the same as out pointer. Why is it there? What can one do with that information? How can I use it?

I assume that standard writers weren't insane and haven't been adding some notes just to make standard bigger and more complex. So what's the reason to include that note as part of the interface?

But these are just wet dreams of C/C++ compiler developers. Right now standard doesn't allow for the possibility of two pointers being equal and comparable and yet them being different.

One concession which standard did in C11 was to declare some pointer comparisons as being UB (e.g. you are not supposed to compare one-past-the-end-pointer to “normal” pointer… that comparison is UB).

Except to claim this they would need to show at least one real-world example where it have improved performance. So far nothing was shown thus we may safely assume all that code was added specifically to break otherwise good programs and there are no other reason.

I think we are talking past each other. When I'm asking about “optimization” I'm not talking about realloc implementation.

I know when realloc can leave things untouched and why. That's not the interesting case.

The interesting case is an “optimization” enabled with -fbuiltin-realloc. The one which forbids user to compare p and q and says that you can't use pointers to the “old” realloced object. What does that one improve?

Precisely. And now I can't just do a simple q == p check to see if object is unmoved. I have to, basically, always do p = q assignment (if q is not nullptr) but what's more important: I can't keep pointers to “old” realloced object anywhere. If I have some other pointers (besides p) and, especially, if I have “internal” pointers (which point to the inside of that object) then I have to turn them all into indexes and then traverse everything after realloc again and turn these indexes back to pointers!

Now, I can easily imagine where such “compiler appease dance” can introduce slowdowns (even if realloc is clever and allocates exponentially for bigger buffers, etc).

But where the heck that dance imposed on me by C compiler developers helps? What kind of program? What benchmark? Can you show me anything at all where -fbuiltin-realloc is faster than -fno-builtin-realloc?

It has everything to do with the concept of provenance. What -fbuiltin-realloc option does is tells the compiler: “if someone does q = realloc(p, …) then you can assume that p and q have different provenance, they could never alias (not even after explicit p == q check) and you can optimize on basis of that”.

The one thing (and very big thing for me): WTH can you use that assertion for? What kind of code should we observe for that rule to speed up anything?

Casting itself is free. But because new “optimized” rules demand that you find and convert all pointers before you call realloc you would need additional code which would traverse additional data structures and scans all these pointers.

And after realloc you have to traverse all these data structures again and turn indexes back into pointers.

Not only that adds additional code which complicates logic, there are no guarantee that this code would be optimized away by the compiler (and in some cases, when external functions are involved, it couldn't be removed).

It's not the same. Version with -fbuiltin-realloc does this:

 mov esi,0x1
 mov edx,0x2
 xor eax,eax
 call 401030 <printf@plt>

Version with -fno-builtin-realloc does this:

 mov esi,0x2
 mov edx,0x2
 xor eax,eax
 call 401030 <printf@plt>

But lots of people claim it's “an optimization”. It's supposed to “improve” something. So… what, where and how does it improve?

If that's a paranoia then surely it would be easy for you to show something which benefits from -fbuiltin-realloc. Anything at all, I'm not that picky, if you don't have a real-world app I can even accept some contrived artificial example for the start.

struct Context { int initial; };

int *test(Context *ctx, int *p, int n) {
    int *q = (int *)realloc(p, n * sizeof(int));
    for (int i = 0; i < n; ++i)
        q[i] = ctx->initial + 1;
    return q;
}