Footguns
Footguns
Posted Jul 11, 2021 15:17 UTC (Sun) by khim (subscriber, #9252)In reply to: Footguns by smurf
Parent article: Rust for Linux redux
> Please use an example that at least has some chance of doing what you think it should be doing.
Hmm. How about something like this:
#include <stdio.h> int main() { const int this_is_zero = 0; *(int*)(&this_is_zero) = 42; printf("Who said this is zero: %d\n", this_is_zero); return 0; }
Better?
> It's buggy trash; calling it "intact" is an insult to any and all less-broken programs.Yet without proper guadance about what kind of “trash” you have to accept and execute correctly and what kind of “trash” you are allowed to turn into pile of goo you have to handle it. Somehow.
Both program are 100% syntactically correct, both work quite reliably on many platforms and many compilers (with optimizations disabled). Why do you can them “trash”? Just because you dislike them? Well, compiler developers dislike programs which rely on integer overflow, so what's the difference?
> It's been done, most recently by writing a Rust compiler instead.Except Rust compiler wouldn't even accept such program thus there are no need to think about what would it do when executed. But yeah, it can be done, sure.
> you can do it again.True. And then you would have another Rust-like language. Because if we are starting splitting the hairs and starting to put some valid programs into “this is trash, we don't plan to support that” bucket then why not change the rules and turn them just a compile-time errors instead?
When C was developed… then answer was obvious: “computers are slow and limited, they don't have resources to deal with that”. But what excuse would you have to do that today?
> But the reason for that isn't that UB-free C code can't be optimized.Of course not! UB-free C can be optimized! It's C code which triggers some UB where sh$t hits the fan. Please read what you argue against, please! Optimizations rely not on UB but on the absence of UB!
> The language simply doesn't have the features required to convey the necessary information.Precisely and exactly my point. In practice anyone who tries to invent that hypothetical -std=safe11
language end up adding such features to C and the end result becomes C#
or Go
, or Rust
.
What you don't get out of these attempts are -std=safe11
. Precisely and exactly because it's “too little gain for too much effort”.
Posted Jul 15, 2021 13:11 UTC (Thu)
by HelloWorld (guest, #56129)
[Link] (50 responses)
Posted Jul 15, 2021 14:37 UTC (Thu)
by khim (subscriber, #9252)
[Link] (27 responses)
But I “know” it's on stack and stack is not const! Because to me they look “sane”. The exact same way I “know” that if you overflow Why one thing would be supported, while other wouldn't be? Well… in C you couldn't eve call As you wanted: no casts, no And yes, I know, there are UB, too. But that's exactly the point: you can't optimize anything in C unless you would declare some subset of syntactically correct C “improper and not allowed”. And that is where all attempts to imagine The problem is: there are no C or C++ “community”. Just lots of people who don't really talk to each other. Even here: you don't even try to understand the point which these program illustrating. Instead you are attacking me as if I'm proposing to write code like that. Thus, as I have said: there would be no
Posted Jul 15, 2021 16:38 UTC (Thu)
by HelloWorld (guest, #56129)
[Link] (1 responses)
const isn't only there to set the relevant pages read-only, it also allows things like constant folding. So I think it's reasonable to make it UB when you try to write to a location that is marked const, whether or not it is static.
> Because to me they look “sane”.
Posted Jul 15, 2021 16:48 UTC (Thu)
by khim (subscriber, #9252)
[Link]
And that's how we end up with bazillion incompatible “friendly C” proposals which would never be implemented. Because none of them would even reach “written spec” stage… and you need it before an actual implementation stage.
Posted Jul 16, 2021 9:21 UTC (Fri)
by anselm (subscriber, #2796)
[Link] (24 responses)
That example may have no casts or const but it is still really crummy C code. I'm not sure exactly what this is supposed to illustrate, let alone prove.
Posted Jul 16, 2021 12:39 UTC (Fri)
by khim (subscriber, #9252)
[Link] (23 responses)
But it's syntactically valid and works on most C compilers with optimizations disabled, isn't it? The point was: that hypothetical proposed You couldn't just say anything syntactically valid and working in And if you start creating spec… and trying to keep it sane… you end up, eventually with a spec for D, C# or Rust… not with a spec for a “friendly C”
Posted Jul 16, 2021 16:12 UTC (Fri)
by HelloWorld (guest, #56129)
[Link] (6 responses)
> The point was: that hypothetical proposed -std=safe11 or -std=friendly-c C dialect is impossible to create without creating another full-blown language spec.
Oh, and coming back to your previous example:
Posted Jul 16, 2021 18:08 UTC (Fri)
by khim (subscriber, #9252)
[Link] (5 responses)
There's Well… the fact that it's a warning says that it's dangerous, but supported thing, isn't it? And I can change the code to make sure there are no warnings:
Better now? You mean: without explicit The really funny thing: just a tiny modification of that code makes it compileable with -O2, too. Thus modern compilers don't really propagate all “undefined behaviors” blindly. No, they are more-or-less testing C and C++ developers patience.
Posted Jul 16, 2021 23:42 UTC (Fri)
by HelloWorld (guest, #56129)
[Link] (4 responses)
Posted Jul 17, 2021 0:11 UTC (Sat)
by khim (subscriber, #9252)
[Link] (3 responses)
Seriously? POSIX guarantees it's availability, Windows doesn't need (it doesn't support signals), MS-DOS & embedded are handled with Maybe, but that's the thing: it works on almost everything. Thus simple rule “things are disallowed if they couldn't ever work” doesn't cover it. Maybe, but safer option would be to switch to something that doesn't change it's rules every year. Most likely. This would break almost all optimizations based on these “noalias” rules which power a lot of optimizations. But the story here is: what's the point of something being speedy if said something is no longer correct? If there were some kind of dialogue between C/C++ compiler developers and users of said compilers then maybe they would have come to an agreement. But there are none.
Posted Jul 17, 2021 13:26 UTC (Sat)
by HelloWorld (guest, #56129)
[Link] (2 responses)
> Maybe, but safer option would be to switch to something that doesn't change it's rules every year.
> But the story here is: what's the point of something being speedy if said something is no longer correct?
Posted Jul 17, 2021 13:49 UTC (Sat)
by khim (subscriber, #9252)
[Link] (1 responses)
Please read the documentation again. You can not ever set But C and C++ are making it harder and harder each year! Because time which is spent making pointless changes which would keep tested and debugging programs working could have been spent on the changes which would make these programs faster instead. And in wast majority of cases you can get more speedup from doing that. Where new version of compiler can buy you 3% or maybe 5% (if you are lucky) manual tuning can often bring 2x, 3x or even 10x improvements. And the more artificial constructs you add to your program the slower it becomes. No wonder that developers of embedded systems where efficiency would be obviously desired typically stick to one version of the compiler (which is not upgraded ever) or, sometimes, even use
Posted Jul 17, 2021 14:51 UTC (Sat)
by excors (subscriber, #95769)
[Link]
Based on a majority of the embedded code I've seen, the actual reason is that most embedded software developers are bad software developers. You're lucky if they'll write code that isn't full of obvious buffer overflows, race conditions, etc, and there's almost no chance they'll appreciate concepts like aliasing or memory barriers. That's why it's no surprise their 'tested and debugged' code breaks when they turn on new optimisations - it was only working in the first place by luck and by the very limited extent of their testing. And even when projects are run by competent software developers, they'll probably have to import some third-party libraries (chip SDKs, drivers, etc) which are badly written and/or are precompiled binaries from an ancient compiler version, so they face the same problems.
(To be fair these embedded software developers may have a much better understanding of hardware than the average software developer, which is crucial for doing their job; but they're still bad at software.)
Posted Jul 17, 2021 21:08 UTC (Sat)
by anselm (subscriber, #2796)
[Link] (15 responses)
What do you mean “works”? Prints “42”? Only by happenstance. That's not something I would rely on, ever. As I said, it's really crummy C code, and it doesn't prove anything.
Posted Jul 17, 2021 21:59 UTC (Sat)
by HelloWorld (guest, #56129)
[Link] (14 responses)
Given some of khim's other comments, this is the kind of thing he seems to expect platform developers to do when dealing with broken code. Except of course he's going to say that said code is not broken because, after all, it has been working for decades, and the problem is clearly you and your toxic attitude.
Well, I have a different opinion. If your code is broken, LLVM (or GCC) developers are under no obligation to accommodate that. The fact that they still do in some ways (with options like -fwrapv or -fno-strict-aliasing) is them being nice.
Posted Jul 17, 2021 23:44 UTC (Sat)
by Vipketsh (guest, #134480)
[Link] (12 responses)
I have to wonder .. do you consider the Linux policy of not breaking user space, even buggy ones, them being overly nice ?
Clearly no one would accept Linux changing the semantics of read() every release and substantiate that with "we never agreed to be posix compliant". This is what user's of compilers feel is happening to them (please read below). It's easy to brush off complaints when you are screwing over others, but it's seriously not great when others are screwing you.
I agree that the examples given are not spectacular in that for most of them I don't think one can expect any explicit well defined result. However, I think that there are still reasonable expectations for things the compiler will not do (e.g. start operating on non-rings).
Reality and de-facto standards are also important. For decades compilers compiled code with the semantics of wrapping 2's complement signed arithmetic thereby creating an implicit agreement that that's how signed integers work. Compiler writers then unilaterally broke the agreement without much warning.
In some ways I can understand that only a few people will ever fully comprehend all the fine print in a standard and sometimes user's have the wrong idea and things break. The issue here is that these breakages occur all too frequently and when they do the argument isn't an explanation about how the new behaviour makes sense or issues about what the old behaviour had it's just bluntly "some paper with black splodges says we can".
The situation isn't helped by the fact that many of these surprising issues occur because the compiler is explicitly special casing something (looking for realloc(), null pointer checks, etc.). Clearly someone put in the effort to look for these things and then make deductions based on them. Presumably there is a reason why all this effort was made but that reason never seems to be given. There is some understanding that the reason is probably some performance improvement in which case I think it is reasonable to expect compiler authors to be able to point to a commit log or discussion where the numbers were passed around, but it never happens.
In the end what users believe is that the compiler is miss-compiling their valid code and after some long & painstaking debugging the other party just tells them to get lost. You see why people get upset ?
Posted Jul 18, 2021 1:25 UTC (Sun)
by HelloWorld (guest, #56129)
[Link] (11 responses)
> Reality and de-facto standards are also important. For decades compilers compiled code with the semantics of wrapping 2's complement signed arithmetic thereby creating an implicit agreement that that's how signed integers work.
Posted Jul 18, 2021 8:14 UTC (Sun)
by Vipketsh (guest, #134480)
[Link] (9 responses)
So, I think this is a key point to understand about C. In C you can and want to have pointers going to any arbitrary piece of memory (e.g. for I/O access) and as you say no standard can hope to define what happens when some data structure, or in more extreme cases, code is arbitrarily overwritten. The reasonable thing to do when analysing C code is to assume such things don't happen, and not to go out of your way to find such cases and generate garbage.
> There's no way the compiler can be expected to guard against that.
Sure, it's unreasonable to expect the compiler to guarantee that your pointers don't overwrite something you didn't expect. It's a completely different matter when the compiler goes out of its way to look for these things and then generating complete garbage, not because it's sane for some surprising reason, but because "it's allowed".
Let's also not forget that in the original example of (be || !be)=false there was no random overwriting of anything. It was reading an un-initialised variable, which is a little easier to reason about -- there is no explicit value one can expect when reading un-initialised variables, at the same time it is also reasonable to expect that the value that one gets does not fall outside of the set of allowed values for that type.
There is also another piece of to this puzzle: only clang assigns values outside the set of allowed values for the type, gcc assigns 0. Yet gcc's performance on the all-important SPEC benchmarks is not lagging miles behind clang so it's clearly not an all important optimisation. So the yet-unanswered question is raised of what advantage does clang's behviour have ? The disadvantages are clear.
> there's an "implicit agreement" that that's how free should “work”?
In some ways, yes, I do think there is implicit agreement here. I also think that user-after-free is not a good example because just like defining what overwriting arbitrary pieces of memory will result in is impossible, so is use-after-free. Still, I think the compiler should not go out of its way to detect use-after-free type situations and then generate garbage in responce.
A different but similar example perhaps would be the case of "The sims", which was known to write past allocated memory areas. It happened to work on Windows 9x. In later versions of windows they changed the allocator and put some internal data structures in those locations resulting in the game not working anymore. In the end Microsoft changed the allocator behaviour such that if you asked for X bytes you got X+Y bytes. Not logical behaviour when looking an allocator but well defined. For me, such changes are a reasonable price to pay for compatibility.
Posted Jul 18, 2021 11:34 UTC (Sun)
by jem (subscriber, #24231)
[Link] (8 responses)
Out of curiousity, what are these clear disadvantages? If be is undefined, then (be || !be) is also undefined. This is "garbage in, garbage out" at work here. You can't create information out of nothing. It's madness to start reasoning about this before fixing the underlying problem first, i.e. make sure the variable has a value before using it.
On the contrary, I would say the compiler did you a favour by giving a hint in the form of an unexpected result. A hint telling you that there is something wrong with the code. Of course it would have been more helpful if the compiler had printed the real error, that the variable is uninitialised. Which is what compilers usually do nowadays, when they are able to detect the it.
Posted Jul 18, 2021 12:43 UTC (Sun)
by Vipketsh (guest, #134480)
[Link] (5 responses)
It breaks the foundations of the arithmetic all users expect.
My experience with Verilog, that has a number system that is not a ring opposed to (almost) all software, tells me that the number one thing people expect is ring-like behaviour, more so than operating on an infinite set. You can argue that all people are mentally completely deficient when they expect that if they have INT_MAX apples, acquire an additional two and finally eat two of them there are going to be INT_MAX left instead of "poison" number of apples, but perhaps it would be easier to just apply the models that work in the real world and have done so for more than a century ?
> You can't create information out of nothing.
Pretty much the entirety of mathematics is about "creating information out of nothing" and that's the beauty of it. Guess what ? In the given case, the maths says that (be || !be) is true. Not undefined or "poison", but true. In all cases. No exceptions.
You could argue that this isn't a normal system but "LLVM special". It's a valid argument, but you can't have it both ways and keep sane results like LLVM tries in vain: it uses a non-ring but then applies transformations that are valid only on a ring.
> On the contrary, I would say the compiler did you a favour by giving a hint in the form of an unexpected result.
What an overly kind compiler that is. What's next ? I should be donating them a kidney because they were oh-so-kind to insert an arbitrary write in my code, corrupting memory as a way of "warning me of an unexpected result" ?
It's incredibly difficult to figure out what happened when the compiler is doing everything right, and your code is clearly at fault (think: memory corruption). Suggesting that the compiler introducing random behaviour "is helping the user" is preposterous.
Posted Jul 18, 2021 13:24 UTC (Sun)
by mathstuf (subscriber, #69389)
[Link] (4 responses)
Of course, this is assuming that `be` and `i` do not change from access to access. However, this is exactly the kind of assumption that is allowed for uninitialized variables. Sure, machines today might have all values in the normal range be the only values supported and they won't change willy-nilly, but the Mill has a NAR result; this could certainly be the representation for uninitialized variables in which case C would either need to inject an initialization for you (what rules exist for this?) or do some complicated runtime NAR tracking to handle the cases given above (because the operations will generally result in NAR propagation).
I think people need to remember that "the hardware" is *not* what C targets. Hasn't been for a long time either. The rules of the C Abstract Machine are not beholden to "what currently prevailing hardware does" and nor should it unless it wants to do one of:
- make uninitialized variables a compile error (like Rust does) and require source code changes to continue using newer compilers;
Of course, C (and C++) have painted themselves into a corner by trying to say:
- old code will continue to compile (though not necessarily with the same behaviors :/ ) (this has been broken somewhat by the removal of trigraphs and maybe removing some EBCDIC allowances);
Rust, on the other hand, says "we're willing to give up some future opportunities and supporting older platforms at the expense of better available reasoning about the code at compilation time on the hardware that is widely used today". Sure, if some new floating point format appears, Rust's IEEE expectations are going to have to be emulated where that gets used preferentially. But it means that Rust has chosen "working on extant hardware today as well as possible" over "works on ancient stuff, current stuff, and whatever may exist in the future". Given how much extant language design ends up influencing marketable hardware, I think the former is more useful. But who knows, maybe security will become a big enough problem that capability-based processors will be a thing and Rust will then be in a corner and C able to morph over into defining its behavior on such a machine. It's not a bet I would take, but it's also a possibility.
Posted Jul 18, 2021 14:17 UTC (Sun)
by khim (subscriber, #9252)
[Link]
Period. Full stop. End of discussion. If certain behavior is dictated by hardware then compiler shouldn't invent it's own rules which permit it to Sure, it may invent some additional rules (e.g. Rust may use the fact that certain types don't use all bit patters to save some memory) — but then it becomes responsibility of the compiler to maintain these additional rules and it must abort an attempt of the user to violate such rules. It's idiotic to just assume that user is simultaneously super-advanced and knowledgeable and remembers all these hundreds of UBs defined in standard (and not defined by standard too, as we now know) yet, simultaneously, is dumb enough to write code which would do something wrong — unconditionally. Why not? The only reason compilers exist and are used is to take programs and execute them on hardware. Why make that task unnecessarily complicated? Dream on. Just don't forget to do a reality check when you would wake up. No. There are no such possibility. People are not writing code for “abstract C machine”. They are writing code for the existing hardware and then fight the compiler till it works. I have talked with a guy who participated in the development of compiler for E2k CPU (which does have capability-based processor… (or, rather, it had in the initial design, not sure if they kept it). Approximately zero non-trivial C programs can be compiled in strict mode. Because in any non-trial program sooner or later you hit a code which assumes that pointer is just a number. Maybe a weird number (like Similarly you hit code which assumes than numbers are two's complement and so on. Hyrum's Law ensures that you couldn't ever transfer non-trivial C or C++ codebase to a radically new architecture (one of the reasons why all CPUs today are so strikingly similar, BTW). Safe Rust rules, on the other hand, can happily coexist with capability-driven CPU. And given the fact that Rust developers try to minimize use of unsafe Rust (and it's always clearly marked when used) port of Rust code to the capability-driven CPU is quite feasible. If we ever would switch to capabilities-driven CPUs then C and C++ wouldn't survive the transition for sure. While Rust might. What's the issue with that code? Pick any single value and calculate the answer, if you can. As long as mental model “uninitialized variable contains whatever garbage which was found there when memory was allocated” holds… people would accept it. Sure, there are programs which would be broken with these optimizations, but it's very hard to find someone who thinks they should work. Using standard as a guide which includes current state of affairs is fine, too. But when following the standard gives surprising (to the user) result then changes to the standards should be contemplated, too. I remind you, once more, what C committee said explicitly: WTH this recommendation (which would have been widely welcomed by C and C++ developers) was followed so rarely and contradicted so often?
Posted Jul 18, 2021 16:49 UTC (Sun)
by Vipketsh (guest, #134480)
[Link] (2 responses)
The reason I think the (be || !be)=false result is unreasonable is not because LLVM is unable to prove that it should evaluate to true. Instead, my problem is because LLVM goes out of its way to use a non-ring to evaluate it and I have no idea what the advantage is in doing that. If it just didn't try to do anything special, perhaps even just emitted the expression as-is in machine instructions, it would arrive at the expected result.
Posted Jul 18, 2021 17:14 UTC (Sun)
by khim (subscriber, #9252)
[Link]
“Dead code elimination”. The idea the same with most “optimizations” which break formerly valid code: programmer is utterly and inescapably deeply schizophrenic entity which:
If we are dealing with such an entity then using #1 to combat #2 is entirely reasonable. Unfortunately in real life programmers are exact opposite: they tend not to produce too much garbage yet sometimes create UBs by mistake. Applying these principles to code written by real people turns compiler writers into their adversaries, but that's OK since these users are not the ones who pay salaries to compiler writers, isn't it?
Posted Jul 19, 2021 8:36 UTC (Mon)
by smurf (subscriber, #17840)
[Link]
That's not the problem here, and if it was then the compiler should treat them as potentially aliased.
Instead, it treats p as both valid (the location it points to can be written to) and invalid (it cannot possibly point at the same location as another pointer) at the same time. That's the very antithesis of correctness.
Posted Jul 18, 2021 13:08 UTC (Sun)
by mpr22 (subscriber, #60784)
[Link] (1 responses)
The compiler doing me a favour would be if it defaulted to throwing a fatal error (not a warning, a fatal error, your code doesn't compile, fix it please) when it is asked to compile code which invokes UB 100% of the time and can be cheaply proven to do so.
Posted Jul 19, 2021 9:56 UTC (Mon)
by farnz (subscriber, #17727)
[Link]
This is something I hope future languages steal from Rust (along with Esteban Kuber's great work on high quality error messages): there is no UB in parts of the code not marked out as potentially containing it.
Now, there's a gotcha with this, in that while it's always an unsafe block in Rust that actually invokes UB, it's possible for the preconditions for that UB to be set up by Safe Rust (e.g. you set an offset into an array in safe code, but deference it in unsafe code), but you always know where UB could possibly be found in a body of Rust code, and you can use module scoping (privacy) to ensure that UB can't leak - with that done, it's possible to check an entire module that contains an unsafe block, and be confident that your human analysis shows no UB.
Posted Jul 18, 2021 10:17 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Another story about compilers: it had taken Microsoft more than 5 years to move their C# and C++ compilers onto a new backend. Not because it was so complex, but because they wanted to maintain bug-for-bug compatibility with the old compiler.
They went to their biggest clients and made sure that huge codebases, like the ones that Adobe has, work perfectly.
This is another reason why Windows platform is still the main desktop environment.
Posted Jul 18, 2021 7:19 UTC (Sun)
by smurf (subscriber, #17840)
[Link]
You're getting off on a tangent. The code in question is not SimCity and their use-after-free bug.
The realloc example we're talking about is not broken in any way, shape or form. It's a valid optimization technique to not update pointers after realloc if you determine that the pointer didn't change. LLVM still breaks. There is no freakin' reason why that can possibly be OK.
If your nice and shiny pointer provenance rules and optimization break valid code, then, sorry, but at least one of those rules or optimizations is broken.
Posted Jul 15, 2021 15:43 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link] (21 responses)
Then why won't the compiler stop me? That is also a sane behavior in the face of such code, but we can't have that either apparently.
Posted Jul 15, 2021 15:58 UTC (Thu)
by smurf (subscriber, #17840)
[Link] (19 responses)
But that would cause old code to *gasp* Not Compile. No, we can't have that, because fixing this problem requires amending the standard so that programmers can tell the compiler what they mean.
Apparently, though, the compiler is by definition the all-knowing oracle which, by following The Holy Standard, is able to understand all by itself what a program may or may not mean. Thus actually telling it to behave differently is thus Abject Blasphemy, and amending the standard to afford such horrors would thus be Vile Heresy.
… or so it seems to me.
Prove me wrong.
Posted Jul 15, 2021 16:09 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link] (18 responses)
Posted Jul 15, 2021 21:22 UTC (Thu)
by mrugiero (guest, #153040)
[Link] (17 responses)
Yet nothing in the standards mandate it. How hard could it be to make them warn about it as part of the spec?
Posted Jul 16, 2021 0:44 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link] (16 responses)
- inundate projects with oodles of warnings which they'll promptly disable anyways; and
But you'd have to ask the implementers for specifics if you want more. All I can say is that suggestions to do such things are not taken lightly on the implementer side at ISO C++ meetings.
Personally, I'd be fine with a warning level to such heights if just to show people how little they actually understand about C and C++ behaviors. Instead, we get monster threads like this one one a semiregular basis.
Posted Jul 17, 2021 15:41 UTC (Sat)
by mrugiero (guest, #153040)
[Link] (15 responses)
> - compilers would become much slower in order to track and issue diagnostics for such things.
> But you'd have to ask the implementers for specifics if you want more. All I can say is that suggestions to do such things are not taken lightly on the implementer side at ISO C++ meetings.
> Personally, I'd be fine with a warning level to such heights if just to show people how little they actually understand about C and C++ behaviors. Instead, we get monster threads like this one one a semiregular basis.
Posted Jul 18, 2021 0:00 UTC (Sun)
by HelloWorld (guest, #56129)
[Link] (14 responses)
Posted Jul 18, 2021 3:33 UTC (Sun)
by mrugiero (guest, #153040)
[Link] (13 responses)
Posted Jul 19, 2021 10:26 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (12 responses)
The problem with UB is not when all ways to interpret the code interpret UB - everyone, even compiler writers, agrees that there should be diagnostics for that.
The problem is when there are multiple ways to interpret the code, such that some interpretations don't lead to UB, but others do. The classic example is:
There are two interpretations here; either forty is NULL, or it isn't. If forty is NULL, then ptr->value; is UB, and thus the code snippet invokes UB. If forty is not NULL, then ptr->value is not UB, and so the compiler reasons that !forty must be false, always.
And generated code, among other uses of C compilers, often depends on the compiler being able to spot that an expression must be true, and optimising accordingly. As a result, there's plenty of real code that depends on the optimizations that fire when forty cannot be NULL, and the compiler simply works backwards from what it knows - that if forty was NULL, then there would be UB, ergo forty must be known not to be NULL.
In this particular example, the compiler's reasoning chain (and the user's mistake) is fairly obvious, and it wouldn't be challenging to have a warning that says "hey, !forty is always false because ptr->value would be UB otherwise, so this if statement is not meaningful", and have the user spot the error from there, but it gets more challenging with complex code. And neither compiler developers nor standard committees seem willing to say that good C compilers give a friendly warning when they find a codepath that invokes UB - in part because it'd trigger a lot of noise on legacy codebases.
Posted Jul 19, 2021 11:20 UTC (Mon)
by smurf (subscriber, #17840)
[Link] (7 responses)
Three. The third is that whoever wrote the program mis-ordered these statements. So the compiler might as well rearrange them (the code has UB, so it's allowed to do that), and emit a warning while it's at it.
Posted Jul 19, 2021 11:45 UTC (Mon)
by farnz (subscriber, #17727)
[Link]
That is not an interpretation of the code as written - that's a guess at the authorial intention. And while this is a simple case, chosen because it's painfully obvious what the author meant, but it's also obvious how the compiler gets to "!forty must be false", one of the things that makes UB so painful is that the chain of reasoning the compiler uses from "if this is true, then UB" to "the meaning of this program is nowhere near what the author intended" can be huge, and even span multiple files if you're unlucky (assumption of not-NULL in a header file resulting in the only non-UB case being pointer is not NULL, or unsigned value is less than 64, or more complex things around signed overflow).
Which is why I think that there should be a way to mark areas where UB could happen (Rust uses the unsafe keyword for this), and the compiler should be on the hook for defining all behaviour outside those blocks. If that's not reasonable (e.g. due to legacy code), then as a QoI issue, I'd like compilers to explicitly call out in a warning when they're reasoning backwards from "anything else would be UB".
And yes, I know this is not easy. But it'd help out the people who want to write good modern C or C++ if their compilers could be trusted to alert them when there's a risk of UB resulting in badly behaved programs.
Posted Jul 19, 2021 12:46 UTC (Mon)
by excors (subscriber, #95769)
[Link] (5 responses)
Reordering the statements means it would have to emit instructions to perform the NULL check. If the function is never called with NULL, that's an unnecessary performance cost. It's not a "might as well" situation - it's choosing to sacrifice performance of correct code, in exchange for less surprising behaviour if the code is buggy.
I think farnz's point about "generated code" is particularly relevant for C++ because many libraries (including the standard library) depend heavily on template metaprogramming, which is basically a program that runs at compile-time where the inputs are C++ types and the outputs are C++ functions, so most C++ programs will include a lot of generated code in that sense. And that relies heavily on inlining, constant propagation, dead code elimination, etc, to get good performance - the metaprogram itself is (usually) functional and very recursive and has huge amounts of dead code, but needs to end up compiling the same as a simple imperative function.
Because the library developers know they can rely on the compiler doing that optimisation, they can design the libraries to be generic and type-safe and easy-to-use, while still getting as good performance as specialised hand-tuned code (sometimes purely by relying on clever compiler optimisation, sometimes by explicitly selecting different algorithms based on properties of the type it's being specialised for). And then users of the library can write much safer programs than if they tried to implement it by hand, because the library has been carefully designed and tested and reviewed by experts for years.
If you try to sacrifice performance for safety, then those libraries may become much slower and developers would stop using them, so they'd write bad manual replacements, and then you'd end up losing performance *and* safety.
(That doesn't apply so much to C, which doesn't have that metaprogramming facility (the closest thing is macros, which are terrible). But I guess the issue there is that few people care enough about C to write a dedicated C compiler, they just write a C++ compiler and stick a C frontend on it, and if C programmers refuse to use C++ then they'll share the costs of that C++-focused optimisation but will miss out on most of the high-level safety benefits.)
Posted Jul 19, 2021 13:22 UTC (Mon)
by mrugiero (guest, #153040)
[Link] (4 responses)
> If you try to sacrifice performance for safety, then those libraries may become much slower and developers would stop using them, so they'd write bad manual replacements, and then you'd end up losing performance *and* safety.
Posted Jul 19, 2021 13:58 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (2 responses)
Trivial plain C counterexample, where the check is seen by the compiler, but the programmer did not intend to tell the compiler that the pointer could be NULL:
In this case, the call to GET_INT_CONFIG_OR_DEFAULT (a macro) has promised that config is not NULL because it does an early return; a compiler that doesn't optimize this will be issuing repeated NULL checks for something that it knows must be non-NULL, or complaining because the user has an unnecessary NULL check in the expansion of GET_STR_CONFIG_OR_DEFAULT.
The code the compiler sees when the preprocessor hands it off is:
Which leads you into the question of provenance of your "always true" if statement; in this case, the compiler has to know that it knows config is not NULL because the programmer did something that's an explicit check, not a "if it's NULL, then we have UB" case, otherwise you get a noisy warning because the programmer included an unnecessary NULL check in the macro. But they did that because that way the macro is safe to use even if it's only used once in a function, and if the user does do:
then the compiler will need to warn because its "config is not NULL" assumption came from "if config is NULL, then there is UB in the execution". But first we need compiler authors to get onboard with the idea that whenever UB is invoked to permit an optimization (noting that this can require you to do complex tracking to get from your low-level IR (MachineInstr in the LLVM case, for example) all the way back to C, so that you can confirm that a given decision in codegen is not causing issues because it results in bad signed overflow behaviour (for example).
Posted Jul 19, 2021 19:11 UTC (Mon)
by mrugiero (guest, #153040)
[Link] (1 responses)
> Which leads you into the question of provenance of your "always true" if statement; in this case, the compiler has to know that it knows config is not NULL because the programmer did something that's an explicit check, not a "if it's NULL, then we have UB" case, otherwise you get a noisy warning because the programmer included an unnecessary NULL check in the macro. But they did that because that way the macro is safe to use even if it's only used once in a function, and if the user does do:
> /* In a C file, using the header file */
> /* Do the amazing things */
> then the compiler will need to warn because its "config is not NULL" assumption came from "if config is NULL, then there is UB in the execution". But first we need compiler authors to get onboard with the idea that whenever UB is invoked to permit an optimization (noting that this can require you to do complex tracking to get from your low-level IR (MachineInstr in the LLVM case, for example) all the way back to C, so that you can confirm that a given decision in codegen is not causing issues because it results in bad signed overflow behaviour (for example).
The only thing I said is always true is that you can't dereference something you can't prove not to be NULL and expect it to not be UB; the check reordering is allowed because it was UB before and we can assume at that point what the author wanted. This doesn't mean a warning should not be emitted unless (somehow) specified otherwise. If there's a check before in the same function it's trivial to prove it's no longer NULL after that. Besides, you don't need to know whether an optimization resulted from the UB. Knowing it exists is enough. And AFAICT that can be known before transforming to IR.
Posted Jul 20, 2021 11:54 UTC (Tue)
by farnz (subscriber, #17727)
[Link]
This is the difference between your model, and the C model. In C, you can dereference something that you can't prove not to be NULL, and expect it to not be UB; it's only UB if, on a specific execution, the thing is NULL. If it happens not to be NULL, then there's no UB. This comes down to whole program analysis - as a programmer, you might ensure that you only call ub(config) with non-NULL pointers, and thus it's fine.
So the C compiler is able to reason backwards - if thing *is* NULL, then there is UB. Ergo is must not be NULL, because UB isn't allowed, which permits optimization on the basis that it's not NULL. This is fine if the explicit NULL check is (e.g.) from a macro, or in generated code, or left over from refactoring and not yet removed; you've taken out something that's always false in this context, and just deleted the dead code.
It only becomes a problem where the human is surprised by the deletion of dead code - i.e. where the human thought the check was still important, but it's actually considered always false (or always true) by the compiler. And as deletion of dead code takes place at every level - from AST, through IR, through machine instruction choice - the compiler needs to link back its elimination of dead code and determine whether or not the human would be surprised by this particular elimination.
And that's what makes it a hard problem - we may know that ub(config) is only ever called with a non-NULL pointer, but that needs a whole program analysis which the compiler cannot perform. We may also know this simply because we've refactored so that callers of ub(config) do the NULL check themselves, and the extra NULL check inside the function may be there because a preprocessor (which can be outside the compiler!) has used a general form of code that works regardless of the presence of the NULL checks; why use two macros and force the human to think about whether or not a NULL check has happened already, when you can use one?
Posted Jul 19, 2021 14:56 UTC (Mon)
by excors (subscriber, #95769)
[Link]
But the compiler may not know that. E.g. you could have code like: (i.e. there's a dynamically-allocated array of pointers to S, and the function deallocates all the Ss that match some condition. This is somewhat artificial, but I don't think it's totally implausible as a component of some larger operation). The function's precondition is guaranteed by the programmer but is not visible to the compiler. (Maybe the caller is in a separate translation unit, or a dynamically-loaded library.) On "item = nullptr", the unique_ptr will deallocate its original contents (to prevent memory leaks). Because unique_ptr is generic and supports custom deallocator functions, and the custom deallocator might not accept nullptr as an argument, the unique_ptr destructor is specified to do something like "if (get() != nullptr) get_deleter()(get());". In this function, the unique_ptr destructor is only called after the "item->n % 2" check, which is dereferencing the pointer. Then the compiler knows the pointer can't be nullptr, so it can omit the "if (get() != nullptr)" test and save some time and code size. (In this case it's only a couple of instructions, but it's not hard to imagine similar cases where the 'if' is followed by an 'else' with a much larger chunk of code, and eliminating the whole 'else' clause could make a big difference.) The null check wasn't intentionally added to this code by any programmer, it's just a part of the (relatively) safe abstraction provided by unique_ptr, so it's included in the code that's generated by the template instantiation. In this function the code is used in a way where the check is redundant and the compiler is able to figure that out because it sees the programmer was deliberately dereferencing the pointer, so the abstraction has zero cost and the programmer doesn't have to revert to a less-safe kind of pointer to get the same performance.
Posted Jul 19, 2021 13:16 UTC (Mon)
by mrugiero (guest, #153040)
[Link] (3 responses)
> In this particular example, the compiler's reasoning chain (and the user's mistake) is fairly obvious, and it wouldn't be challenging to have a warning that says "hey, !forty is always false because ptr->value would be UB otherwise, so this if statement is not meaningful", and have the user spot the error from there, but it gets more challenging with complex code. And neither compiler developers nor standard committees seem willing to say that good C compilers give a friendly warning when they find a codepath that invokes UB - in part because it'd trigger a lot of noise on legacy codebases.
Posted Jul 19, 2021 13:27 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (2 responses)
> AFAICT, the logical conclusion here is that every pointer argument can be NULL unless proved otherwise. Since you can't prove that for non-static functions, that's necessarily UB. This is why many compilers have extensions to explicitly tell the compiler an argument can never be NULL.
This is why the function included the check "if (!forty) { return; }"; after that test, forty is provably not NULL, and the compiler can optimize knowing that forty is not NULL. The gotcha is that the user invoked UB before that check by dereferencing ptr, and the compiler used that information to know that forty cannot be NULL after the dereference, because if it is, UB is invoked.
> > In this particular example, the compiler's reasoning chain (and the user's mistake) is fairly obvious, and it wouldn't be challenging to have a warning that says "hey, !forty is always false because ptr->value would be UB otherwise, so this if statement is not meaningful", and have the user spot the error from there, but it gets more challenging with complex code. And neither compiler developers nor standard committees seem willing to say that good C compilers give a friendly warning when they find a codepath that invokes UB - in part because it'd trigger a lot of noise on legacy codebases.
> Which was exactly my point. Besides, the point of triggering noise is moot IMO, since you can always add a flag to ignore the warning like with any other warning.
And my point is that this is entirely a social problem - nobody with the power to say "compilers that use UB to optimize must tell the user what UB the compiler is exploiting" is willing to do so. If compiler writers for C were doing that, even if it was the preserve of the very best C compilers (so GCC and CLang), it'd be enough to reduce the number of developers who get surprised by compiler use of UB.
Posted Jul 19, 2021 18:57 UTC (Mon)
by mrugiero (guest, #153040)
[Link] (1 responses)
> And my point is that this is entirely a social problem - nobody with the power to say "compilers that use UB to optimize must tell the user what UB the compiler is exploiting" is willing to do so. If compiler writers for C were doing that, even if it was the preserve of the very best C compilers (so GCC and CLang), it'd be enough to reduce the number of developers who get surprised by compiler use of UB.
Posted Jul 19, 2021 21:10 UTC (Mon)
by farnz (subscriber, #17727)
[Link]
And the problem with your point is that C is not very amenable to whole program analysis (for a variety of reasons to do with only having a global and a local static namespace, plus using textual inclusion instead of modules), which means that there's a lot of legacy code where the first dereference is not UB. It's only UB if the program execution can call ub with forty being NULL, which is not guaranteed. If the rest of the code ensures that ub is always called with a valid pointer to configs, then no UB is present.
As I've pointed out in another comment, that check could be the consequence of a macro expansion - in which case, pessimising the output machine code because you've used a macro to access configs isn't a good look - there was no UB before, now you're adding code that verifies something that the human writing the code has already verified is true outside ub, and the use of a macro has been sufficient to introduce extra machine instructions.
This is part of why it's a damn hard social problem - it's possible that ub is only called from code in 3 different C files, and all of those files already check for NULL before calling ub, so in the actual final application, there is no UB. The compiler using the fact that the check can only be true and cause an early return if UB has already happened to optimize ub simply speeds things up at no cost.
Posted Jul 15, 2021 16:38 UTC (Thu)
by HelloWorld (guest, #56129)
[Link]
Footguns
Because you're writing to a memory location that you previously declared to be const. Why would you expect sane results when you do things that are so obviously not sane? And in fact this code requires a cast, which is a pretty clear signal by the compiler that you're doing something fishy.
> Because you're writing to a memory location that you previously declared to be const.
Footguns
int
it becomes negative I know also that there are no const objects on stack (at least on most OSes).qsort
without casts, but if you want to try to optimize something without casts… here we go:
#include <stdio.h>
int foo() {
int i;
i = 42;
}
int bar() {
int i;
return i;
}
int main() {
foo();
printf("%d\n", bar());
}
const
. Am I safe now?-std=safe11
fail.-std=safe11
. Ever. Deal with it.Footguns
Well, your example doesn't look sane to me 🤷🏻♂️
> Well, your example doesn't look sane to me 🤷🏻♂️
Footguns
Footguns
As you wanted: no casts, no const. Am I safe now?
> That example may have no casts or Footguns
const
but it is still really crummy C code.
-std=safe11
or -std=friendly-c
C dialect is impossible to create without creating another full-blown language spec.-O0
mode should work. Because that would mean you have to, somehow, correctly compile also an atrocities I have shown.Footguns
I don't think it does. What if a signal or interrupt is delivered after the call to foo()? It would invoke a signal handler which is probably going to clobber the memory where foo's stack frame used to be.
That may or may not be true, but either way, your example doesn't demonstrate that. Every compiler these days has warnings to prevent that sort of thing.
> const int this_is_zero = 0;
> But I “know” it's on stack and stack is not const!
Why would the compiler allocate this on the stack? It's not using any local variables or function parameters, so it might as well allocate it statically on a read-only page.
> What if a signal or interrupt is delivered after the call to foo()?
Footguns
sigaltstack
for that. Or, if that's MS-DOS, cli
and sti
.
void foo(int* i) {
i[1] = 42;
}
int bar() {
int i = 0, j = 0;
foo(&i);
return j;
}
int main() {
printf("%d\n", bar());
}
auto
? I think even C89
had implicit auto
. I can add explicit auto
, if you want. Compiler still misbehaves.Footguns
There's sigaltstack for that. Or, if that's MS-DOS, cli and sti.
Interesting, I didn't know about that. Anyway, it's a platform-dependent mechanism which isn't guaranteed to be available, and even on Linux it's only used when SA_ONSTACK is used. So I still believe it's reasonable to make that undefined.
Well… the fact that it's a warning says that it's dangerous, but supported thing, isn't it?
That's up to you. If you want, make it -Werror=uninitialized instead.
Better now?
I don't consider it reasonable to make assumptions about how the compiler lays out the memory for local variables.
That said, I'll grant you that reasonable people can differ about something like this:
void foo(int *i, int o) {
i[o] = 42;
}
int bar() {
int i = 0, j = 0;
foo(&i, &j - &i);
return j;
}
int main() {
printf("%d\n", bar());
}
But I'm not enough of a compiler expert to be able to assess what the performance impact would be if this were allowed. I suspect it would be severe.
> Anyway, it's a platform-dependent mechanism which isn't guaranteed to be available
Footguns
cli
/sli
(and analogues)… so what kind of
“important platform” is in the problem?Footguns
As I already mentioned, sigaltstack is only used if you specify SA_ONSTACK. Or that is how I understand the documentation, anyway.
Sure, most software shouldn't be written in C or C++.
Well, nobody said writing high performance software was going to be easy. But there are legitimate use cases where you really do want to squeeze every last cycle out of the hardware.
> As I already mentioned, sigaltstack is only used if you specify SA_ONSTACK.
Footguns
SA_ONSTACK
. It's not something you set, but something you can read: sigaltstack
couldn't change state of altstack if you are in the middle of a signal handler and said altstack is currently in use!-O0
to free time for manual optimizations.Footguns
Footguns
But it's syntactically valid and works on most C compilers with optimizations disabled, isn't it?
Footguns
Footguns
Footguns
Let's say you do an out-of-bounds write somewhere in your program. There's a chance that it might overwrite the piece of machine code that was generated from the "return be || !be" statement, and it might replace it with "return 0" (I'm aware of W^X etc, but not all platforms support that). There's no way the compiler can be expected to guard against that. I think a reasonable debate can be had about what exactly should trigger undefined behaviour, and it seems to me that compiler authors agree, as evidenced by the fact that they offer flags like -fwrapv. But I don't think it's reasonable to expect compilers to make any guarantees about what happens after UB has been invoked, not even for the be || !be example.
Again, SimCity was working fine for many years, despite having use-after-free bugs. Does that mean there's an "implicit agreement" that that's how free should “work”?
I think the standard should be that a behaviour should be preserved (at least optionally) if it has been around for a long time and a reasonable person might have specified it in some form of documentation. Reasonable people might have written a C standard where two's complement representation is specified for signed integer types. But no reasonable person would write a specification for free that says you can still use the data you just freed, because that just doesn't make sense. Now of course people have different ideas about what reasonable people would or wouldn't do, but my point is that “this has been working for years, so it must continue to work” just isn't sufficient by itself.
Footguns
Footguns
Footguns
Footguns
- pessimize future potentials (such as having to inject initialization into every POD variable on the Mill); or
- declare that hardware today has all of the properties we can expect and enshrine them.
- we don't want to enshrine existing architectures and require future hardware to emulate any decisions made here (e.g., twos complement (yes, C++ has done this, but I don't know the status for C), IEEE floating point formats, etc.);
- ABI compatibility; and
- as close to "zero cost abstractions as possible".
> Sure, machines today might have all values in the normal range be the only values supported
Footguns
optimize destroy programs.far
pointer in Windows 16bit) but still a number.i < 2 || is_odd(i) || goldbach_conjecture_holds(i)
Undefined behavior gives the implementor license not to catch certain program errors that are difficult to diagnose. It also identifies areas of possible conforming language extension: the implementor may augment the language by providing a definition of the officially undefined behavior (emphasis mine).
Footguns
> Instead, my problem is because LLVM goes out of its way to use a non-ring to evaluate it and I have no idea what the advantage is in doing that.
Footguns
Footguns
Footguns
Footguns
Footguns
Footguns
Footguns
Footguns
Footguns
Footguns
Footguns
- compilers would become much slower in order to track and issue diagnostics for such things.
Footguns
As long as there's a flag, it's only a minor annoyance.
I thought of this, but I'd rather have a slower build than a broken program. Besides, the flag is enough to mitigate this if I'm OK with my program being completely broken but want faster builds.
I'm aware. But my point is this shouldn't be up for implementors to decide IMO. If our code is not doing what we think it's doing we need to know, it's as simple as that.
Agreed.
Footguns
So you're saying it's simple for the compiler to figure out what you think the code ought to be doing?
Footguns
No, I'm saying it's simple for the compiler to figure out that what you code means nothing and you obviously meant something, so in that case it isn't doing what you think it's doing :)
That's what happens when you hit UB. Nasal demons and all that. If your program has no meaning, that's not what you want it to mean.
Footguns
struct useful {
int value;
/* And a lot more bits */
};
void ub(void * forty) {
struct useful * ptr = (useful*) forty;
int value = ptr-> value;
if (!forty) {
return;
}
/* Do stuff with value */
}
Footguns
Footguns
Footguns
Footguns
If the programmer put the check there then the programmer told the compiler the pointer could be NULL, so it shouldn't assume the opposite unless it can absolutely prove the programmer was wrong.
The template metaprogramming case is special in the sense the compiler knows if for that particular invocation the argument would be NULL (although you can't pass pointers as template arguments AFAIR, if we assume that it relies on inlining then we know for sure once the check has been performed at the outmost call we can erase it from all of the recursive calls). So you don't really sacrifice that much performance.
Footguns
/* In a header file, for config handling */
struct configs {
/* Lots of pointers to config members here, omitted */
};
#define GET_STR_CONFIG_OR_DEFAULT(config, member, default) \
(config ? config->member ? config->member : default : default)
#define GET_INT_CONFIG_OR_DEFAULT(config, member, default)
(config ? config->member ; default)
/* In a C file, using the header file */
void do_amazing_things(void *config) {
int rating = GET_INT_CONFIG_OR_DEFAULT(config, rating, 0);
char * expertise = GET_STR_CONFIG_OR_DEFAULT(config, expertise, "novice");
/* Do the amazing things */
}
struct configs {
/* Lots of pointers to config members here, omitted */
};
/* Rest of headers etc */
void do_amazing_things(void *config) {
int rating = (config ? config->member : 0);
char * expertise = (config ? config->member ? "expertise" : "expertise");
}
/* In a C file, using the header file */
void do_amazing_things(void *config) {
int rating = ((struct configs*) config)-gt;rating;
char * expertise = GET_STR_CONFIG_OR_DEFAULT(config, expertise, "novice");
/* Do the amazing things */
}
Footguns
The compiler can prove the extra check is unnecessary there. I'm not sure it would be wrong for it to complain, but it's an entirely different case than adding a test after a dereference of something you can't prove to not be NULL.
> void do_amazing_things(void *config) {
> int rating = ((struct configs*) config)-gt;rating;
> char * expertise = GET_STR_CONFIG_OR_DEFAULT(config, expertise, "novice");
> }
Footguns
> The template metaprogramming case is special in the sense the compiler knows if for that particular invocation the argument would be NULL (although you can't pass pointers as template arguments AFAIR, if we assume that it relies on inlining then we know for sure once the check has been performed at the outmost call we can erase it from all of the recursive calls)
Footguns
struct S {
int n;
// ...
};
template <typename T, typename Pred>
void reset_if(T &items, Pred pred) {
for (auto &item : items)
if (pred(item))
item = nullptr;
}
// Precondition: all items are not nullptr
// Postcondition: all items are even-numbered or are nullptr
void remove_odd(std::vector<std::unique_ptr<S>> &items)
{
reset_if(items, [](auto &item) { return item->n % 2 != 0; });
}
Footguns
AFAICT, the logical conclusion here is that every pointer argument can be NULL unless proved otherwise. Since you can't prove that for non-static functions, that's necessarily UB. This is why many compilers have extensions to explicitly tell the compiler an argument can never be NULL.
Which was exactly my point. Besides, the point of triggering noise is moot IMO, since you can always add a flag to ignore the warning like with any other warning.
Footguns
Footguns
Of course. My point was that _on entry_ it isn't provably not NULL, so NULL dereference is a possibility in that line, and thus the conclusion should consistently that the first dereference is UB. My point is that there's no true ambiguity about that.
So we agree, then?
Footguns
Footguns
It does, unless you explicitly tell it not to by casting away const.