|
|
Subscribe / Log in / New account

The Linux Foundation's "security mobilization plan"

The Linux Foundation has posted an "Open Source Software Security Mobilization Plan" that aims to address a number of perceived security problems with the expenditure of nearly $140 million over two years.

While there are considerable ongoing efforts to secure the OSS supply chain, to achieve acceptable levels of resilience and risk, a more comprehensive series of investments to shift security from a largely reactive exercise to a proactive approach is required. Our objective is to evolve the systems and processes used to ensure a higher degree of security assurance and trust in the OSS supply chain.

This paper suggests a comprehensive portfolio of 10 initiatives which can start immediately to address three fundamental goals for hardening the software supply chain. Vulnerabilities and weaknesses in widely deployed software present systemic threats to the security and stability of modern society as government services, infrastructure providers, nonprofits and the vast majority of private businesses rely on software in order to function.



to post comments

The Linux Foundation's "security mobilization plan"

Posted May 25, 2022 16:53 UTC (Wed) by nickodell (subscriber, #125165) [Link] (47 responses)

These seem like some smart ideas. Some seem a little optimistic - I don't think you could really make a dent in, e.g. "Eliminate Root Causes of Many Vulnerabilities Through Replacement of Non-Memory-Safe Languages" with the equivalent of 20 developers working for a year. But they're targeting the replacement of OpenSSL and writing of a better Rust FFI, both of which seem like they're the lowest-hanging fruit, security-wise.

Does anyone know how likely this proposal is to get funded?

The Linux Foundation's "security mobilization plan"

Posted May 25, 2022 18:01 UTC (Wed) by jrtc27 (subscriber, #107748) [Link] (6 responses)

You'd get much more bang for your buck looking at cleaning up C/C++ code in order to be able to adopt CHERI. We have just a handful of developers working on CheriBSD and porting third-party software yet have all of FreeBSD^1 (userspace and kernel), a barebones KDE desktop stack running via Xvnc, PostgreSQL, nginx and WebKit all ported. 20 developer years could really make quite a difference.

That doesn't mean rewriting things in Rust or other languages is a bad idea (they provide more than just memory safety, after all), it's just not realistic for large swathes of the open source software corpus. Contrast that with adopting CHERI which, in the recent CHERI desktop study^2 (that gave rise to the KDE desktop stack and Xvnc), only 0.026% of the ~6 million LoC needed changing, or 1584 lines (the exact figure).

Of course, it'd also require new hardware, unlike adopting a safe language, and the hardware is still in the prototyping stage, though with Arm's Morello^3 you can get real silicon and if the program is a success one would reasonably expect a version of the technology to be incorporated into future versions of the architecture for real. More people demonstrating its viability and pushing for its adoption would surely help, too.

^1 minus ZFS and DTrace, for no real reason other than they're large chunks of software that need engineering effort put in
^2 https://www.capabilitieslimited.co.uk/pdfs/20210917-caplt...
^3 https://www.arm.com/architecture/cpu/morello

The Linux Foundation's "security mobilization plan"

Posted May 25, 2022 21:20 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (3 responses)

1. There is no reason we can't do both at once.
2. Software may be expensive/difficult to write, but hardware is expensive/difficult to deploy. There's no free lunch either way, and it's not immediately obvious to me how you conclude that the hardware route gives us "more bang for your buck."

The Linux Foundation's "security mobilization plan"

Posted May 25, 2022 21:25 UTC (Wed) by jrtc27 (subscriber, #107748) [Link] (2 responses)

Creating new hardware and recompiling software (with minor cleanups and, in the case of kernels/toolchains, porting) is rather easier than rewriting that same software; even creating the new hardware is still incremental, just a big leap. We've done the former for many new architectures and had the 32-bit to 64-bit transition, but nobody's rewritten the entirety of Linux, FreeBSD/OpenBSD/NetBSD, macOS, Windows, WebKit, KDE, GNOME and so in a new language.

But yes, you absolutely should do both. I even said as much...

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 15:19 UTC (Thu) by developer122 (guest, #152928) [Link] (1 responses)

Spoken as someone who's never seen the extreme cost or difficulty of hardware development.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 17:10 UTC (Thu) by willy (subscriber, #9762) [Link]

You might want to apologise for that comment. Jess worked on the CHERI project alongside hardware developers. I don't know exactly how much involvement she had with the hardware side of things, but suggesting that she has no understanding is just offensive.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 1:09 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

You'd get much more bang for your buck looking at cleaning up C/C++ code in order to be able to adopt CHERI.

This seems unrealistic. If you get things ready for CHERI, you still have an extended period where there's lots of vulnerable software out there because people don't want to replace their hardware, no matter how much extra security you promise them for doing so. In contrast, improving the security of software on existing hardware means everyone using it can benefit as soon as the patches are available. Basically, rewriting critical code to be more secure benefits everyone, while improving code to take advantage of CHERI benefits only people who have CHERI- which right now is hardly anyone.

That's not to denigrate what CHERI offers. We've been suffering for years from hardware that has sacrificed security in the name of performance (hello Spectre and related vulnerabilities), and fighting those weaknesses has been expensive in developer time and loss of some of the benefit the faster hardware was supposed to provide. It's great that someone is finally working on hardware that should make security easier instead. Once it is widely available, I expect more code will take advantage of it, and people who really care about security will want to buy hardware that works with it.

The Linux Foundation's "security mobilization plan"

Posted May 29, 2022 9:26 UTC (Sun) by geuder (subscriber, #62854) [Link]

For those who (like myself) don't remember what CHERI was again: https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/

ARM has deliverd a prototype. I guess it will take many years until we normal mortals working not in any leading edge project will see such stuff. If ever.

Not trying to suggest that rewriting software would necessarily be fast(er). After well over 10 years I still see more X11 than Wayland, although somewhat faster transitions do exist. Several approaches are probably a good thing in real life, only time will tell what wins.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 5:01 UTC (Thu) by pabs (subscriber, #43278) [Link]

> Does anyone know how likely this proposal is to get funded?

Sounds like the USA govt has plans in this area so probably would fund this.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 5:07 UTC (Thu) by pabs (subscriber, #43278) [Link] (10 responses)

> Eliminate Root Causes of Many Vulnerabilities Through Replacement of Non-Memory-Safe Languages

I wonder how the rewrites are going to be maintained. I expect there are lots of maintainers who aren't going to be interested/able to continue after the rewrite happens and the rewrite team aren't going to be interested in ongoing maintenance of the rewrite.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 7:04 UTC (Thu) by roc (subscriber, #30627) [Link]

That could be a problem. On the other hand there is also a different problem: aging maintainers need to be replaced and a lot of the newer generations of developers aren't enamoured of old-school C code.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 11:49 UTC (Thu) by tpm (subscriber, #56271) [Link] (8 responses)

> I wonder how the rewrites are going to be maintained. I expect there are lots of maintainers who aren't going to be interested/able to continue after the rewrite happens and the rewrite team aren't going to be interested in ongoing maintenance of the rewrite.

Throw-over-the-wall rewrites by third parties are unlikely to fly with existing maintainers, but I don't know if anyone is suggesting such a thing anyway?

There are plenty of maintainers of C/C++ libraries who are open to moving to memory safe languages and this is where the process could be accelerated massively with some funding.

Having said that, nowhere in the plan do I see a proposal for such funding (it would have to be more than a million or two per year, realistically, if you're targetting the top 10,000 dependencies), or a way for such maintainers to reach out and say 'Please consider my proposal'.

As the wider ecosystem moves towards memory safe languages, the dependency stack used will organically gravitate towards memory safe dependencies where possible anyway.

C/C++ libraries that don't adapt will over time simply fall out of use.

There are very few libraries that are so special or complex that they can't be reimplemented or haven't already been reimplemented to a significant part.

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 14:25 UTC (Fri) by bauermann (subscriber, #37575) [Link] (7 responses)

> Having said that, nowhere in the plan do I see a proposal for such funding (it would have to be more than a million or two per year, realistically, if you're targetting the top 10,000 dependencies), or a way for such maintainers to reach out and say 'Please consider my proposal'.

In the PDF there's a section called “Stream 4: Eliminate root causes of many vulnerabilities through
replacement of non-memory-safe languages.” and it says “Cost: $5.5M for the first year, $2M per year beyond.” and in appendix 4 which details stream 4 it says “Grants to examine, support best efforts, and port other critical software to Go and Java based on the same criteria as posed above: $2M” so my understanding is they do plan to fund it.

As for a way for maintainers to reach out, my understanding is that they have specific criteria to identify which codebases are most worthwhile to be rewritten so they seem to expect that they will do the reaching out to projects, not the other way around.

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 14:58 UTC (Fri) by tpm (subscriber, #56271) [Link] (6 responses)

Thanks, that's interesting. I misread that section initially.

It looks like they're basically going to give grants through other projects like ISRG, the Rust Foundation, etc.

I wonder if that's the best way to reach maintainers of existing C/C++ projects that are looking to migrate though (who might not feel enough part of the Rust community yet in order to apply for grants there).

The Linux Foundation's "security mobilization plan"

Posted May 28, 2022 1:02 UTC (Sat) by tialaramex (subscriber, #21167) [Link] (5 responses)

One thing that paying Rust developers could do for a C++ maintainer in particular is knock off pain points where there isn't yet a nice way to do what they're used to in Rust.

For example, Rust doesn't yet have explicit Trait specialisation. You can write "For any type T which is Copy and Default, here's an implementation of my Trait Foo" in Rust, and we do, but you can't write "Also, for Foo<bool> here's a much faster specialised way to implement it". Rust will complain that now there are two ways to implement Foo<bool> since bool is Copy and Default. Ideally we could tell Rust, "Yes, but the bool specialisation is more specific, so that's fine", but today this requires an unstable feature because it's not necessarily Sound to do this.

[I suspect this specifically is too hard for Linux Foundation money to get it done, but it's the sort of thing]

C++ programmers can write a specialisation easily because C++ doesn't care whether it's sound, if you broke the type system too bad. If your C++ codebase has several such specialisations, then Rust is a big problem, you can't just port what you did to Rust, you lose your (presumably performance critical) specialisations. You might well be confident that what you did is sound, and you're probably right, but opting in to an _unstable_ Rust feature is a harder sell for a C++ programmer who was told this is their route to safer code.

More const might be a smaller group of tasks in the same space. Today C++ has a lot of ability to perform compile-time constant expression evaluation. C++ code can allocate at compile time, do some stuff on the allocated RAM, and then return a constant value, on the rationale that so long as the allocation itself was gone by the time the evaluation finished we don't need to worry about it at runtime. You can have a program that logically needs 4MB of RAM to calculate, compile it on your normal workstation, but then run it on a 64kB micro controller, knowing actually the 4MB RAM overhead was all compile-time constant evaluation, at runtime 64kB is more than enough. Rust isn't there, it's much more capable of constant evaluation than it was five years ago, but for example Rust can't yet even do a for loop in a constant function, today that loop would need to be re-written even if it's obviously in fact constant.

The Linux Foundation's "security mobilization plan"

Posted May 28, 2022 10:11 UTC (Sat) by tpm (subscriber, #56271) [Link]

Thanks for the perspective. I'm sure you're right that there are cases where Rust isn't quite "there" yet to replace certain C++ features, and I didn't really mean to suggest otherwise, or start a language discussion.

As for helping knock off pain points like this in Rust, that sounds like a perfect case for the existing Rust Foundation grants programme.

The Linux Foundation's "security mobilization plan"

Posted May 29, 2022 23:32 UTC (Sun) by NYKevin (subscriber, #129325) [Link] (3 responses)

> You can have a program that logically needs 4MB of RAM to calculate, compile it on your normal workstation, but then run it on a 64kB micro controller, knowing actually the 4MB RAM overhead was all compile-time constant evaluation, at runtime 64kB is more than enough. Rust isn't there, it's much more capable of constant evaluation than it was five years ago, but for example Rust can't yet even do a for loop in a constant function, today that loop would need to be re-written even if it's obviously in fact constant.

You could work around this by writing two programs, one which does the computation and outputs its results, and one which consumes those results from an input file.

Is that the best design for your particular use case? 🤷‍♂️
Is it a design that you could use, if you were so inclined? Yes.

The Linux Foundation's "security mobilization plan"

Posted May 30, 2022 1:02 UTC (Mon) by excors (subscriber, #95769) [Link] (2 responses)

> You could work around this by writing two programs, one which does the computation and outputs its results, and one which consumes those results from an input file.

The output of the compile-time code might be more code, not just data. E.g. it might compile a regex into an efficient matching function (like https://github.com/hanickadot/compile-time-regular-expres...). In general it seems impractical to implement that as two separate programs, since the first program would need to be a preprocessor with a sophisticated code parser/generator - and at that point it's no longer a workaround, it's a full implementation of metaprogramming for your language (but harder because you're choosing to serialise the processed code to text and parse it again).

Maybe regexes aren't the best example, I think they're self-contained enough that you could do most of them with Rust's procedural macros (which usually operate over ASTs). But that C++ library also lets you split the pattern from the regex compilation, like "constexpr auto pattern = ctll::fixed_string{"..."}; ...; auto matcher = ctre::match<pattern>;", which I think is impossible with Rust macros because the AST can't tell you that the two 'pattern' identifiers are the same variable - most name resolution doesn't happen until after macro expansion. Other use cases for metaprogramming (maybe like Eigen - https://eigen.tuxfamily.org/dox/TopicInsideEigenExample.html) may depend more heavily on resolving names/types/etc from the surrounding code, so they couldn't work at all with Rust macros. And they can't work with generics or const fn which are far too limited in other ways.

The Linux Foundation's "security mobilization plan"

Posted May 30, 2022 1:55 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> The output of the compile-time code might be more code, not just data.

You can also do code-generation, like we do for RPC stubs and other similar applications. It might be too heavyweight for mere regexes, but still absolutely possible.

The Linux Foundation's "security mobilization plan"

Posted May 30, 2022 1:58 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Code generation during compilation is seriously underrated for some reasons, probably related to difficulty of integrating it with many build systems. But it's extremely powerful and at least for me it's usually easier to debug than compile-time macros.

For example, I have a code generator that calculates a seed to produce perfect hashes for a given set of strings (so you can do "switch" on hash values without collisions). Writing it in C++ compile-time expressions would have been... painful.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 11:51 UTC (Thu) by wtarreau (subscriber, #51152) [Link] (27 responses)

It's the usual same garbage. "70% of bugs are due to memory safety, hence we should avoid such languages so that in the future 70% of bugs are caused by logic error introduced by the need to work around the language's protections".

I'm writing MORE bugs since gcc emits tons of bogus warnings, because shutting them down requires tricks that are all but obvious and that are easier to get wrong than the original code itself. Worse, opensource development is collaborative, and everyone gets inspired by others. As such, the tricks used to work around a compiler's stubbornness are not understood but are happily copied by those who face the same issue. These ones can as well result in quite some vulnerabilities.

The real problem of C right now (and the Rust guys don't deny it) is the vast amount of UB. Just defining an UB-less variant of the standard would be quickly adopted, would allow to get rid of many ugly tricks, would allow compilers to emit much more accurate warnings and would eliminate many bugs.

And this is much cheaper than inventing new languages or porting 30 million lines of code to $language_of_the_year.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 12:20 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

> The real problem of C right now (and the Rust guys don't deny it) is the vast amount of UB. Just defining an UB-less variant of the standard would be quickly adopted

There have been several attempts at doing variants of C or coding standards to try to ease the transition (Misra C, Cyclone etc) and while atleast some have seen some limited adoption, I don't think they are sufficient. Otherwise, why hasn't it been done already in several decades.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 18:21 UTC (Thu) by mpr22 (subscriber, #60784) [Link] (1 responses)

MISRA C is a catalogue of forbidden constructs (some of which revolve around "this construct invokes UB according to the standard, so don't do it" and some of which revolve around "the well-defined semantics of this construct according to the standard are weird and gnarly and liable to shoot you in both feet through your spleen, so don't do it").

What wtarreau seems to propose is a C variant where all UB is transmuted into some combination of "the behaviour of this construct is implementation defined" and "this construct is forbidden". I imagine a conservative version of that looking like:

Signed integer overflow? Well, it can trap, or it can wrap, or it can saturate, and it can refuse to compile code that provably causes signed integer overflow, but the implementation must say what it does. Caveats like "unless someone has altered the CPU control registers behind the implementation's back" are taken as read.

Assigning the value of a double of great magnitude to a float? Well, it can trap, or store Inf, or store a qNaN, or store an sNaN, or do whatever the OS has configured the CPU to do, and it can refuse to compile code that provably does it, but the implementation has to say what it does. Caveats like "unless someone has altered the CPU control registers behind the implementation's back" are taken as read.

Read or write through a null pointer? Well, it can trap, or it can return the contents of the address corresponding to the bit pattern used for the stored representation of NULL, or whatever, but the implementation must say what it does. ("What the hardware and OS decide to do when you dereference the address represented by the all-zeroes bit pattern" is an acceptable answer.)

memcpy() between overlapping regions? It can copy forward, or backward, or from the middle out, or from the outside in, and it can refuse to compile code that provably does it, but whatever the result, the implementation has to document what it does.

Invoking a function without having a prototype in scope is forbidden.

Defining a function without a prototype is forbidden.

If two pointers have fully identical bit patterns, they refer to the same object.

etc.

The Linux Foundation's "security mobilization plan"

Posted May 28, 2022 8:59 UTC (Sat) by tialaramex (subscriber, #21167) [Link]

MISRA also has two other things here:

First it has places where it requires certain constructions that the authors believe (in some cases with good justification from studies showing this works, in others it seems like just a weird style preference) are safer or encourage safer programming.

For example MISRA insists you use C's switch statement to write exhaustive and irrefutable matches. You must write break after each case clause, and you must provide a default case and these are to be presented in a particular order.

Second it has a bunch of unenforceable rules about engineering practice. For example your code should be documented and tested. C doesn't actually provide you with any help actually doing this, but MISRA does at least tell you that you should do it.

I'm sure MISRA or anything like it would make wtarreau miserable while providing very little benefit.

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 20:00 UTC (Thu) by roc (subscriber, #30627) [Link] (22 responses)

People have considered defining "less undefined" C with a compiler to match --- John Regehr for example --- but you immediately run into the problem that no-one can agree on what the language should be. For every instance of undefined behavior in C, someone thinks it's vital that remain undefined (e.g. because it enables compiler optimizations that matter to them). That's probably why no-one has done this even though it's been obvious for a long time that it could be useful.

Logic errors introduced by working around gcc warnings don't predict similar errors will occur in a language like Rust. Of course patching up a broken language like C requires tradeoffs that are unnecessary in languages that are better designed from the start.

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 11:39 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (21 responses)

> you immediately run into the problem that no-one can agree on what the language should be

That's exactly why I think it must not be done via a committee nor by seeking a consensus, but it must
be done exactly the same way the problems were introduced in the first place: purely centralized and imposed. Most of the choices would either follow what any relevant hardware does (e.g. integer overflows just wrap) or what costs the least (e.g. deciding whether a shift by nbits gives zero or nop, let's just compare output code for both cases on x86 and ARM and pick the cheapest one). Integer promotion is a total mess (how come an unsigned char be promoted to signed int?), and will need to be more intuitive (probably by respecting signedness first), etc. As long as the choices follow what some implementations already do, only already broken code would require adaptation, and that broken code is currently getting sabottaged by gcc and clang anyway.

The difficulty would be to make projects like gcc or clang accept to merge a patch implementing such a standard variant, mostly because it would ruin their efforts that consist in eliminating non-perfectly compliant code for the sole purpose of showing higher numbers than the competitor :-(

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 12:59 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

I think this way would lead to incompatibilities when using different compilers. If all you can say is "what we did for this other compiler" when trying to update the behavior of another, your test suite is going to be bonkers.

IMO, the better path would to get compilers to make noise when they do things that rely on UB to justify the transformation (I mean, you're looking to change the behavior in these cases anyways…). If this means that every `if (ptr)` check in a macro makes noise because it is expanded after a `*ptr` usage in a function, then so be it…maybe you should use fewer macros ;) .

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 23:35 UTC (Fri) by roc (subscriber, #30627) [Link]

You have to get the gcc and LLVM maintainers to agree, otherwise you can't "impose" anything. That's already a significant number of opinionated people. Then, if your new language dialect has to be opted into by developers (which it will, because otherwise you're forcing every project to take performance regressions), there is a high risk that developers won't want to opt into it because their desired set of UBs does not match yours.

Again, people have been talking about this problem for a long time. No-one has done anything about it. I think that's because it's incredibly hard and has a low likelihood of success. If you think it's not hard, it may be up to you to do something about it.

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 23:38 UTC (Fri) by roc (subscriber, #30627) [Link] (13 responses)

Also, you said you wanted your C dialect to be "UB-less". That's actually not practical since it means, for example, preventing use-after-free bugs and buffer overflows. You can't just "define the behavior" to make those UBs go away.

The Linux Foundation's "security mobilization plan"

Posted May 28, 2022 9:57 UTC (Sat) by tialaramex (subscriber, #21167) [Link] (1 responses)

I think you actually can pull this off, it's just a miserable result quite unlike what C programmers are used to, and so I'm sure once somebody did all the work such programmers would say "I hate it" and go back to moaning about the Undefined Behaviour in C.

You could define the language as-if there's really just this huge fixed memory space and the allocator assigns you pieces of that space, but you can just ignore it whenever you like and directly index into the space. The allocator probably has to be defined in a pretty suboptimal way to have this work, and obviously your runtime performance is not good, but hey, no more Undefined Behaviour.

For example in this model use-after-free does what they "expect", if the allocator didn't give this memory to anybody else yet it just works. Buffer overflows likewise, if the vector of 32-bit integers is right next to the output string, overflowing it just scribbles on the string. Everything has this defined (but often horrible) behaviour in exchange for giving up performance.

All the cases where an object can have invalid representation need resolving (because now it can be accessed via a pointer with the wrong type), again I think the way to do this is sacrifice performance. For example probably we want to just carve off some pointers as invalid, we can insert an arithmetic check for every pointer dereference to accomplish this. For enumerated types we explicitly define that all the invalid representations are fine, this sort of C programmer always thought of enumerated types as just integers anyway so they'd probably be astonished it matters. Why shouldn't day_of_week be 46 sometimes instead of WEDNESDAY?

The Linux Foundation's "security mobilization plan"

Posted May 28, 2022 21:33 UTC (Sat) by roc (subscriber, #30627) [Link]

I think for this to """work""" you would need to define the interpretation of every bit pattern in the address space, which would be impossible to do in a portable way.

The Linux Foundation's "security mobilization plan"

Posted May 30, 2022 21:56 UTC (Mon) by Wol (subscriber, #4433) [Link] (10 responses)

> Also, you said you wanted your C dialect to be "UB-less". That's actually not practical since it means, for example, preventing use-after-free bugs and buffer overflows. You can't just "define the behavior" to make those UBs go away.

That again is easy to define. That memory does not belong to the program. Therefore it's a bug, and the consequences of executing faulty code are undefineable, not undefined.

The whole point of getting rid of UB is to enable the programmer to reason about the code, and draw reasonable inferences therefrom. A bug is asking a program to perform an action which cannot be reasoned about. As soon as you define that, your UB becomes "undefineable behaviour", and outside the scope of a valid program.

You need to take the attitude that "if it CAN be defined, it MUST be defined. If it CAN'T be defined, then any program attempting it is invalid".

Cheers,
Wol

The Linux Foundation's "security mobilization plan"

Posted May 31, 2022 13:08 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (8 responses)

> Therefore it's a bug, and the consequences of executing faulty code are undefineable, not undefined.

Could you please be specific what you mean here? What is the difference? What should the compiler do if it detects "undefineable" behavior at compile time? If the compiler sees this, you have an invalid program (per your words), so what's wrong with the compiler saying "assume that doesn't happen" (because you can do anything from such a situation) and then optimizing based on it? At that point…what's the difference?

The Linux Foundation's "security mobilization plan"

Posted Jun 2, 2022 12:41 UTC (Thu) by Wol (subscriber, #4433) [Link] (7 responses)

Let's rephrase that then. "The behaviour is non-deterministic". In which case, you are either writing a statistical program in which case the compiler should know that non-derministic behaviour is acceptable, OR the compiler has detected a BUG in which case it should fail to produce an executable! After all, isn't that pretty much what ALL compilers apart from C do?

At the end of the day, UB is a *BUG* waiting to bite people, and when statements like "if I detect an access to *freed_ptr it's UB so I can format the hard drive if I like" are literally true, it shows up the C spec as the farce it really is.

C was (and is still seen as) a glorified assembler. It should not be deleting safety checks because "this isn't supposed to happen" when the programmer has determined it can - and does - happen.

I'm quite happy with the C spec offloading responsibility - 6-bit bytes and all that. I'm quite happy with the spec saying "this shouldn't be possible - here be dragons". The problem is when compiler writers spot UB in the spec whether by accident or looking for it, and then CHANGE the behaviour of the compiler.

If the C spec is changed to say "there is no such thing as UB, it is now as a minimum Implementation-Defined, even if the IDB is simply "Don't do it, it's not valid C"", then it will stop C being a quicksand of nasty surprises every time the compiler is upgraded.

Otherwise we will soon end up (and quite possibly a lot quicker than people think possible) where the core of the Linux kernel is written in Rust, and the only C left will be legacy drivers.

Cheers,
Wol

The Linux Foundation's "security mobilization plan"

Posted Jun 2, 2022 14:04 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

> "The behaviour is non-deterministic". In which case, you are either writing a statistical program in which case the compiler should know that non-derministic behaviour is acceptable, OR the compiler has detected a BUG in which case it should fail to produce an executable!

The problem is the third case: the behavior *may* be deterministic or non-deterministic but the compiler doesn't have enough information to be sure. It may depend on other parts of the program, such as separately-compiled library code, or on runtime inputs. It's not necessarily unreasonable to say that every program (every translation unit) must be provably well-defined regardless of the input it receives or what any other part of the program might do—some languages do take that approach—but the result would not look much like C, and likely would not be suitable for system-level programming due to dependencies on platform-specific behavior the compiler knows nothing about. Most, if not all, currently valid C programs would fail to compile under those rules.

When the behavior of a piece of code is known to *always* be undefined the compiler generally produces a warning, which you can escalate into an error (with -Werror) if you so choose.

The Linux Foundation's "security mobilization plan"

Posted Jun 2, 2022 14:36 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (5 responses)

> when statements like "if I detect an access to *freed_ptr it's UB so I can format the hard drive if I like" are literally true, it shows up the C spec as the farce it really is.

Note that no compiler (anyone wants to use) will actually insert such code on its own. But, if you have code in your program that does that, there is nothing saying that it won't be called by some consequence of the UB (say it flips a bit in a function pointer or jump table). I mean, I guess you could make a C compiler that is absolutely paranoid and actually believes that *freed_ptr can happen at any time and therefore inject pointer verification code around all pointer uses, but I suspect that you then lose the "glorified assembly" property because you have all this invisible sanity checking.

The rules for UB are there because they are assumptions that need to be there to reasonably contemplate what code does in any instance. If race conditions exist, every read is a hazard because any thread could write to it behind your back (meaning that you need to load from memory on every access instead of hoisting outside the loop). If free'd pointers are usable in any way, any new allocation has to be considered potentially aliased (again, pessimizing access optimizations). If integer overflow is defined, you cannot promote small integers to bigger ones if the instructions are faster because you need to get some specific answer instead of "whatever the hardware does". Or you need to always promote even if the smaller code would have offset the slower instruction because it all fits in the instruction cache now.

Saying that these behaviors are UB and that, if they happen, all bets are off allows you to reason locally about some code. What you want may be possible, but I don't think it looks like what you think it would look like. I mean, I guess you could just say C "translates to the hardware", but that means that you have zero optimizations available because you cannot say "this code acts as-if this other code" because that addition is inside the loop, so it needs to be done on every iteration instead of factored out and done with a simple multiply.

The Linux Foundation's "security mobilization plan"

Posted Jun 2, 2022 18:06 UTC (Thu) by mpr22 (subscriber, #60784) [Link] (4 responses)

> If integer overflow is defined, you cannot promote small integers to bigger ones if the instructions are faster because you need to get some specific answer instead of "whatever the hardware does".

This is an interesting one, because whether that's true is a function of exactly what calculation you're trying to perform, what the machine's behaviour on overflow is, and what results you're trying to extract from it.

The Linux Foundation's "security mobilization plan"

Posted Jun 3, 2022 14:46 UTC (Fri) by Wol (subscriber, #4433) [Link] (3 responses)

I guess this is where the C spec should say "x86_64 follows IEEEwotsit rules, therefore when compiling for x86_64 that's what you get".

Thing is, most UB has an obvious outcome. Most UB was because different compilers did different, apparently sensible, things. So where we can, let's codify what it does, even if we have conflicting behaviour between compilers.

And if there isn't a sensible, codifiable behaviour (like accessing *freed_ptr), we simply call it out as something that - if it happens - it's a bug.

In practice, pretty much everything that happens now is "whatever x86_64 does" now, anyway, so let's codify that as the default. If you're compiling for a ones-complement CPU, you know that the hardware is going to behave "strangely" (to modern eyes, at least), and any compiler for that architecture should default to ones-complement.

If you're shifting code between architectures, you expect behaviour to change. You should not get nasty surprises when the only thing that's changed is the compiler version.

Cheers,
Wol

The Linux Foundation's "security mobilization plan"

Posted Jun 3, 2022 19:17 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (2 responses)

> So where we can, let's codify what it does, even if we have conflicting behaviour between compilers.

Much to the amusement of C developers fielding questions from others using different compilers I'm sure. At least today we can say "C doesn't let anyone do that" even if different compilers do different things in response to the code; there's a common base to code against (the C abstract machine). But now with this proposal, one could say "GCC-C works here, use a better compiler".

I'm also sure all of the embedded developers will enjoy having x86-isms forced upon them for their resource-constrained devices instead of considering that maybe *their* behavior should be preferred given that beefy x86 machines tend to have the horsepower to do something special.

The Linux Foundation's "security mobilization plan"

Posted Jun 4, 2022 10:22 UTC (Sat) by Wol (subscriber, #4433) [Link] (1 responses)

> I'm also sure all of the embedded developers will enjoy having x86-isms forced upon them for their resource-constrained devices instead of considering that maybe *their* behavior should be preferred given that beefy x86 machines tend to have the horsepower to do something special.

So we have switches that say "I want the hardware-defined behaviour, not the C default behaviour". Which I said should be the case several posts ago!

As others have said, the problem with UB is that - GCC IN PARTICULAR - they take advantage of UB to change the compiler behaviour without warning.

And I completely fail to see why requiring compiler developers to DOCUMENT CURRENT BEHAVIOUR and PROVIDE SWITCHES IF BEHAVIOUR CHANGES is going to force x86isms onto embedded developers? Surely it's going to HELP because they WON'T have x86isms stealth-thrust on them!

Cheers,
Wol

The Linux Foundation's "security mobilization plan"

Posted Jun 6, 2022 14:33 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

> they take advantage of UB to change the compiler behaviour without warning.

I'm all for better warning emission, but I think the number of false positives would be overwhelming (unless you truly never use macros).

> And I completely fail to see why requiring compiler developers to DOCUMENT CURRENT BEHAVIOUR

You're talking about emergent behavior here. No one consciously said "let's trash user code". People wrote optimization transformations that are valid given code that follows C's rules. When putting these together, you can end up making mush of code that wasn't valid to begin with. What optimization rule should have this detection attached to it?

I mean, I guess you could just use `-O0` since that seems to be what you want anyways: take the code I wrote, as I wrote it, and output instructions for the CPU. But I suspect you'll be sad at how much work *you'll* now have to do to get better performing code (not that it matters in a lot of cases, but when it does, it does).

The Linux Foundation's "security mobilization plan"

Posted May 31, 2022 19:15 UTC (Tue) by roc (subscriber, #30627) [Link]

Your "undefinable behavior" is "undefined behavior" by another name. It would work exactly the same way in practice: "if your program does this, it's invalid and there are no guarantees at all about what will happen".

Claiming to have eliminated undefined behavior by defining some of it and choosing another name for the rest would be rather misleading.

The Linux Foundation's "security mobilization plan"

Posted May 30, 2022 21:44 UTC (Mon) by Wol (subscriber, #4433) [Link] (4 responses)

> > you immediately run into the problem that no-one can agree on what the language should be

> That's exactly why I think it must not be done via a committee nor by seeking a consensus, but it must be done exactly the same way the problems were introduced in the first place: purely centralized and imposed.

Or you simply require that each compiler must declare what it does, and provide a switch to enable other options, possibly with a majority vote of the committee saying which options must be provided.

That way, UB becomes "hardware defined" or "OS defined" or "implementation defined", but UNdefined becomes forbidden by the standard. And if a new version of the standard defines the "majority behaviour" as the new standard (or even if it doesn't), then ALL compilers MUST provide a switch to enable old behaviour whenever a compiler changes its default.

Cheers,
Wol

The Linux Foundation's "security mobilization plan"

Posted May 31, 2022 13:06 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (3 responses)

> Or you simply require that each compiler must declare what it does, and provide a switch to enable other options, possibly with a majority vote of the committee saying which options must be provided.

> And if a new version of the standard defines the "majority behaviour" as the new standard (or even if it doesn't), then ALL compilers MUST provide a switch to enable old behaviour whenever a compiler changes its default.

Do you know how many C compilers were found by the committee some time ago? It's in the hundreds. How do you propose to get a majority of these to declare any meaningful behavior of `*freed_ptr` other than "good luck" without some (probably serious) performance impairments?

The Linux Foundation's "security mobilization plan"

Posted Jun 2, 2022 12:19 UTC (Thu) by Wol (subscriber, #4433) [Link] (2 responses)

> Do you know how many C compilers were found by the committee some time ago? It's in the hundreds. How do you propose to get a majority of these to declare any meaningful behavior of `*freed_ptr` other than "good luck" without some (probably serious) performance impairments?

Well, I think the following definition would work pretty well ... "accessing *freed_ptr accesses memory that no longer belongs to the program, with potentially disastrous consequences". Isn't that good enough? It's pretty accurate!

Likewise "the result of accessing *null_ptr either behaves as defined by the hardware/OS, or must be defined by the compiler".

If you take the attitude "it's too big a problem so I'm not going to try" then of course you're going to fail. All *I'M* asking is to take the attitude "let's make a start and see how far we get". And as a first step, simply saying "there will be no such thing as undefined behaviour in the C spec, all behaviour MUST be DOCUMENTED", will get us a long way. Even if the C spec says "whatever the hardware does", at least we have a reference document so the programmers know what to *expect*. (And if C *is* a glorified assembler, as many people think of it, then "whatever the hardware does" is indeed a good, and valid, definition).

And while *freed_ptr is an easy example of "you shouldn't do this" there's a lot more less obvious examples. At least if the C spec says "you must document what you do, and provide flags if you change it", then it's going to reign in pretty much all the compiler writers doing stupid things. And then if the compiler users start saying "what the brass monkeys do you think you're doing!!!" we might start getting compilers that actually stop shooting their users in the feet.

Cheers,
Wol

The Linux Foundation's "security mobilization plan"

Posted Jun 2, 2022 13:53 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

> "accessing *freed_ptr accesses memory that no longer belongs to the program, with potentially disastrous consequences". Isn't that good enough?

int *const p = (int*)malloc(sizeof(int));
free(p);
int *const q = (int*)malloc(sizeof(int));
*q = 5;
*p = 7; /* use after free */
if (*q != 5) launch_missiles();

Say the second malloc() happens to return the same address as the first, which is perfectly valid since that memory was freed. The memory _does_ belong to the program, but now it's part of a different object. The consequences of writing to it cannot be defined by the hardware or the compiler as they depend on the rest of the program, implementation details of the C library, and the particular circumstances present at runtime (e.g. other threads allocating or freeing memory).

I'll grant you "potentially disastrous consequences"—but that's just another way of saying "undefined behavior".

> Likewise "the result of accessing *null_ptr either behaves as defined by the hardware/OS, or must be defined by the compiler".

This is a slightly more reasonable request, since it can be accomplished portably by inserting checks for null pointers at every point where a potentially-null pointer might be dereferenced. Of course this comes at a considerable cost in performance in situations where there is no reliable hardware trap for dereferencing a null pointer, such as most systems without MMUs. If you're doing away with the strict aliasing rules as well then you'll need even more checks since previously valid pointers could be changed to null pointers as a side effect of seemingly unrelated memory accesses.

The Linux Foundation's "security mobilization plan"

Posted Jun 16, 2022 15:02 UTC (Thu) by nix (subscriber, #2304) [Link]

> And as a first step, simply saying "there will be no such thing as undefined behaviour in the C spec, all behaviour MUST be DOCUMENTED", will get us a long way.

You don't seem to realise what "undefined behaviour" means. Undefined behaviour is simply any behaviour that has no definition within the standard. The set of such behaviours is, of course, almost infinite: almost every C implementation has at least some things that go beyond the C standard. Any call to a library function not defined within the standard is undefined behaviour, so as soon as you have separate compilation you are doing a lot of stuff that (from the perspective of a compiler running over some other translation unit) is not defined (which is why it has to assume that e.g. arbitrary things can happen to memory across function calls). It is quite possible that such functions are not written in C at all.

POSIX is similarly undefined: the C standard does not define it (though POSIX is more or less a superset of it and includes the entire C standard by reference, this is certainly not true of *everything* that impacts systems that run C code but is actually rather rare).

In places, the standard does specifically call out that some behaviours are not defined in addition to simply not defining them, but even if you changed all those places to explicitly nail down a definition, there would still be countless places *with no corresponding text in the standard* where some element of the behaviour of a system running C code was not defined in the standard (as you'd expect when there was no text there). A good thing too, or C wouldn't be much use anywhere.

The Linux Foundation's "security mobilization plan"

Posted May 29, 2022 23:36 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

A UB-less variant of C is memory safe, because dereferencing an invalid pointer invokes UB, and there is no sensible alternative definition you could come up with that still retains an ounce of C's portability (e.g. specifying the exact behavior of malloc so you can statically prove what happens to a use-after-free bug) and isn't hilariously inefficient (e.g. bounds-checking all pointers at runtime like Java does). The "UB-less variant of C" that you describe is, in fact, Rust (or at least something similar to Rust).

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 4:38 UTC (Thu) by pabs (subscriber, #43278) [Link]

Here is the report if you don't want to go through their tracking redirect:

https://8112310.fs1.hubspotusercontent-na1.net/hubfs/8112...

The Linux Foundation's "security mobilization plan"

Posted May 26, 2022 16:34 UTC (Thu) by jmclnx (guest, #72456) [Link] (1 responses)

I wonder how the rule "do not break user land" will be effected ?

The Linux Foundation's "security mobilization plan"

Posted May 27, 2022 23:46 UTC (Fri) by deepfire (guest, #26138) [Link]

Excellent question -- I don't personally believe keeping a reasonable backward compat is on the table.


Copyright © 2022, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds