A GCC -fstack-protector vulnerability on arm64

(Even if the program is processing trusted input - e.g., it's part of something like 'make' whose purpose is running code anyway, and so there cannot really be security vulnerabilities - there still isn't a point in having it conditionally overrun the stack and crash. Just detect when the inputs are too big and conditionally throw an error at the beginning of the program. The effect for the end user is no worse, and probably a bit better really.)

This scenario only makes sense, I think, if you can somehow guarantee that when A makes a large VLA, B and C definitely will not, etc. But I'm having trouble thinking of how you'd end up with code like that. Most of the time, if you are processing large input in one function and call another, that second function is going to also process large input too, or at best process data of constant size. It isn't going to get smaller.

Maybe your logic sometimes does lots of work in B, and sometimes lots of work in C instead, but only in one or the other? But you can solve that by just creating a stack array in B (or A) and passing a pointer to it down to C, instead of doing another allocation in C. Pointers to stack variables remain valid as long you're somewhere deeper on the stack.

In a multi-threaded process, stack space of any thread apart from the main one is limited because the stack has to be mmapped at a particular virtual address upon thread creation and cannot move.

I guessed that might cause problems with setjmp/longjmp, which will restore the stack pointer (effectively undoing any stack allocations) but won't know how to deallocate any automatic heap allocations. But it turns out the C standard already says that any VLAs allocated between the setjmp and longjmp may be leaked; apparently the original Cray Research implementation of VLAs used the heap as a fallback when it ran out of stack space (https://www.open-std.org/jtc1/sc22/wg14/www/docs/n317.pdf). So it's okay to use the heap and just tell programmers not to combine VLAs and longjmp.

If you're developing for a single-application embedded environment, then I think it often *is* a good idea to calculate your worst-case heap usage and statically allocate that, so you can be sure the application will meet its specification and won't crash from resource exhaustion when given a valid input. Limited dynamicity can be done with statically-sized pool allocators, where higher-level code allocates a whole complex data structure and allocation failure can either be prevented (e.g. by verifying the resource requirements of a request before accepting it, or applying backpressure to a message queue before you get overloaded, etc) or handled gracefully (unwinding the operation and returning a meaningful error to the user), in contrast to a global heap which might fail in any of your many thousands of low-level std::vector/etc operations where it's practically impossible to recover except by crashing and restarting the whole application.

And a typical system now has gigabytes of RAM.