GNU C Library 2.40 released [LWN.net]

Version 2.40 of the GNU C Library has been released. Changes include partial support for the ISO C23 standard, a new tunable for the testing of setuid programs, improved 64-bit Arm vector support, and a handful of security fixes. See the release notes for details.

to post comments

C23 reference links

Posted Jul 23, 2024 14:42 UTC (Tue) by dveeden (subscriber, #120424) [Link] (42 responses)

C23 reference links

Posted Jul 23, 2024 15:01 UTC (Tue) by fishface60 (subscriber, #88700) [Link] (41 responses)

Thank you for the links.

> # Removed
> * Support for calling realloc() with zero size (the behavior becomes undefined)

Shame about this one. A NULL pointer and 0 size are natural and ergonomic sentinel values.
Is there somewhere I can read more about why this change was made?

C23 reference links

Posted Jul 23, 2024 15:29 UTC (Tue) by pbonzini (subscriber, #60935) [Link]

The main problem is the conflict between realloc(NULL, n), which arguably should be the same as malloc(n), and realloc(ptr, 0).

BSD never freed for realloc(ptr, 0) even if ptr is NULL. AIX never allocated for realloc(NULL, n) even if n is zero. glibc allocated for realloc(NULL, 0), like malloc(0) does, but if n!=0 it freed the pointer.

For more information see https://www.open-std.org/jtc1/sc22/wg14/www/docs/summary....

Here is where the final change was made: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2464.pdf

C23 reference links

Posted Jul 23, 2024 15:29 UTC (Tue) by mb (subscriber, #50428) [Link] (29 responses)

It was implemented differently across platforms, so portable code could not use it anyway:

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2464.pdf

C23 reference links

Posted Jul 23, 2024 21:15 UTC (Tue) by khim (subscriber, #9252) [Link] (28 responses)

So what? Pick some choice and make it the proper way to work for C23 and declare all other compilers non-compliant.

Even if platform doesn't want to make an effort to become C23-compliant it may just declare that it have some implementation difference (most platforms already have some).

Making it undefined behavior just turns the whole thing into a minefield without giving anyone any tangible benefit.

But then again, if the goal is to never have any discussions between language developers and language users and to never change anything anywhere then declaring it UB and punishing the developers instead of anyone else is the only sensible choice.

C23 reference links

Posted Jul 23, 2024 21:22 UTC (Tue) by mb (subscriber, #50428) [Link] (23 responses)

So, break all platforms, for what benefit?

C23 reference links

Posted Jul 23, 2024 22:19 UTC (Tue) by khim (subscriber, #9252) [Link] (22 responses)

To be able to write cross-platform programs, obviously. Or to write any programs actually. Language with hundreds of undetectable UBs is basically unusable, because any UB forgotten by developer may lead to unpredictable and undebuggable consequence.

Isn't that why C exists in the first place?

And there are no need to break anything, there are plenty of ways to keep realloc with old behavior for code not comiled with -std=c23 flag, adding few ifdef lines to a header is not a rocket science.

Heck, even something like what Rust does for Mutex ( The exact behavior on locking a mutex in the thread which already holds the lock is left unspecified. However, this function will not return on the second call (it might panic or deadlock, for example)) is better than declaring something UB.

UB is, essentially, a permission to take any program where it happens and turn it into a pile of goo… why would you want to do that to realloc?

C23 reference links

Posted Jul 24, 2024 5:52 UTC (Wed) by mb (subscriber, #50428) [Link] (21 responses)

>To be able to write cross-platform programs, obviously.

You could never use zero-sized realloc in cross-platform code, because all platforms behave differently.
Just don't use zero-sized realloc and you get portable code.
Before and after the C23 change.

>UB is, essentially, a permission to take any program where it happens and turn it into a pile of goo…

Unless the platform defines it. Which at least Posix does.

C23 reference links

Posted Jul 24, 2024 8:37 UTC (Wed) by khim (subscriber, #9252) [Link] (9 responses)

> You could never use zero-sized realloc in cross-platform code, because all platforms behave differently.

Why should they behave differently? C23 may easily specify one way that is supported and make others behave in the same way.

> Just don't use zero-sized realloc and you get portable code.

Just don't ever do any mistakes, never vioplated any of hundreds of rules that standard specifies and everything would just be peachy? We tried that for half-century. It doesn't work. Humans are not made to work that way!

> Unless the platform defines it. Which at least Posix does.

Nope. It doesn't do that. POSIX, these days, very explicitly says this volume of POSIX.1-2017 defers to the ISO C standard.

Which means that yes, eventually, POSIX would be redefined to ensure that compiler can break user's programs willy-nilly.

We have precedent, after all. Originally POSIX was saying that realloc was changing the size of the memory object, which, naturally, would leave pointers valid if it's not moving them.

After a few years that note that made these programs valid was still there, but text that says that this volume of IEEE Std 1003.1-2001 defers to the ISO C standard was added and compilers started breaking valid programs.

Years later, an actual text of POSIX was changed to ensure that program-breaking compilers are not invalid.

Given that history I assert that C23 intentionally and explicitly changed the standard to break previously valid programs.

Including, yes, the ability to use 0 size on POSIX. Note that POSIX defers to the ISO C standard was enough to justify one kind of breakage WRT realloc-related code, why wouldn't it be enough to justify another kind?

C23 reference links

Posted Jul 24, 2024 16:56 UTC (Wed) by mb (subscriber, #50428) [Link] (8 responses)

>Why should they behave differently?

They *do* behave differently. That's what the reality looks like.

>C23 may easily specify one way that is supported and make others behave in the same way.

Changing the behavior would very likely break existing non-portable programs.

There are only two realistic options: Make it implementation defined or undefined.

C23 reference links

Posted Jul 24, 2024 17:18 UTC (Wed) by khim (subscriber, #9252) [Link] (7 responses)

> Changing the behavior would very likely break existing non-portable programs.

None of these programs are C23-compliant, anyway, so that's a moot point.

> There are only two realistic options: Make it implementation defined or undefined.

Or keep it like it was in C17. If developers may cope with two decades of non-decision for important matters then why couldn't they live without sane resolution for some fringe and not too interesting case?

And between bad, worse and worst choices… worst was picked — why exactly?

It's really one of the craziest cases of “we have to do something… this is something… then let's do it!”

C17 definition of realloc(_, 0)

Posted Jul 24, 2024 19:42 UTC (Wed) by farnz (subscriber, #17727) [Link] (6 responses)

Keeping it like it was in C17 is making it undefined behaviour, just with obfuscated language to hide that from people who don't spend all their time following through the implications of standardese. If the C17 definition was OK, then simplifying the language used to define it but keeping the definition the same should also be OK.

It would have been better to redraft the C17 language so that you couldn't select the set of options that make it UB, but that's not what the committee chose to do - in large part (as far as I can tell as a non-attendee) because any attempt to change the allowed set of options upset people who felt that other people were Wrong to implement realloc(_, 0) the way they did. Rather than continue trying to find some way to make it work for two groups who refuse to agree, the wording just got simplified to remove the long chain of logic leading to it being UB, since the two conflicting groups can just as easily define it downstream if they care.

C17 definition of realloc(_, 0)

Posted Jul 24, 2024 20:48 UTC (Wed) by khim (subscriber, #9252) [Link] (5 responses)

> Keeping it like it was in C17 is making it undefined behaviour, just with obfuscated language to hide that from people who don't spend all their time following through the implications of standardese.

Tell me what gives compiler writer the right to turn this function:

int foo(int x) {
  int y = x * 2;
  realloc(malloc(1), 0);
  return y;
}

into empty sequence of instructions (without even ret at the end). We may go from there.

> If the C17 definition was OK, then simplifying the language used to define it but keeping the definition the same should also be OK.

Only if you explain how the optimization described above is allowed. I coudn't see how C17 may allow that. C23, most definitely, makes it Ok.

> It would have been better to redraft the C17 language so that you couldn't select the set of options that make it UB, but that's not what the committee chose to do - in large part (as far as I can tell as a non-attendee) because any attempt to change the allowed set of options upset people who felt that other people were Wrong to implement realloc(_, 0) the way they did.

And, instead, they have picked an approach which would make every single one of these people ask about sanity of people who pushed this approach to the language. Is it really an improvement?

> Rather than continue trying to find some way to make it work for two groups who refuse to agree, the wording just got simplified to remove the long chain of logic leading to it being UB, since the two conflicting groups can just as easily define it downstream if they care.

Except we already know where that leads: no one would bother to do anything, but compiler writers, few years down the road, would interpret the wording in the most ruinous way possible.

That decisions would be funny if not for the past experience with this exact function.

realloc(malloc(1), 0) as UB under C17 rules

Posted Jul 24, 2024 22:40 UTC (Wed) by farnz (subscriber, #17727) [Link] (4 responses)

It's a nasty little mistake in drafting, in paragraph 3 of 7.22.3.5, and clearly a defect. However, the standard only imposes requirements on realloc if memory for the new object is not allocated. It is permissible for realloc to allocate a new object, but not to allocate memory for that new object (since it's zero sized); at this point, the general escape hatch in 3.4.3 is open, since the standard imposes no requirements here, and you've got UB on a technicality. It then returns a non-null pointer, since it allocated an object, and you're in the bad place since the standard doesn't say what to do with the old object.

Now, this is clearly a drafting error and should be fixed by removing "memory" - but, AFAICT, people got angry at the idea of fixing the drafting error without also fixing the required behaviour to match their idea of the "only correct" way to define this. And rather than deal with that, the standards committee chose to simply open the escape hatch wide and say "screw you all".

I personally think this was the wrong decision, but it's the decision the committee made. And it's reflective of the current direction of travel of C - rather than make hard decisions for the benefit of users of the language (since the smaller the area covered by undefined behaviour, the easier it is to work within the language), they've chosen to punt a hard problem at the users and rely on "real programmers not making mistakes". I doubt history will be kind to the C committee over this, since it's clearly a case of "it's easier for us to do this, even if it makes life harder for language users".

realloc(malloc(1), 0) as UB under C17 rules

Posted Jul 25, 2024 8:42 UTC (Thu) by khim (subscriber, #9252) [Link] (3 responses)

> It is permissible for realloc to allocate a new object, but not to allocate memory for that new object (since it's zero sized)

Objects is a “region of data storage in the execution environment”, if you don't have a storage then you can't have an object.

It's unclear whether zero-sized objects are even possible in C17 (it makes it impossible to have zero-sized arrays or structs, but doesn't say if zero-sized objects are possible to get from realloc), but I don't see how the definition of object as “regions of data storage in the execution environment” permits you to allocate one without also allocating memory (maybe zero bytes of memory!) for it. Sorry. No “region of data storage in the execution environment”, no object. It's as simple as that.

The worst you can expect from C17 AFAICS is that realloc would allocate one, single, zero-sized region of storage for objects and would hand over different pointers to that same single region with all-identical bits.

That's a bit of strange situation, sure, but it's still not UB. In Rust that's perfectly normal, defined (and often used!) behavior, I don't see how and why C should be different.

realloc(malloc(1), 0) as UB under C17 rules

Posted Jul 25, 2024 15:11 UTC (Thu) by farnz (subscriber, #17727) [Link] (2 responses)

C is different because the standard says that unless it explicitly defines something, that thing is UB. And in this case, by talking about memory in this one clause (something which, unlike a region of storage, is undefined), it infects the whole clause with UB if you're sufficiently motivated to find UB in a program.

If they'd not thrust that word "memory" in, this would be a full definition; but by adding an undefined term, the licence to treat the whole clause as UB is introduced.

realloc(malloc(1), 0) as UB under C17 rules

Posted Jul 25, 2024 15:37 UTC (Thu) by khim (subscriber, #9252) [Link] (1 responses)

> C is different because the standard says that unless it explicitly defines something, that thing is UB.

Yes, but that doesn't include terms. For unknown terms it defers to ISO/IEC 2382.

> And in this case, by talking about memory in this one clause (something which, unlike a region of storage, is undefined)

Are you sure it's not defined? I suspect it's common enough term than ISO/IEC 2382, somehow.

> it infects the whole clause with UB if you're sufficiently motivated to find UB in a program.

No, unknown terms like that. They may be interpreted differently, because not all combinations of ISO standard terms have meaning, but that would be a defect in the standard, it wouldn't, suddenly, lead to UB.

For something to be UB it shouldn't be defined at all, not defined in a strange and/or undeciperable manner.

Otherwise you may pick some term which is not defined by C standard but used extensively enough (e.g. syntax is mentioned many times, but never defined) and then claim that standard doesn't define anything at all.

realloc(malloc(1), 0) as UB under C17 rules

Posted Jul 25, 2024 15:45 UTC (Thu) by farnz (subscriber, #17727) [Link]

All I can tell you is that this is the argument given by acquaintances on the committee - syntax is defined by ISO 2382, but memory is not, which is why there's wiggle-room for saying that the behaviour is undefined.

I, personally, think this is sucky, because you shouldn't be stretching to find UB whenever possible, but should be trying to reduce it to the minimum reasonable scope, but that's not current C culture.

C23 reference links

Posted Jul 24, 2024 11:05 UTC (Wed) by Wol (subscriber, #4433) [Link] (10 responses)

> You could never use zero-sized realloc in cross-platform code, because all platforms behave differently.
> Just don't use zero-sized realloc and you get portable code.
> Before and after the C23 change.

Just declare one version (preferably the "0 returns an invalid pointer you can safely free") as the "blessed" version, and require that the others are available with an option switch. Those people who "realloc(ptr,0)" can now declare the behaviour they want, and those who don't can ignore it.

As I've said, imho "ptr=realloc(ptr,0)" is good defensive programming if it returns null, and if it returns a value that will cause the program to barf if de-referenced - even better? Why not encourage defensive programming, not make it even harder ...

We should be getting rid of UB, and defining a "blessed" behaviour with the option of asking for the "old" behaviour is far better than creating new UB.

Cheers,
Wol

C23 reference links

Posted Jul 24, 2024 11:40 UTC (Wed) by khim (subscriber, #9252) [Link] (9 responses)

> We should be getting rid of UB

We couldn't remove all UBs from the low-level, “everything is possible” languages (e.g. if you don't say that reading ununinialized memory is UB then you are then asked to support really crazy stuff, like do something with the bug report when someone notices that on Windows 8 the contents of the EBX register no longer contained a copy of the executable’s instance handle when the executable entry point as called), but they should be justified!

> and defining a "blessed" behaviour with the option of asking for the "old" behaviour is far better than creating new UB.

Yup. Response to the “we have found out that implementation are doing different things in some cases” issue should be “can we make them do the same thing, instead?” and not “ooh, yes, that's a problem, let's make millions of developers remember that and keep avoiding this landmine till the end of eternity”.

I guess it's broken windows theory applied to language development: if your language have no UBs or it have mere dozen of them with each thoroughly justified then adding a new one become “a really big deal”. But when you language includes, literally, hundreds of them… it becomes a knee-jerk reaction: “some compiler does things differently from all other… let's name that difference UB and make developers cater to that one, too… they are, obviously, superhumans if they can track all these hundreds of existing UBs in their head, they wouldn't mind another one”.

P.S. Originally the position of C and C++ standard was “normal people shouldn't use that standard directly, than't indirect effect document just for the compiler writers and users should consult their compiler documentation instead”, and in that world stance that C23 applied to realloc was sane and justified. But that have changed: other documents (like POSIX specification I already cited) directly reference C standard, compiler developers directly use it in their work and they expect users to follow rules of the standard, too! C and C++ ISO committee may not like that change, but it have already happened! Either they would adapt or they would perish (when C and C++ would be replaced with something saner, be it Carbon, Rust or Zig). Because current situation with C/C++ standards is completely insane for the documents that are supposed to be used directly by developers.

C23 reference links

Posted Jul 24, 2024 13:53 UTC (Wed) by Wol (subscriber, #4433) [Link] (8 responses)

> P.S. Originally the position of C and C++ standard was “normal people shouldn't use that standard directly, than't indirect effect document just for the compiler writers and users should consult their compiler documentation instead”

In which case, again it shouldn't be UB. It should be "we defer to the standard for your platform, be it Posix, Windows or whatever". In which case, as far as C/C++ is concerned it may be undefined, but it explicitly says "to find your definition, go ..." ie it's DSE (Defined Somewhere Else).

Cheers,
Wol

C23 reference links

Posted Jul 24, 2024 14:38 UTC (Wed) by farnz (subscriber, #17727) [Link] (6 responses)

UB is defined as "behavior, such as might arise upon use of an erroneous program construct or erroneous data, for which this International Standard imposes no requirements. Undefined behavior may also be expected when this International Standard omits the description of any explicit definition of behavior. [Note: permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)."

"behaving … in a documented manner characteristic of the environment" is the statement that another standard can define things that C leaves as UB, just expressed in ISO standardese.

If you're saying it shouldn't be UB, then it either needs to be implementation-defined behaviour, or unspecified behaviour; but unspecified has the problem that "usually, the range of possible behaviors is delineated by this International Standard".

C23 reference links

Posted Jul 24, 2024 14:58 UTC (Wed) by Wol (subscriber, #4433) [Link] (5 responses)

> to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)."

That sounds like my DSE. If so, why not say it? Given that realloc (and especially in its form realloc(ptr,0) ) is documented in many places as "well formed code", most of the description of UB does not apply. To declare previously well-formed code as UB is dangerous! To explicitly defer instead to a different standard makes much more sense.

It may still be "here be dragons", but it's tame dragons, not wild ones. And if it requires flags to enable the programmer to specify the behaviour, they are not merely tame, but tamed.

Cheers,
Wol

C23 reference links

Posted Jul 24, 2024 15:23 UTC (Wed) by farnz (subscriber, #17727) [Link]

Because that's how ISO standards work; any standard that is referred to by an ISO standard must be included in the references for that standard. So, to refer to the platform standard from the ISO C standard requires that all the platform standards (including version) that you're referencing are in the reference list for the ISO C standard. This avoids the problem where a standard refers to a document that you can't even identify in order to purchase - if an ISO standard references a document, then a unique identifier for that document at the referenced is present in the references list.

Further, if the dependent standard is updated to a newer version, the reference remains to the older version; you have to issue a new version of the depending standard with updated references to update to the newer standard. Wording like that used by the C standard escapes this, since now the platform standard depends on the C standard, rather than the C standard depending on the platform standard.

C23 reference links

Posted Jul 24, 2024 15:29 UTC (Wed) by khim (subscriber, #9252) [Link] (3 responses)

> That sounds like my DSE.

No. That's your DSE plus carte blanche to do anything else, too!

What would be a lazy implemented doing if one option gives it lots of work and no bonuses and the other one gives less work and faster results on benchmarks, that can be used by marketing team, hmm?

> Given that realloc (and especially in its form realloc(ptr,0) ) is documented in many places as "well formed code", most of the description of UB does not apply.

Not true. POSIX, in particular, defers to the ISO C standard in that regard.

And it incorporates it by reference which means that when C23 would be ratified (expected to happen this year) suddenly, on all POSIX platforms, realloc(ptr,0) would stop being defined.

Neato, isn't it?

That's what happens when different people stop talking to each other.

P.S. Of course compilers wouldn't start breaking existing programs on the next day after ratification of C23. It would take some time before someone would realise that these programs that call realloc(ptr,0) were always non-portable and since 2024 they are also, formally, broken so why not treat them as non-existing and not optimize well-behaving program (do such programs even exist?) better.

C23 reference links

Posted Jul 24, 2024 21:54 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

> Not true. POSIX, in particular, defers to the ISO C standard in that regard.

> And it incorporates it by reference which means that when C23 would be ratified (expected to happen this year) suddenly, on all POSIX platforms, realloc(ptr,0) would stop being defined.

I thought you said that the original C standard defered to POSIX et al? And that's never been officially changed?

> Neato, isn't it?

It's brilliant :-)

> That's what happens when different people stop talking to each other.

Reality disappearing down the event horizon of a bathtub ...

Cheers,
Wol

Relationship between POSIX and C

Posted Jul 24, 2024 22:44 UTC (Wed) by farnz (subscriber, #17727) [Link] (1 responses)

In older POSIX standards, the ISO C standard was brought in by reference, and then POSIX imposes requirements on top of ISO C. That's the way round that ISO envisages its standards being used (ISO provide a base, a higher level standard tightens up requirements in ways that work for a specific use case), but POSIX is imposing fewer and fewer requirements on top of ISO over time, which is why this is now a problem.

The ideal case would be for POSIX to impose requirements on realloc that aren't in conflict with ISO; if POSIX said "realloc(ptr, 0); must be the same as free(ptr); return malloc(0);", for example, that would not conflict with C17 or C23, but would tighten up the behaviour and make it defined in a reasonably sane way.

Relationship between POSIX and C

Posted Jul 25, 2024 8:01 UTC (Thu) by Wol (subscriber, #4433) [Link]

And who's got the balls to do such an eminently sensible thing? :-(

Cheers,
Wol

C23 reference links

Posted Jul 24, 2024 15:14 UTC (Wed) by khim (subscriber, #9252) [Link]

> In which case, again it shouldn't be UB.

Why? It's said quite explicitly in the rationale: Undefined behavior gives the implementor license not to catch certain program errors that are difficult to diagnose. It also identifies areas of possible conforming language extension: the implementor may augment the language by providing a definition of the officially undefined behavior.

> It should be "we defer to the standard for your platform, be it Posix, Windows or whatever".

But that's precisely what role UB is supposed to also serve! Yes, it's conflating two roles, but in a world where standard is only ever read by compiler (and platform) developers and everyone else relies on what compiler (and platform) documentation says it makes sense.

And as you saw POSIX (in it's version 1997) actually does that: it defines how realloc works and thus some programs which C++ standard rejects are accepted by this version of POSIX.

But at some point platform developers have become lazy and stopped doing that, instead 2004 version says: this volume of IEEE Std 1003.1-2001 defers to the ISO C standard. Same with all the later versions.

And this is when things have went downhill: C and C++ standards are still developed on the assumption that there are some mythical implementer that may and would “augment the language” with sane definitions for some of these same 200+ UBs… but “platform developers” see no reason to do that!

It was funny in 2011, sad in 2020, today it's just looks a complete and utter denial of the current reality.

> In which case, as far as C/C++ is concerned it may be undefined, but it explicitly says "to find your definition, go ..." ie it's DSE (Defined Somewhere Else).

That's called implementation defined behavior, only standard committee added an additional rule for themselves not to use it for things that they couldn't adequately explain. This turned DSE (Defined Somewhere Else) into “defined with all possible options listed”, which was supposed to be a good thing… except it made committee to put bazillion things that they couldn't define into the “undefined behavior” bucket.

Again: not a big difference in an world where implementers use standard as a base and then decide what would they define and what would they ignore… catastrophic difference on our world where implementers just say “look in the standard for the definitions, we don't have resources to define these things”.

Committees have limits

Posted Jul 24, 2024 9:55 UTC (Wed) by CChittleborough (subscriber, #60775) [Link] (3 responses)

Writing and updating standards is done by humans, so it is always (small-scale) political.

Expecting the standards committee(s) to agree to something like this is ...err... very optimistic.

Committees have limits

Posted Jul 25, 2024 9:17 UTC (Thu) by kleptog (subscriber, #1183) [Link] (2 responses)

Indeed. I am struck by the parallels between this discussion and discussions in a completely different arena. A certain well-known grouping of sovereign states, when discussing some technical point where for $REASONS they each do things differently, sometimes choose to declare that an equivalent of UB.

This irritates larger businesses and consumer advocacy groups (in the role of users) because UB is Bad and declare that $SOMEONE should simply impose a choice by force and get everyone to fall in line. And on the other hand the sovereign states (in the role of compiler writers) claim they have always done things this way and a see no reason why they should change, because doing so will make a different group of people unhappy. And they would be severely annoyed if someone imposed a choice them (if that's even possible).

The end result doesn't really make anybody happy, but whether it's a real problem strongly depends on the topic. How many programs really use realloc() in a way where this behaviour causes problems?

Committees have limits

Posted Jul 25, 2024 9:49 UTC (Thu) by CChittleborough (subscriber, #60775) [Link]

Excellent analogy! I wish I had written something this good.

Committees have limits

Posted Jul 25, 2024 11:24 UTC (Thu) by Wol (subscriber, #4433) [Link]

> The end result doesn't really make anybody happy, but whether it's a real problem strongly depends on the topic. How many programs really use realloc() in a way where this behaviour causes problems?

Probably not a lot, because (my desired) behaviour was pretty much only MS C, as far as I can make out. But rephrase that as "how many programs SHOULD use realloc in that way" (as in "ptr = realloc(ptr, 0)") and I would say pretty much all. It makes "freeing and destroying" a pointer a single op, so long as you haven't made multiple copies of the pointer. A little bit of programmer discipline, and you've probably turned 99% of "use after free" bugs into a "dereference null pointer" crash.

Okaay, if you don't catch them all in testing you might get the odd DoS. But that's probably better than an exploit - or corrupt data, or a crash miles away from the bug site ...

It's an extremely useful defensive programming technique.

Cheers,
Wol

C23 reference links

Posted Jul 23, 2024 15:30 UTC (Tue) by josh (subscriber, #17465) [Link] (9 responses)

I'm surprised by this as well. This is "non-portable" but long-standing behavior in numerous implementations, and has the lovely behavior that the "realloc" function is everything you need to define a memory allocator (since it can handle malloc if given original pointer NULL, or free if given size 0).

C23 reference links

Posted Jul 23, 2024 17:31 UTC (Tue) by excors (subscriber, #95769) [Link] (7 responses)

WG14 n2464 says:

> Classifying a call to realloc with a size of 0 as undefined behavior would allow POSIX to define the otherwise undefined behavior however they please.

and POSIX does define it already: realloc(ptr, 0) returns either null, or a pointer to allocated space which you must not access, or (if it tries to reallocate space and fails) null with errno=ENOMEM. (Note that's not equivalent to free(ptr), since it may return a new allocation that you must still free.)

glibc defines it as equivalent to free(ptr), when ptr!=NULL. (But realloc(NULL, 0) is equivalent to malloc(0) which may return a new allocation.)

So if you were previously happy relying on the non-portable behaviour defined by POSIX or by glibc etc, I believe you can continue relying on that behaviour. It's explicitly undefined by the C standard (as of C23), but it is defined by other parts of the platform you're targeting, so your code will not be exhibiting undefined behaviour in that environment.

I think this only really affects people who were trying to write portable code that depended only on the C standard, but given how messy and implementation-defined (and, as far as I can tell, incompatible with reality) it was in C17, portable code should have been avoiding this anyway.

C23 reference links

Posted Jul 24, 2024 10:41 UTC (Wed) by Karellen (subscriber, #67644) [Link] (6 responses)

> Classifying a call to realloc with a size of 0 as undefined behavior would allow POSIX to define the otherwise undefined behavior however they please.

and POSIX does define it already:

...

So if you were previously happy relying on the non-portable behaviour defined by POSIX or by glibc etc, I believe you can continue relying on that behaviour.

Sure, but surely C23 doesn't need to make it Undefined Behaviour to allow that. Why not Unspecified Behaviour? That way, non-POSIX compilers have to do something vaguely sensible with such a call (even if just returning NULL/EINVAL (or equivalent)), rather than being able to do literally anything at all.

Undefined, implementation defined, and unspecified behaviour

Posted Jul 24, 2024 11:36 UTC (Wed) by farnz (subscriber, #17727) [Link] (3 responses)

From what I can see, there's two pressures resulting in the standard making it undefined:

The standards committees want it to be clearer to downstream standards like POSIX that they're welcome to partially or fully define undefined behaviour. Making more things UB is a hint that they're absolutely fine with the idea that some things are undefined in Standard C, but fully defined in POSIX C.
Unspecified behaviour is expected to come with a list of acceptable behaviours (even though the compiler is free to pick a different one every time it encounters the construct); undefined behaviour is not. If there's a huge row about what realloc(NULL, 0) or realloc(ptr, 0) is "supposed" to mean, the politics of the situation can make it harder to get consensus on acceptable behaviours than to simply declare it undefined.

Undefined, implementation defined, and unspecified behaviour

Posted Jul 24, 2024 15:49 UTC (Wed) by khim (subscriber, #9252) [Link] (2 responses)

> Making more things UB is a hint that they're absolutely fine with the idea that some things are undefined in Standard C, but fully defined in POSIX C.

Have they actually talked to POSIX guys about the removal of this volume of POSIX.1-2017 defers to the ISO C standard words, then?

Because it's not enough to “give a hint”, someone would have to go and change things to stop “defering to the ISO C standard”. Who would that someone be?

> If there's a huge row about what realloc(NULL, 0) or realloc(ptr, 0) is "supposed" to mean, the politics of the situation can make it harder to get consensus on acceptable behaviours than to simply declare it undefined.

And making existing program invalid on the day they would publish C23 is, somehow, better? How?

Dysfunctionality in standards land

Posted Jul 24, 2024 16:23 UTC (Wed) by farnz (subscriber, #17727) [Link] (1 responses)

Yes, they absolutely have talked to the POSIX guys about this - the issue is that there are three different conflicting POSIX-land interpretations of what realloc(ptr, 0) should do (AIX, BSD, glibc), and so we've ended up in this mess where POSIX doesn't want to pick sides, the C standard doesn't want to pick sides either (since as well as the three POSIX-land behaviours, it wants to support non-POSIX behaviours, too), and the result is a mess because no standards body is willing to stand up and say "this is the behaviour that you must have if compliant with this standard".

And no existing program is made invalid under C23; an implementation is allowed to define UB itself, and the older standards made it implementation defined with an absolute dogs' dinner of a set of permitted behaviours (notably, it was permissible to return a non-NULL pointer that you could not use). Because it was implementation defined, implementations have to document what they do with realloc(_, 0), and the C standard does not overrule your documented behaviour; rather, your documented behaviour overrules the C standard.

The dogs' dinner of choices for the behaviour of realloc(_, 0), in turn, was sufficiently bad that it was effectively UB under a different name, thanks to bad drafting in an effort to keep the conflicting behaviours of AIX, BSD and glibc all as permitted behaviours. As a result, if you're not going to choose a winner, it's more honest to define it as UB, and not as returning a pointer that you may or may not be allowed to dereference, with a requirement that you might be required to free the pointer that you passed to realloc if it returned NULL, but you also must not free the pointer that you passed to realloc if it returned NULL.

Underlying all of this is an unfortunate truth about the state of Standard C right now; ISO doesn't want to pick winners among existing implementations, unless it's clear that there is a "right" and a "wrong" answer. The Austin Group (POSIX) don't believe that they have sufficient weight with compiler authors to get compilers to comply with POSIX C in places where it defines things that ISO does not. And language users don't have enough weight with compiler authors to get an agreement together that can be taken to The Austin Group, or to ISO. Which leaves us in a situation where ISO is (correctly, per their charter) not defining parts of C that don't have a generally agreed upon meaning, but there is no way to get compiler authors to define things that ISO doesn't.

And that's bad for C users, since it leaves C users in a position where anything that's not defined the same way by every obscure C compiler out there cannot be relied upon between compiler version upgrades of a big-name compiler like Clang or GCC. Heck, from what I can gather, The Austin Group aren't even convinced that, if they were starting again now, they could insist that CHAR_BIT == 8 - the ISO standard says that CHAR_BIT >= 8.

Dysfunctionality in standards land

Posted Jul 24, 2024 21:11 UTC (Wed) by khim (subscriber, #9252) [Link]

> And no existing program is made invalid under C23; an implementation is allowed to define UB itself, and the older standards made it implementation defined with an absolute dogs' dinner of a set of permitted behaviours (notably, it was permissible to return a non-NULL pointer that you could not use).

Sure, but it wasn't possible to back-propagate the fact that someone tries to call realloc with zero size into functions that precede or follow that piece of code and remove security checks and other such things.

New working makes such optimizations more-or-less no-brainer for anyone who wants to earn some brownie points in their resume.

> Because it was implementation defined, implementations have to document what they do with realloc(_, 0), and the C standard does not overrule your documented behaviour; rather, your documented behaviour overrules the C standard.

That's fine. The biggest issue with UB is not cases where someone uses it on purpose, but cases where someone triggers that condition because of accident (attempt to allocate zero-sized buffer, e.g.) and then compiler turns your program into a pile of goo (like new wording allows).

That is the biggest issue and it was introduced in C23, it wasn't in any other standard AFAICS.

> The dogs' dinner of choices for the behaviour of realloc(_, 0), in turn, was sufficiently bad that it was effectively UB under a different name

No, it wasn't. Any behavior, not matter how crazy it is, still keeps your program in the set of a strictly conforming programs and thus guarantees somewhat sane behavior in the rest of program. But if your program triggers UB then it's no longer a strictly conforming program and compiler have the right to turn it into a pile of goo.

> And that's bad for C users, since it leaves C users in a position where anything that's not defined the same way by every obscure C compiler out there cannot be relied upon between compiler version upgrades of a big-name compiler like Clang or GCC. Heck, from what I can gather, The Austin Group aren't even convinced that, if they were starting again now, they could insist that CHAR_BIT == 8 - the ISO standard says that CHAR_BIT >= 8.

No, it's not just “bad”. It's a complete disaster. Currently Google is looking for way to completely ditch C++ (and C, of course, too), long term. Sure, its says we don't know if any of these options will be feasible or how much they will cost; unless some of them prove both feasible and cost effective, we will continue investing in C++ despite its problems, but given the fact that they explicitly say that C++ is a long-term strategic risk for…

Microsoft wanted to ditch C++ in favor in C# and that plan failed, but they may succeed with Rust.

But in the end… I guess the worst thing that may actually happen is C and C++ becoming new COBOL and FORTRAN. COBOL 2023 and Fortran 2023 both exist thus C and C++ committee would be able to travel around the world to write and publish new papers even if no one sane would even bother to read them, thus for them such outcome is acceptable, I guess.

C23 reference links

Posted Jul 24, 2024 12:05 UTC (Wed) by excors (subscriber, #95769) [Link] (1 responses)

It used to be unspecified behaviour (meaning the standard "provides two or more possibilities and imposes no further requirements on which is chosen in any instance"), and I think the basic problem was that the standard was so vague as to be practically useless. With C11, portable code doesn't know whether realloc(ptr, 0)==NULL means success (ptr freed) or failure (ptr not freed), so the only way to write a correct program is to never call it with size=0.

C17 tightened the definition, but badly - the wording was internally inconsistent, and also incompatible with glibc, probably unintentionally. (https://sourceware.org/legacy-ml/libc-alpha/2019-08/msg00...)

(C17 also said "Invoking realloc with a size argument equal to zero is an obsolescent feature", which means it "may be considered for withdrawal in future revisions of this International Standard", so there's been plenty of warning.)

My guess is that the standards people realised it was really hard to come up with a precise definition that was self-consistent, useful, and compatible with reality, and it's not a good use of their time to keep discussing it when it's such a pointless feature anyway (just call free(ptr) if that's what you want), so they gave up and called it undefined. Platforms and compilers can still document the behaviour they actually implement, so nothing will really change.

C23 reference links

Posted Jul 24, 2024 15:53 UTC (Wed) by khim (subscriber, #9252) [Link]

> It used to be unspecified behaviour (meaning the standard "provides two or more possibilities and imposes no further requirements on which is chosen in any instance"), and I think the basic problem was that the standard was so vague as to be practically useless.

Seriously? I have a newsflash for you: any practical programmer would attest that even extra-vague “anything but UB” definition is still better than UB.

Because “anything but UB” behavior should be predictable and debuggable. It may not be what you want or need, but it's not “your program works for years, then stops working because starts aligned differently” that UB signifies!

realloc/malloc of size zero (C23 reference links)

Posted Jul 23, 2024 17:35 UTC (Tue) by mjw (subscriber, #16740) [Link]

Because of the different implementations valgrind does have warnings for realloc of size zero. You can even say which behaviour you expect with --realloc-zero-bytes-frees=yes|no or if you never want to see valgrind memcheck warn about it with --show-realloc-size-zero=no. In general there are various ways you can use realloc "wrongly" that valgrind can catch (or with newer gcc warning flags at compile time): https://developers.redhat.com/articles/2023/07/26/checkin...

One issue seems to be that it isn't entirely clear what it means to have a zero sized malloced block of memory. glibc malloc (0) will return an unique pointer (so not NULL) that you have to call free on.