Would you like signs with those chars? [LWN.net]

Would you like signs with those chars?

Posted Oct 25, 2022 15:46 UTC (Tue) by khim (subscriber, #9252)
In reply to: Would you like signs with those chars? by dvdeug
Parent article: Would you like signs with those chars?

> Pretty much everything is defined in terms of Unicode now; even if you're processing C code, you should still be prepared for identifiers in Russian, Greek or Chinese.

Sure, but only part of your program which deals with identifiers needs adjustment.

You can write int ｆｏｏ = 42; but can not write int foo = ４２; which means that you can easily use “C” locale and all ASCII-only functions with a simple change: where before Unicode you used is isalpha(c) or isalnum(c) now you would use c < 0 || isalpha(c) and c < 0 || isalnum(c).

That's how doxygen handles it, I would assume someone may use isalpha(c) and/or isalnum(c) in a similar way.

to post comments

Would you like signs with those chars?

Posted Oct 25, 2022 16:01 UTC (Tue) by dvdeug (guest, #10998) [Link] (3 responses)

That diff shows doxygen keeping UTF-8 characters together. The question is why do you need to know that a character is alphabetic but only in ASCII. In your example, the lex is "int" identifier "=" integer ";". It's not looking for ASCII letters; it's looking for 'i', 'n', 't', or the identifier in Unicode.

Would you like signs with those chars?

Posted Oct 25, 2022 16:10 UTC (Tue) by khim (subscriber, #9252) [Link] (2 responses)

> It's not looking for ASCII letters; it's looking for 'i', 'n', 't', or the identifier in Unicode.

Only identifier is not “Unicode”. It's alpha or Unicode then alnum or Unicode (where Unicode is defined as “anything with a high bit set”).

Doxygen does that with lex, but in simpler cases you may do the same with ctype.h.

Would you like signs with those chars?

Posted Oct 25, 2022 23:41 UTC (Tue) by dvdeug (guest, #10998) [Link] (1 responses)

No, not if that was C or C++ code. I dug up a copy of the C++2003 standard I had laying around, and it specifically defines the set of letters in an identifier, and there's a limited number of usable Unicode characters. I pretty sure that any standard updated this century will have been made with reference to Unicode Standard Annex #31. The JVM (not Java) standard goes the other way and only restricts . ; [ / from being in a name.

Would you like signs with those chars?

Posted Oct 26, 2022 0:36 UTC (Wed) by khim (subscriber, #9252) [Link]

Sure, but standard doesn't say what compiler (or, even worse non-compiler) have to do with broken programs.

And if you ignore what standard says and just go with isalpha/isalnum + Unicode (where Unicode == “high bit is set”) then you would handle all correct programs perfectly. And if someone feeds incorrect one… who cares how would it be handled?

It's not as if we live in a world where everyone cares all that much about following the standard to a T.