The "Trojan Source" vulnerability

I think the problem arises when people look at PR on github or an email message in their mail reader, and it looks visually like it's fine. Sure, it would cause your cursor to behave very strangely if you tried to move through it, and a text editor would likely reveal that the resulting file consists of a very strange sequence of code points, but anyone just looking at a display of the diff won't notice. If the patch is to a file that doesn't need work often, it could be years before anyone looks at it in a text editor.

Yes, C has allowed Unicode identifiers since C99, and while supporting unescaped characters is not required, most compilers do so now--GCC since GCC 10.

As a matter of fact, GCC already has -Wcomment: "Warn whenever a comment-start sequence /* appears in a /* comment, or whenever a backslash-newline appears in a // comment." Since this warning is enabled by -Wall which is widely used, I guess the performance hit is more than acceptable, and I don't think checking for balance of bidi controls would be significantly harder or slower. Clang has a something similar.

Yeah, and it would be nice if the compilers grew a '-Werror=non-ascii-in-code' that projects could use to say "we do want to make use of some features in C99/C11/C++987, but not random unicode chars in identifiers, TYVM".

The only proper Unicode support for a C compiler is to obey Annex D.1 of the standard w.r.t. permitted codepoint ranges in identifiers, and to parse UTF-8/16/32 multibyte sequences correctly within comments and string literals to find the delimiters correctly. Anything beyond that is a syntax error.

I'm not dismissing the number of people using different alphabets or writing directions at all. I mean they don't need to *mix* characters from various sets without quoting. For example why would you find the Cyrillic "H" (equivalent to the latin "N") between latin characters ? It's not an "H", it uses the same representation but is a character to be used within another alphabet. For me there should be instructions to switch the alphabet and this ought to always appear only on word boundaries. The quoted-printable mode was not much different form this (or at least made it easy to check). Also why would you want to write using a latin alphabet in RTL mode, except to have fun ? Especially when *some* characters are switched directions (parenthesis, brackets etc) but not all! At the very least we ought to see completely reversed characters, as used to appear in ancient greek where you could write both directions and the characters directions indicated where to start from.

> Because English happens to be the ONLY language in the whole world that can be correctly written limiting yourself to ASCII.

Not even then because "café" and "naïve" are (adopted) English words and they came with their spelling and pronunciations. Sure, English orthography is weird enough that the unadorned versions are plausible, but it is certainly easier with the accents on them.

What? Arabic numbers are written right-to-left, even in left-to-right languages. (It's in the name!)

> Are you really going to respell the person's name into the Arabic abjad?

Japanese writers respell European personal and place names into katakana all the time, which does vastly more violence to the phonology of English than writing it in Persian script(1) does.

(1) Arabic script extended with letters for consonants found in Persian but not Arabic; coincidentally, these turn out to be the same consonants found in English but not Arabic.

> * Suppose you are writing a newspaper article about some guy from the UK who did something newsworthy. But you're in Iran.

This is exactly the type of things I was mentioning previously: here it's about complex documents and you can use plenty of typesetting options offered by the software involved for writing this newspaper, you can switch to other charsets or directions just like you can change the font style (bold/italics) or include math formulas or paste images in the middle of the text. All stuff that isn't covered by UNICODE and properly supported by LaTeX or HTML for example. I remember having had to write "é" in HTML to see an "é" in a page, and so what ? It did work fine, and editors were made to ease this input.

You'll notice that for regular use, even for pure humans it's difficult to switch directions on the same line. You have to leave room for what you need to write without being certain it will fit. And for printers like Gutenberg in their time, it would have required to push all the characters on the same line in the same order anyway. We just created a new requirement that didn't exist for pure convenience or laziness.

Similarly it's not correct to use multiple charsets inside a same word. Some glyphs look similar, and they will all be read as coming from the same charset by a human (or even an OCR software), which is the problem caused by homoglyphs, which is another problem that was purely created by UNICODE and that didn't exist in the real world.

I sincerely think that being able to state the charset on a per-word basis would fit the usage real humans have of charsets. And the writing direction could then be applied to the charset itself.

> * And, once you need to embed one script inside of another, somebody has to write an algorithm

Yes absolutely but just like your browser will decide where to place an image or like LaTeX will decide how to render a math formula. We must not confuse the encoding and the software.

But anyway we have what we have now, which covers much more than what we need and cause tons of problems. Just think that with the help of IDN it might even be possible to register the domain "<RTL>nwl<LTR>.net" and try to impersonate "lwn.net", or to do the same using some homoglyphs, though there are not that many possible in "lwn", maybe the "l" could be replaced with something looking like an l. It's not something we needed.

While I fully agree with your last two points, I don't think the arguments in the first three points are quite correct.

> Are you really going to respell the person's name into the Arabic abjad?

As a matter of fact, yes of course you are. Here's an experiment everyone can do, no need to speak multiple languages:

1. pick a wikipedia page about a even that you know is going to name a number of people, for example https://en.wikipedia.org/wiki/2016_United_States_presiden...
2. locate the "languages" section in the side bar and open any number of translations in languages that are not written in the Latin alphabet
3. check if you see any names written in the Latin alphabet in the middle of the text - you won't (except perhaps for pictures of campaign posters in this case).

I hope this will convince you that people are in fact transliterating names all the time. Now I'm going to argue that it's indeed the most sensible thing to do in most circumstances.

> What if their name is phonotactically invalid in Arabic?

The thing about loss of phonetic information in transliterations is, it doesn't matter at all to the intended reader: the phonetic information that's lost mostly doesn't make sense to the reader anyway. That is, the reader will likely not hear the difference, and is probably unable to pronounce the name correctly. (Assuming of course the reader doesn't speak the source language.)

Also, even when using the same alphabet (non-random example: a French person reading the name of a Polish colleague), a lot of phonetic information is lost: the mapping between (groups of) letters and sounds varies wildly across languages that use the Latin alphabet (and for a given language, can also be pretty complex and very far from one-to-one). So, information loss if far from being a specific feature of transliteration.

It's actually even possible for less phonetic information to be lost when transliterating. As an example based on personal experience, if I show my name written in the Latin alphabet to a random Arabic-speaking person (who doesn't happen to also speak French) the most likely result is they're going to pronounce it a bit like an English speaker who's not familiar with that name would pronounce it, which is quite different to how it's pronounced in French. If I show them an Arabic transliteration instead, their pronunciation is going to be closer to the original.

Of course, when you're not familiar at all with the writing system of the name (like, if I were to read a Russian name in Cyrillic for example), then the phonetic information is exactly zero, while a transliteration would at least convey non-zero (if approximative) information on how the name is pronounced.

Finally, apart from phonetics, in an unfamiliar writing system, even the most basic operations like testing for equality can't be taken for granted: if you see the same name written in different fonts, will you recognize that it's the same? If you see two blobs that look similar, can you confidently conclude that they're indeed the same name?

So yes, transliterating is the right thing to do: it's the only reasonable option when the target reader can't be assumed to be familiar with the original writing system, and even when they are, not transliterating imposes a high cognitive load on the reader, while not transmitting more phonetic information unless the reader is actually familiar enough with the original language, not just its alphabet.

> * Obviously, this goes both ways, so you can just as easily need to embed Hebrew or Arabic into LTR writing.

I think we should acknowledge that the situation is very far from being symmetrical. Take your first point and "reverse" it: "you're writing an article in a UK newspaper about some guy from Iran who did something newsworthy. Are you really going to respell the person's name into the Latin alphabet? What if their name contains sounds that are not part of English?" Unless we live in _very_ different bubbles, all the names you're seeing in your news sources are systematically transliterated to English, and most of the time you've never seen them written in their original writing system even once. And most of them are full of sounds to which the transliteration can't do justice. (Case in point: did you know that "Arab" in Arabic starts with a consonant sound, before the first "a" sound?)

I think your examples rely on the implicit assumption that everyone has a least some familiarity with the Latin/English alphabet. I'm not saying that's not true, I'm just saying the situation is very asymmetrical.

> * What if you're living in Israel (Hebrew is RTL), and you want to write a message to someone, telling them the address of a location in the United States. You certainly won't respell that, if you actually want your recipient to find it!

I think that's a more convincing example that the name of a person in a newspaper, but even then, that depends on what you want your recipient to be able to do with the address. If I'm given an address in a writing system I'm not familiar with, about the only things I can do with that is (1) copy-paste it and (2) show it to a native speaker. I can't pronounce a reasonable approximation of it. I can't break it into parts. Other than copy-pasting, I can't copy it (either using a keyboard on another device, or on a piece of paper). It's not even sure I can recognize the street name on a street sign or a map because it's going to be using a different font (and I can't tell which part of the address in the street name). Granted, if I'm using a navigation app, all I need to do is to copy-paste the name into the app. But now consider if I'd been given GPS coordinates instead: compared to the native name, GPS coordinates make less sense to a native speaker, but OTOH I'm able to type them on a keyboard or write them down on paper. So, I don't think it's that clear-cut.

Have you travelled to parts of the world where the main language does not use the Latin alphabet? Do you remember being given addresses in the local writing system or an English transliteration? I can't answer for you, but as a data point, I just checked the French-speaking travel guide about Jordan that's in my library: there's not a single Arabic character in the book, and all addresses are given in French/English transliteration. (Which makes sense considering it's printed on paper so I can't copy-paste from it.)

So, while I agree that writing bidirectional text is useful and needs to be supported (including in plain text), I don't think the particular examples you're giving are that convincing, because if we followed that line of reasoning, we should see a lot more non-Latin scripts in the parts of world that use the Latin alphabet, and that's just not what we observe.

I think perhaps a stronger argument would be that in practice there is a dominant writing system, which is written in a certain direction, and if your native language happens to be written in the other direction, then you'll end up having to handle bidirectional text, because the dominant writing system is everywhere (especially when computers are involved). Said otherwise, for people whose native language is written LTR, bidi is a nice to have for some cases, but for people whose native language is written RTL, it's a must. It's all too easy to forget about that when we're in the former category.

Cyberax told something on which I agree: for the identifiers use only the ascii charset. This would reduce the risk of "homoglyphs" , and solve another issue about multi-language intereoperability: the ascii charset is the most common "subset" of characters recognized by everyone, so it create less problem from a interoperability point of view.

As Italian people, I never though about writing a function name like caffè().

For the other UNICODE codes that affect the directionability, I would allow them only in the comments that span a full line; something like

/*
here I would allow code that affect the directionability
*/

/* here I don't */

For the literal string, I would allow all the characters (even the homoglyphs ones, even I am sure that this pose some concerns) but the one that affect the directionability.

But I don't know the UNICODE so deep to exclude that other risks exist.