LWN: Comments on "The "Trojan Source" vulnerability"

The "Trojan Source" vulnerability

nybble41 — Mon, 08 Nov 2021 15:49:19 +0000

To the extent that they can be applied to durable storage outside of computer systems, such as on paper, the terms "little-endian" and "big-endian" refer to the order of the digits as they are placed with respect to the surrounding text—for example, compared to the order in which you would number the words or paragraphs. If the text is right-to-left and numbers are written with the least significant digit on the right, then the numbers are little-endian. (In computers, a little-endian field in a structure does not become big-endian just because a program loads or stores the most significant byte first; only the locations of the digits matter, not the order in which they are read or written.)

Reading aloud is akin to a serial communication protocol, and the order used for serialization can differ from the order used for storage (i.e. the written order). From your description, in Arabic (as in English) the numbers are read in big-endian order since the most significant digit is pronounced first.

The "Trojan Source" vulnerability

mpg — Mon, 08 Nov 2021 13:23:13 +0000

That's my understanding as well. Though I'd like to point out that numbers are read the same way in German (that is, "five and twenty", "hundred five and twenty"), so it's probably not related to the direction of the surrounding text (RTL for Arabic, LTR for German).

The "Trojan Source" vulnerability

farnz — Mon, 08 Nov 2021 10:13:16 +0000

Arabic's an interesting one in that regard; there are multiple versions of spoken Arabic in use in the Middle East, and (e.g.) Iraqi Arabic is identifiably different from Syrian Arabic to the point where they are close to becoming different languages. But because of the importance in Islam of memorising the Quran accurately, and the respect accorded to those who can recite the Quran from memory, that split has never actually happened.

The "Trojan Source" vulnerability

Cyberax — Mon, 08 Nov 2021 09:01:17 +0000

That sometimes happens, but not always. Pretty often different languages would use different characters.

The "Trojan Source" vulnerability

Wol — Mon, 08 Nov 2021 07:45:27 +0000

I always understood China had one written language, but many spoken languages, as in all the languages used the same glyph for the same word, but two different regional languages might actually say them as two completely different words.

I know we actually use slightly different spellings and the pronounciation is similar, but like our lake/loch/lough - all pretty much the same word for the same thing. But we also have mere, and there's probably others I can't think of right now ...

Cheers,
Wol

The "Trojan Source" vulnerability

ghane — Mon, 08 Nov 2021 01:52:27 +0000

I live in a Chinese-majority-speaking country, but everyone I know speaks good English, the exceptions being elderly parents, who speak basic English. I have survived here for many years with nothing but English.

The taught form of Chinese (Mandarin-dialect) writing is in Simplified Characters, students also learn Pinyin. Whenever I have actually seen it used (store fronts, etc), it is written un-accented. I have been told that the assumption is that if you really cared, you would be able to read the Chinese characters, and if you can not, then the un-accented letters are good enough, you wouldn't understand what the tones mean. This works.

I understood early on that Peking had not been renamed, it was simply written in a different script, which reused the glyphs found on an English typewriter. But it was still a shock some years ago in Taiwan, when I wanted to visit the tomb of Chiang Kai Shek, and no one, including my English-fluent work colleagues, knew who I was talking about, till I wrote it down. Then there were "Oh, sure, of course". They wrote it down *exactly* the same way as me, but pronounced totally different, and insisted they were pronouncing it as I had written it down.

I blame Wade and Giles.

The "Trojan Source" vulnerability

dtlin — Mon, 08 Nov 2021 00:35:24 +0000

From what I understand, in Arabic the 1's and 10's digits are read little-endian, but all other digits are big-endian. For example, 25 is خمسة وعشرون (five and twenty), 125 is مائة وخمسة وعشرون (hundred five and twenty).

The "Trojan Source" vulnerability

karkhaz — Mon, 08 Nov 2021 00:29:30 +0000

See my two comments on this topic: https://lwn.net/Articles/829994/ and here https://lwn.net/Articles/830017/

Not sure what you mean by the normal order, but when writing in Arabic, numbers are 'little-endian' in that when you read them aloud, you read in the opposite direction to the surrounding text. With words, you read a sentence from right to left. When your eyes encounter a number in the middle of a sentence, you 'skip ahead' to the leftmost digit (which is the highest-magnitude one, as with English), and begin reading the number from left to right. When you've finished reading the number, your eyes jump past the number (leftward) again and continue reading the rest of the sentence from right to left.

(Though in practice, nobody reads numbers one digit at a time, in Arabic or any other language. Unless the number is very long, your eyes can probably parse the entire number with a single glance.)

When hand-writing Arabic, you write from right to left, and if you need to write a number, you move your hand leftwards to leave a gap large enough to fit the whole number that you intend to write. You then write the number left-to-right, as with English. One of the disadvantages of RTL languages is that when handwriting as a right-handed person, you're much more likely to smudge the ink because your hand glides over text that you've only just written.

The "Trojan Source" vulnerability

anselm — Sun, 07 Nov 2021 23:26:45 +0000

The problem is that of course he was a powerful man in the 30s and 40s and back then if Germany had had a nontrivial empire it would likely have been common to name people in non-German parts of that empire after someone powerful in the imperial capital.

Possibly. But in 21st-century Germany, even the neo-Nazis seem to realise that it's not a great idea. Even calling your son “Adolf” (without the “Hitler” – after, say, your beloved late grandfather) probably amounts to setting him up for a huge amount of schoolyard grief that he could just as well do without.

Having said that, in Spanish-speaking countries “Jesus” appears to be a reasonable name for boys but although it is perfectly legal here in Germany, it is practically a non-starter. According to statistics, only 2 out of 100,000 kids born in Germany per year are called Jesus, and the general consensus even among religious people is that it's not exactly doing them a favour.

The "Trojan Source" vulnerability

Gaelan — Sun, 07 Nov 2021 22:40:08 +0000

I wonder if people who speak RTL languages think of Arabic numerals as "big endian, written left-to-right" (i.e. the same way we do), or as "little endian, written in the normal order"?

The "Trojan Source" vulnerability

nix — Sun, 07 Nov 2021 17:16:37 +0000

> I have no words for parents who think it is a good idea to name their offspring “Adolf Hitler”

The problem is that of course he was a powerful man in the 30s and 40s and back then if Germany had had a nontrivial empire it would likely have been common to name people in non-German parts of that empire after someone powerful in the imperial capital.

Which is how we get M. K. Stalin, the leader of Tamil Nadu and a notable threat to the BJP (because popular, successful, and competent). Literally named after Joseph Stalin, shortly before the latter's death.

I suppose it might have been barely possible to get his parents to pick a different name, but his father, M. Karunanidhi, was a fairly powerful figure even back then... tricky.

The "Trojan Source" vulnerability

mpr22 — Sun, 07 Nov 2021 17:09:29 +0000

We call them Arabic numerals because they were introduced to Europe by Arab merchants; the Arabs had previously acquired them from Hindu mathematicians :)

Both Western Arabic numerals (the ones you and I use) and Eastern Arabic numerals (the ones traditionally used in the middle east) are written with the MSD on the left.

The "Trojan Source" vulnerability

nix — Sun, 07 Nov 2021 17:08:39 +0000

> (By the way, Wikipedia often gives both a transliteration and the original writing, often with the API transcription too, and sometimes even a recording, which I find very helpful. Continuing with my previous example, if you look at the article for each candidate rather than the article fro the election, the original name is usually given in the first paragraph. But I think that's because we're looking at an encyclopedia, not a newspaper.)

A hilarious consequence of Wikipedia's transliteration rules combined with people not being able to figure out what words in foreign writing systems mean (or, it seems, even that they are different words or not): http://itre.cis.upenn.edu/~myl/languagelog/archives/00518...

The "Trojan Source" vulnerability

nix — Sun, 07 Nov 2021 16:57:07 +0000

What? Arabic numbers are written right-to-left, even in left-to-right languages. (It's in the name!)

The "Trojan Source" vulnerability

Wol — Sat, 06 Nov 2021 19:53:08 +0000

> Your mum was presumably born before 1975. Before 1975, the rule was that for someone to obtain German citizenship at birth, their father¹ had to be German, or their unmarried mother had to be German.

:-)

I started secondary school that year ...

Cheers,
Wol

The "Trojan Source" vulnerability

ghane — Sat, 06 Nov 2021 02:08:32 +0000

... and Malay, as used in Malaysia and Singapore. I can think of other examples, with less assurance, but the common theme seems to be that the writing, spelling, and general orthography was designed by a British educator or priest, and he had only a (what we now call) ASCII pen and paper.

The downside of this has been that certain spoken sounds and accents have been lost in Malay (they still exist, but no one under 50 uses them), because the written (purely phonetic) form is what is taught. Since only one diagraph exists ("sy", as in "SHoulder") the Arabic loan word "khabar" (news) must be written as "kabar", and ends up pronounced that way by non-natives, and most native speakers too.

This was seen as a good thing, as it enabled literacy rates to rise very fast in the 60s and 70s. The has been cemented by the T9 cellphone keyboard and US-en keyboards.

And so, although most Malaysians are somewhat bilingual (everyone has school friends whose native language is Malay, Hokkien, or Tamil), the idea that you would need anything but a Latin keyboard is surprising here.

The "Trojan Source" vulnerability

anselm — Sat, 06 Nov 2021 00:52:11 +0000

Oh - and as for citizenship depending on the status of the parents, why can't I claim German nationality? My mum was born in Germany to a German mother but she wasn't entitled to citizenship, so I can't inherit ...

Your mum was presumably born before 1975. Before 1975, the rule was that for someone to obtain German citizenship at birth, their father¹ had to be German, or their unmarried mother had to be German. If they had a German mother who was married to a non-German, that was just too bad as far as their (then non-existent) German citizenship was concerned. Without knowing more about the specific circumstances, it seems that your mum might be a victim of that unfair rule, and since in that case she's not entitled to German citizenship by birth, neither are you.

The law was changed in 1975 after the German federal constitutional court had pointed out its blatant asymmetry, and the rules are now more relaxed. If the current rules had been in effect when your mum was born, she would indeed have been entitled to German citizenship, and that would quite likely have applied to you, too, even if your mum had been permanently living in the UK at the time of your birth (there are more rules).

1. “Father” here means “person legally married to their mother”, not “male person involved in their conception”. That particular rule still applies.

The "Trojan Source" vulnerability

anselm — Sat, 06 Nov 2021 00:26:41 +0000

You children would become a German citizens (in addition to having citizenship of parents) and that is when shit hits the fan.

The rule is that if at least one parent has lived in Germany for eight years or more and has an indefinite right of residence in the country, any of their children who are born in Germany are entitled to German citizenship (this is generally not viewed as a disadvantage). They can have German citizenship in addition to whatever citizenship they may get from their parents. When they're 21, they may be asked to pick one or the other unless they have spent a significant part of their youth in Germany (eight years of residence, or six years of school, or graduation from school, or completion of a vocational qualification), in which case they may keep both. I don't know offhand how the Putin/Putina issue is dealt with in such a case, but since the parents aren't German citizens, the restriction that all their children must have the same surname may not apply; if Mr Putin and Mrs Putina go to the Russian embassy to make sure that their Germany-born kids have Russian citizenship as Vladimir Putin and Katerina Putina, then when the question comes up of whether little Katerina can also have German citizenship because her mum and/or dad has been living in Germany for the last eight years, she may already have the Russian “Putina” surname and the German authorities may well pick that up and run with it (as we said before, if you're naturalised in Germany you get to keep whatever name you already have). But maybe not.

Generally, countries award citizenship to newborn children based on the citizenship of their parents (ius sanguinis, or “law of the blood”) or based on whether they're born in the country (ius soli, or “law of the ground”). Germany mostly goes by ius sanguinis (i.e., if one of your parents is German, you get to be German, too, even if you're born abroad) but we operate the ius soli exception discussed above for the benefit of the children of migrant workers who came to Germany from abroad and settled here (the EU makes this very easy now, but in the 1960s there was a large influx of workers from places like Turkey, and in consequence there are now many second-generation or third-generation immigrants who are German citizens of Turkish extraction). This is in contrast to, say, the United States, which is very much a ius soli country – if you're born in the US, you're entitled to US citizenship no matter what your parents' citizenship is (unless your parents are foreign diplomats), but that's not how it works in Germany.

The "Trojan Source" vulnerability

Wol — Fri, 05 Nov 2021 22:50:11 +0000

> > In Germany, citizenship status generally depends on the citizenship status of one's parents, so in that case the kids would be Russian and not German, and Russian rules would apply.

And what if Russian rules conflict?

A friend of mine was born in Nigeria. So although she was a British national, as a "national born abroad" her children had no automatic right to inherit. The first two were born in England so that wasn't a problem. The third was born in Bahrain and could not inherit. Fortunately the little girl's father was British-born so she inherited from him.

Oh - and as for citizenship depending on the status of the parents, why can't I claim German nationality? My mum was born in Germany to a German mother but she wasn't entitled to citizenship, so I can't inherit ... (actually, I could try claiming under the Jewish refugee rules, but it's a bit of a long shot...)

Cheers,
Wol

The "Trojan Source" vulnerability

khim — Fri, 05 Nov 2021 18:07:11 +0000

Note that you may live in Germany without having German citizenship (if you have a Blue Card, e.g.).

You children would become a German citizens (in addition to having citizenship of parents) and that is when shit hits the fan.

The "Trojan Source" vulnerability

anselm — Fri, 05 Nov 2021 17:49:18 +0000

Thus we have Marina Putina who gave birth to two twins. Boy and girl. Boy should be Vladimir Putin while girl needs to be Katerina Putina. Would German authorities accept that?

If Mrs Putina was living in Germany at the time, with German citizenship, then as far as German authorities are concerned both her kids would have to be registered as “Putin”. Those are the rules. (If she moved back to Russia later then the girl would presumably get to be “Putina” there, if that's OK with the Russian authorities.) Alternatively Mr Putin and Mrs Putina could opt to have both kids be called “Putina”, but Mr Putin and/or his son, later in life, might not be enthusiastic. I don't know exactly what happens if the Russian citizens Mr Putin and Mrs Putina are visiting Germany and Mrs Putina happens to give birth to twins while they're here. In Germany, citizenship status generally depends on the citizenship status of one's parents, so in that case the kids would be Russian and not German, and Russian rules would apply. (Mr Putin and Mrs Putina would presumably go to the Russian embassy/consulate to have the birth registered, and German authorities would not be involved at all.)

The general rule is that if you already have a name and become a naturalised German citizen, you get to keep your name as it is, but you can optionally have it adjusted to suit German conventions. So if Ms. Putina, upon assuming German citizenship, would, for example, prefer to be called “Putin”, that's not a problem. The same would presumably apply to her children if they had been born outside Germany. (These are exceptions to the general rule that you're not supposed to change your surname at all except when you marry, get a divorce, or become adopted.)

The "Trojan Source" vulnerability

Wol — Fri, 05 Nov 2021 16:25:49 +0000

> In Germany in general, changing your first name and especially your surname (without getting married or divorced) is very difficult – not like in, say, the UK, where you can adopt a new name basically whenever you like by making an official declaration.

My wife went through THREE different names the week we got married. She'd been married before so turned up at the rehearsal with one name, and left with a completely different one (new first name, original surname) courtesy of my best man who is a Solicitor. Then a few days later we got married and she left with my surname :-)

Cheers,
Wol

The "Trojan Source" vulnerability

mpr22 — Fri, 05 Nov 2021 16:00:11 +0000

> What happens if mother doesn't know who was father of the child?

If the mother doesn't know who the father is, or doesn't want the father involved in the child's life, then her child gets a matronymic instead of a patronymic.

This has apparently been acceptable practice for a very long time, with Wikipedia's page about Icelandic names citing not only modern examples, but also a mediæval poet called Eilífr Goðrúnarson.

The "Trojan Source" vulnerability

khim — Fri, 05 Nov 2021 15:32:18 +0000

> This is probably because we like things to be nice and orderly here, and allowing people to have arbitrary name changes somehow doesn't feel right.

Yeah, sure, but let's consider a very simple example (small modification or real-world example). Assume a different world where Vladimir is not father of Marina but her son instead.

Thus we have Marina Putina who gave birth to two twins. Boy and girl. Boy should be Vladimir Putin while girl needs to be Katerina Putina. Would German authorities accept that? I'm afraid this would be in direct violation of the rule all of your children must have the same one, but you can defer that decision until your first kid is officially registered.

> If you then get divorced, revert to the surname you had before your marriage, and keep custody of your kid whose surname is that of your ex-spouse, it is possible for your surname to be completely different from your kid's surname, to everybody's confusion.

Oh, sure, but that's other story. I knew a family where mother, father and child all had different surnames because of complex history. We are just talking about birth certificate here.

> If you want to see a place with really strict naming rules, consider Iceland, where the state will force you to give your kids Icelandic names from an officially approved list (although to be fair you can submit new names to be added to that list), and where your surname derives from your parents' first name, so if you're Olaf and your dad is Erik, you'll be Olaf Eriksson but your son Thor will be Thor Olafsson.

Wow. What happens if mother doesn't know who was father of the child? I guess they may do a DNA paternity testing today (although in same cases this may be difficult if two potential fathers were marines who drowned before child was born), but what they done before it became available?

The "Trojan Source" vulnerability

anselm — Fri, 05 Nov 2021 15:09:36 +0000

that article

“That article” is a reasonable overview of how the German bureaucracy deals with people's names. The first-name rules can, in practice, be boiled down to “whatever you can get past the person at the registrar's office”; some of them are stricter than others but in general, if you can show that, e.g., an unusual first name is common elsewhere in the world, you'll be fine. I have no words for parents who think it is a good idea to name their offspring “Adolf Hitler”; there is no legal problem if you wanted to name your son “Adolf” (perhaps after his great-grandfather) but for obvious reasons it's not exactly a popular name for children hereabouts today.

As far as surnames are concerned, the tradition is indeed that women who marry will adopt their husband's surname and that will also be the surname of any children resulting from the union. Today it is not uncommon for men to adopt their wife's surname instead, for either the husband or wife to adopt the other's surname as a “double name”, or for both husband and wife to just keep whatever names they were using before they got married. As a married couple you then need to decide what your children's surname will be, and all of your children must have the same one, but you can defer that decision until your first kid is officially registered. (If you then get divorced, revert to the surname you had before your marriage, and keep custody of your kid whose surname is that of your ex-spouse, it is possible for your surname to be completely different from your kid's surname, to everybody's confusion.)

In Germany in general, changing your first name and especially your surname (without getting married or divorced) is very difficult – not like in, say, the UK, where you can adopt a new name basically whenever you like by making an official declaration. This is probably because we like things to be nice and orderly here, and allowing people to have arbitrary name changes somehow doesn't feel right.

All of this of course means that things may indeed not be straightforward for people who arrive in Germany from places where the traditions are different. They're usually dealt with reasonably in the end. If you want to see a place with really strict naming rules, consider Iceland, where the state will force you to give your kids Icelandic names from an officially approved list (although to be fair you can submit new names to be added to that list), and where your surname derives from your parents' first name, so if you're Olaf and your dad is Erik, you'll be Olaf Eriksson but your son Thor will be Thor Olafsson.

The "Trojan Source" vulnerability

khim — Fri, 05 Nov 2021 10:59:28 +0000

> We do have a reputation for rampant bureaucracy but we're not that dense.

You may not be that dense, but you authorities… oh yeah. Please read that article, e.g., specifically this passage: you can apply for a letter from the Kenyan embassy in Berlin, then explains this tradition thus allowing you to name your child based on your traditions. Although Kenyans have more troubles because of their traditions, but believe me, it's not a simple as saying that you are Putina, but Russian and thus male child needs to have surname Putin.

Apply for the letter from the embassy! That's what is needed to resolve cases like these.

Now, I'm not saying that German authorities are completely heartless. Far from it. They would actually help you, suggest what forms would you need to fill, what letters from embassy to bring and so on… but naming child properly would still remain quite a quest.

P.S. Thankfully in case which I observed personally situation was not that dire: father was available, mother and father had the same Cyrillic surname (different from German law POV), so it was simple matter to requesting to pick surname of a different parent. But in case of a single parent and no official custody… or parents with different surnames… this may become quite a challenge.

The "Trojan Source" vulnerability

anselm — Fri, 05 Nov 2021 09:02:43 +0000

single mother can not get compatible documents in Germany and Russia because Germans would insist on unchanged family name while Russian authorities would expect changed one.

I'm in Germany and I can assure you that the concept of kids having different surnames than their parents is not unknown here. We do have a reputation for rampant bureaucracy but we're not that dense. In particular, it is absolutely possible in the eyes of the law for a single mother to have a surname that is completely different from that of her child (and that means completely different, not just an appended “a”).

The "Trojan Source" vulnerability

Cyberax — Fri, 05 Nov 2021 07:52:26 +0000

> E.g. if you write code for accounting program then you have to deal with the fact that many terms from local law can be translated and transliterated differently to US-ASCII

Then you place a fat big "#pragma ignore_unicode_rules" at the top of the file and affirm that you are OK with keeping all the pieces.

Everything else should limit Unicode-ness to string constants and comments.

This is already what's happening effectively in practice. For example, I tried to find a project on Chinese Github ( https://gitee.com/ ) that actually uses Chinese for identifiers and I couldn't find any. But there are plenty of Chinese-language comments. E.g.: https://gitee.com/dotnetchina/TimeCrontab/blob/master/src... (from the first project on their "Explore" page).

The "Trojan Source" vulnerability

khim — Thu, 04 Nov 2021 23:56:05 +0000

> Cyberax told something on which I agree: for the identifiers use only the ascii charset.

That's always possible but not always feasible. E.g. if you write code for accounting program then you have to deal with the fact that many terms from local law can be translated and transliterated differently to US-ASCII (basically the story of names discussed just above with a bit smaller amount of craziness).

In such case the ability to use the terms exactly as they are written in official documents is godsend.

Of course that means that programmer who can not understand the appropriate script wouldn't be able to easily edit such a program, but then it may be a blessing in a disguise: chances are high that such person wouldn't be able to understand requests made by officials, too, which means that it makes sense to separate that tricky part of code from the ORM or web-server sources anyway.

The "Trojan Source" vulnerability

khim — Thu, 04 Nov 2021 23:47:31 +0000

> I've even seen cases where members of the same family end up getting differently spelled family names.

And there are cases where you need differently spelled family name, but officials refuse to accept these.

Simplest example: Vladimir Putin have two daughters: Maria Putina and Katerina Putina.

Russians would say it's the exact same family name (it's Putin for male and Putina for female), but German authorities, of course, refuse to accept it — which means single mother can not get compatible documents in Germany and Russia because Germans would insist on unchanged family name while Russian authorities would expect changed one.

The "Trojan Source" vulnerability

tialaramex — Thu, 04 Nov 2021 21:32:53 +0000

Emojis also really helped with a case which might have eventually been solved by the dominance of UTF-8 but meanwhile sucked.

A bunch of software implemented UCS-2 ("Unicode" as it was conceived when it was strictly a 16-bit code in the 1980s) and then said, well, we've basically done Unicode. Before Emojis were popularized, anybody explaining why that's wrong is just saying technical mumbo jumbo you don't care about. But when your users can't write U+1F4A9 Pile of Poo suddenly the fact you're limited to the Basic Multilingual Plane jumps out as the problem it was all along.

Everything relying on MySQL is affected for example, because MySQL's "utf8" is an alias for "utf8mb3" aka "Not UTF8, but we hoped you wouldn't notice".

The "Trojan Source" vulnerability

mpg — Thu, 04 Nov 2021 21:28:07 +0000

Aw, why can't things just be simple for once?

Indeed, most locals won't know how to type those diacritics (most French people don't even know how to type É, À, Ç, which are supposed to be part of French writing), so they're gonna omit them, and we lose unicity again, in that perhaps one document will say your name is Chěng, another will have Cheng, and perhaps yet another is going to spell it Cheng3 (which IIUC is how you're supposed to indicate tone if you can't use the diacritics), and local officials are going to start questioning if those are really all the same...

Sigh.

The "Trojan Source" vulnerability

mpg — Thu, 04 Nov 2021 17:09:32 +0000

I had no idea, thanks for sharing, the article was an interesting read.

The "Trojan Source" vulnerability

nybble41 — Thu, 04 Nov 2021 16:30:05 +0000

> This may be getting a bit far afield from the issue of source code vulnerabilities, but the relevant point is that people have legitimate reasons for wanting to be able to write words, including possibly their own name, that include code points outside a single standard character set.

I think if you want to be that accommodating then you need to accept an image rather than mixed character sets. People may well have legitimate reasons for wanting to be able to write words, including possibly their own name, that include glyphs outside *any* standard character set. There isn't much you can do with something like that, however, beyond display it back exactly as it was entered. It's more reasonable to insist on Unicode everywhere for words which are actually intended to be understood by other people and/or processed by computers, and deal with any missing glyphs by getting them added to the standard.

The "Trojan Source" vulnerability

mpg — Thu, 04 Nov 2021 15:48:18 +0000

I mean, don't get me wrong: transliterations are terrible and I avoid them like the plague while learning Arabic, not least because as you say everybody's using a different system which is a pain in the neck. But outside the specific context of learning a language, I fail to see what the better alternative would be. So, I have two questions for you:

1. Say you're writing an article in English about the relationships between پاکِستان and भारत. How are you going to refer to the names of the various people involved?

2. If transliterating is the wrong thing to do, why do you think it's so common? I'm not saying the majority is always right, it often isn't, but as a general rule I think when disagreeing with the majority, it's good to be able to articulate what we think it is that all these people are missing.

The "Trojan Source" vulnerability

mpg — Thu, 04 Nov 2021 15:31:27 +0000

I fully agree that there's (usually) no such thing a universal "Latin" transliteration, and transliterations need to be language-specific. And I think that's actually what people are doing: recycling my little wikipedia experiment we can look at translations of https://en.wikipedia.org/wiki/Dmitri_Shostakovich - this time in Latin-based scripts - and observe the variety of transliterations used in different languages using (various extensions of) the same script.

> a transliteration only makes sense when targeting a specific language.

I don't disagree, but all texts I can think of target a specific language, just by virtue of being written in that language.

> Many transliterated names make zero sense because they are used in language where phonetics are different from the language they were transliterated to in the first place.

Yes, of course if people make the mistake of using a transliteration that was made for a different language than theirs, the results are not going to be good. But that's a pretty avoidable mistake, and again, that's not at all specific to transliteration: if I look at a Polish name, I have little idea how it's actually pronounced, regardless if the name is originally Polish or a Polish-oriented transliteration of a Russian name.

So, I stand by my opinion that transliterating (in a way that's appropriate for the target language) is the right thing to do most of the time for names of people in the context of a news article. (For names of command-line programs or websites, you probably want the original name instead or in addition. In the context of an encyclopedia, I think giving both a transliteration, the original name, plus an IPA transcription and when possible a recording, as wikipedia does, is ideal. In the context of administrative documents such as visas etc, honestly I don't know what the ideal solution would be: non-standard transliterations are a source of important problems, but I feel like using the original script would be a source of other problems; I think there's some irreducible complexity here.)

The "Trojan Source" vulnerability

Cyberax — Thu, 04 Nov 2021 07:58:11 +0000

> I hear Mandarin has an official romanization called pinyin, which I hope makes the administrative life of Chinese people in Latin script countries less painful.

Pinyin needs tonal marks (diacritics) to be understandable. And guess what gets omitted all the time?

The "Trojan Source" vulnerability

nim-nim — Thu, 04 Nov 2021 07:49:33 +0000

> So yes, transliterating is the right thing to do:

It is not because there is no such thing as an authoritative latin (for example) transliteration, different languages use the same symbols to encode different sounds, therefore a transliteration only makes sense when targeting a specific language.

Many transliterated names make zero sense because they are used in language where phonetics are different from the language they were transliterated to in the first place.

The "Trojan Source" vulnerability

nim-nim — Thu, 04 Nov 2021 07:35:39 +0000

Mixing scripts in text is very natural for anyone who speaks multiple languages; that leads to mixed scripts within “words” since computer languages are found of concatenation when constructing identifiers (an “evil” English-derived trait, also present in many human languages).

The "Trojan Source" vulnerability

sfeam — Thu, 04 Nov 2021 03:02:44 +0000

"copying the name from an application form into a computer system for processing" is not guaranteed to be something a typical local official can do in places like Japan and China where the set of characters used for names includes some that are not in any available font, or not in unicode at all. And even if there is a code point that would serve, the official may well not know how to enter it. See for example A Limitation on Names in the PRC".

This may be getting a bit far afield from the issue of source code vulnerabilities, but the relevant point is that people have legitimate reasons for wanting to be able to write words, including possibly their own name, that include code points outside a single standard character set.