Setting up international character support (Linux.com)

Let's start (I'm skipping nitpicks and I'm sure I'm missing a lot):

1. "Unicode" is not backwards compatible with ASCII. Unicode defines a number of encodings (the most common of which are UTF-8 and UTF-16/UCS-2). Of all the encodings, UTF-8 at least is backwards compatible. I suspect that CESU-8, UTF-7 and the bastardized Java UTF-8 are all also compatible. That being said, thinking that UTF-8 and Unicode are effectively synonymous in this day and age is wrong as Windows NT/2000/XP all use UTF-16 as their default.

2. UTF-8 is not called UTF-8 because "because it uses four octets per character". Number 1: It encodes bytes in *UP TO* 4 bytes. It encodes all of ISO 8859-1 in 1 byte/character for example. Besides, that's 4 bytes or 32 bits. Not sure where the 8 is coming from. It's called UTF-8 because it's basic unit has *SHOCK*, 8 bits.

3. Lack of glyphs in fonts is not UTF8's problem. In fact, it's kind of a benefit, because you can use fonts which only have the character sets you need. If unicode specified requirements for fonts (well out of it's authority I mught add), it would be something of a mess because all of our fonts would be 32MB big. If I only speak English, why do I want to waste memory storing glyph information for Icelandic, Japanese and Bengali?

4. Still on the subject of fonts, if he's such a king of Unicode, why didn't he point us to one of the unicode complete fonts that are available. MS Office's version of Arial is pretty close to Unicode complete. The free CODE2000 font is also pretty close to Unicode complete. Many unix systems will allow for a kind of font fallback, where it tries to render the glyph in the font you chose. If it can't, it falls back to the next font on the list and tries to get the glyph from there and so on until it finds it or gives up. Put CODE2000 at the bottom of the list, and no matter what font you're using, everything will still display.

5. From TFA, "But if all you want is the ability to type the occasional accent over a letter, then your best choice is a UTF-8 locale". No, taking advantage of deadkey keymaps is the best choice. That doesn't have anything to do with unicode per se, only the input subsystem. Given that he's probably talking about english speaker who occasionally want to type an ümlaut, ISO-8859-1 will also work just fine. Not that I'm advocating you use something other than UTF-8 mind you.

6. The author says you need to append ".utf8" to the locale name. Then he proceeds to use ".utf-8" and ".UTF-8" throughout the rest of the article. I work with a large number of linux distros and "UTF-8" always seems to work. Avoid the others. As for appending it in the xkblayout config line, I'm pretty sure this gets discarded. I have never used it, yet everything I do still ends up as UTF8. I have also made some of my own layouts and don't remember ever seeing that in the config files. Frankly, it's not the keyboard layout handler's job to deal with encoding. It's only concerned with what keystroke generates what character. In any case, the format in which X sends the bits to a program is going to be defined by the system locale so that's probably why I've never had to specify it. It just get's pulled from the environment.

7. I can't speak to the veracity of his statements regarding Gnome as I'm a KDE user.

8. In the section regarding dead keys, the author continues blurring the line between locale and input method. He says, "Both UTF-8 and many legacy locales support deadkeys". It is not the character encoding that supports the deadkeys, it's the keymaps. Allow me to state how this works: what you can type in and how you can type it is governed by kaymaps. What you can display and store is governed by locale. You may be able to type in an acute accented E because you keymap allows you to, but you get a question mark or a square because either the locale or the font cannot deal with the character. If the locale failing you (the question mark), the data is lost because the computer is incapable of storing it. If the font is failing you (the square), the data is still intact but simply cannot be displayed.

I realize he sort of point this all out, but he makes it sound like they are all the same thing, rather than 3 discrete systems with their own responsibilities. It's just sloppy, sloppy thinking.

In any case, I have a splitting headache right now and anime is starting so I'm going to leave it here. Suffice it to say my issues with this article keep right on going and going. Some clearer thinking and some better research/understanding would have resulted in a better article. Thank goodness he didn't start in about CJK input.

The last point I'd like to make is regarding his statement "probably the greatest annoyance is the need to press the apostrophe key followed by the space bar to type a straight quotation mark". If you use AltGr to move the deadkeys up into the 3rd shift group, then you can type just as though it were a normal US keyboard but still have access to all your extended characters.

To finish that point allow me to demonstrate. My keyboard is a standard us keyboard using a slightly modified keymap and an IM. Anyone using my keyboard would be totally unaware that it wasn't standard. I can type these without compositing:

äåé®þüúíóö«»¬áßðfghjkø¶æ©ñµçÄÅÉ®ÞÜÚÍÓÖÁ§ÐذƢѵÇ

With compositing (via AltGr which is mapped to right-alt on my keyboard), theres a lot more. And then of course I can bring the IM into the picture and do 10's of thousands of CJK symbols. I'd type some but LWN seems to filter them out (which is to say, convert them to things like す which is a Japanese katakana "su").

Anyway, take the article with a grain of salt (or maybe a salt lick), but be sure to always use UTF-8 when you get the chance.

P.S. Sorry if there are any grammatical or spelling errors in this one. I need a tylenol and don't feel like proofreading for another 20 minutes before I get it. Besides, there's anime afoot!

It confuses a locale (information about the user's culture and language, e.g. paper sizes, currency, calendar) with a character set (which is just a bunch of characters, possibly an ordered set) and both of those with a character encoding (a way to turn characters into byte sequences). The last bit is understandable, even early IETF documents confuse character encodings with character sets, because back then the two were often synonymous.

en_GB is the name of a UK locale, with English language, A4 paper, £ sterling currency, the first month is called "January" and decimals are separated with a point not a comma.

UTF-8 is a character encoding, and it encodes the character set ISO 10646 which is sometimes called "Unicode" since it's identical to the Unicode character set, although Unicode.org standardises many things beyond the ISO 10646 character set.

A previous poster pointed out numerous additional errors. If you don't know anything about i18n, Unicode or UTF-8, stay clear of this article. If you do know something, be prepared to be annoyed by it. Next time we see an article which says Linux is "not for commercial use" or that the GPL is "untried, and probably illegal" we should consider this article and remember that incompetence is a more common explanation than maliciousness.

*** ./us.orig Fri Feb 10 11:23:46 2006
--- ./us.nrb Fri Feb 10 11:12:51 2006
***************
*** 71,88 ****
partial alphanumeric_keys
xkb_symbols "intl" {

! name[Group1]= "U.S. English - International (with dead keys)";

include "us(basic)"

// Alphanumeric section
! key <TLDE> { [dead_grave, dead_tilde, grave, asciitilde ] };
key <AE01> { [ 1, exclam, exclamdown, onesuperior ] };
key <AE02> { [ 2, at, twosuperior, dead_doubleacute ] };
key <AE03> { [ 3, numbersign, threesuperior, dead_macron ] };
key <AE04> { [ 4, dollar, currency, sterling ] };
key <AE05> { [ 5, percent, EuroSign ] };
! key <AE06> { [ 6, dead_circumflex, onequarter, asciicircum ] };
key <AE07> { [ 7, ampersand, onehalf, dead_horn ] };
key <AE08> { [ 8, asterisk, threequarters, dead_ogonek ] };
key <AE09> { [ 9, parenleft, leftsinglequotemark, dead_breve ] };
--- 71,88 ----
partial alphanumeric_keys
xkb_symbols "intl" {

! name[Group1]= "U.S. English - International (with shifted dead keys)";

include "us(basic)"

// Alphanumeric section
! key <TLDE> { [ grave, asciitilde, dead_grave, dead_tilde ] };
key <AE01> { [ 1, exclam, exclamdown, onesuperior ] };
key <AE02> { [ 2, at, twosuperior, dead_doubleacute ] };
key <AE03> { [ 3, numbersign, threesuperior, dead_macron ] };
key <AE04> { [ 4, dollar, currency, sterling ] };
key <AE05> { [ 5, percent, EuroSign ] };
! key <AE06> { [ 6,asciicircum, onequarter, dead_circumflex ] };
key <AE07> { [ 7, ampersand, onehalf, dead_horn ] };
key <AE08> { [ 8, asterisk, threequarters, dead_ogonek ] };
key <AE09> { [ 9, parenleft, leftsinglequotemark, dead_breve ] };
***************
*** 109,115 ****

key <AC09> { [ l, L, oslash, Ooblique ] };
key <AC10> { [ semicolon, colon, paragraph, degree ] };
! key <AC11> { [dead_acute, dead_diaeresis, apostrophe, quotedbl ] };

key <AB01> { [ z, Z, ae, AE ] };
key <AB03> { [ c, C, copyright, cent ] };
--- 109,115 ----

key <AC09> { [ l, L, oslash, Ooblique ] };
key <AC10> { [ semicolon, colon, paragraph, degree ] };
! key <AC11> { [apostrophe, quotedbl, dead_acute, dead_diaeresis ] };

key <AB01> { [ z, Z, ae, AE ] };
key <AB03> { [ c, C, copyright, cent ] };

That's it. Just moved the dead keys into the upper shift group so you have to use AltGr to type them. I did this, because I totally agree with the author that having to contend with dead keys *all the time* when you only want accented characters once in a while is highly annoying.