printk() indexing [LWN.net]

printk() indexing

Posted May 28, 2021 22:24 UTC (Fri) by Cyberax (✭ supporter ✭, #52523)
In reply to: printk() indexing by flussence
Parent article: printk() indexing

Yeah. gettext() is the best way to do message i18n, but for some reason this whole approach got forgotten.

to post comments

printk() indexing

Posted May 29, 2021 2:43 UTC (Sat) by pabs (subscriber, #43278) [Link] (19 responses)

I hear Mozilla's Project Fluent provides a better option to do i18n than gettext:

https://projectfluent.org/

printk() indexing

Posted May 29, 2021 2:55 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (18 responses)

I dislike it.

The main feature of gettext() is that your messages are right in the source code. You don't have to invent names and then put the messages into a separate file. This makes the l10n much easier to get into, just put T() around the text and you're done.

printk() indexing

Posted May 29, 2021 3:08 UTC (Sat) by pabs (subscriber, #43278) [Link] (1 responses)

That aspect is definitely more convenient for English-speaking developers, but it sounds like Fluent's approach results in better outcomes for people viewing the translations?

printk() indexing

Posted May 29, 2021 3:17 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Not really. gettext supports the same features: multiple plural forms, inflections, etc. In much the same way as Fluent, actually. I'm a native Russian speaker, so I've actually encountered most of the translation-based complications.

See here: https://www.gnu.org/software/gettext/manual/html_node/Tra...

gettext definitely shows its age, the PO files are poorly designed, the toolchain is somewhat clunky, etc. You can just feel that it's software from 90-s. But the core feature of NOT externalizing the messages is still genius.

printk() indexing

Posted May 29, 2021 11:12 UTC (Sat) by mpr22 (subscriber, #60784) [Link]

> The main feature of gettext() is that your messages are right in the source code.

I hate that and refuse to use gettext() in the idiomatic manner recommended by the authors of its documentation.

(I hate having to rebuild just because I changed the text of a message that isn't being machine-parsed.)

printk() indexing

Posted May 30, 2021 15:35 UTC (Sun) by Jandar (subscriber, #85683) [Link] (14 responses)

> The main feature of gettext() is that your messages are right in the source code

In my eye this is the worst feature of gettext. As this message string is essentially a key into the translation database it has to stay the same. You want to fix a typo, not possible in the source code. Now there a two places to look for the real output string. Any gettext message you have to look up in the po file to see if there is a changed version. While the source code *looks* like a real message it may be an output string or may be not. Worst design ever.

printk() indexing

Posted May 30, 2021 17:00 UTC (Sun) by madscientist (subscriber, #16861) [Link] (10 responses)

Gettext is the best design. Being allowed to keep the same key but change the text is terrible. It's much better to have a correct output message even if it's in a different language, than it is to have a wrong output message in the expected language.

The FOSS software I maintain uses the translation project and has 30 translations. It's totally infeasible for me to coordinate (read: wait for) all those translations to be updated by 30 different translation teams, some of whom are just one person with limited time.

It's also better to be able to print some message than just some random error code, if somehow the translation catalog is lost.

These days, with the internet, it's straightforward enough to translate messages on your own which were not translated automatically for some reason.

printk() indexing

Posted May 30, 2021 18:27 UTC (Sun) by mathstuf (subscriber, #69389) [Link] (9 responses)

What prevents having a list of fallback languages if the preferred one doesn't have it in the key-lookup-for-all-languages? For example, pt_PT -> pt_BR -> en_US (having "everything").

printk() indexing

Posted May 30, 2021 20:11 UTC (Sun) by madscientist (subscriber, #16861) [Link] (8 responses)

Nothing at all. It's even already implemented ... if you're using GNU gettext.

See https://www.gnu.org/software/gettext/manual/html_node/The...

I'm not sure I see how this relates to my comment though.

printk() indexing

Posted May 30, 2021 20:17 UTC (Sun) by mathstuf (subscriber, #69389) [Link] (6 responses)

Gah, I was thinking about it backwards. I now see what you mean. Since the key is the "default text", if it changes, the translations automatically need changed as well. Versus where the key is some ID where if the default text is updated, the other translations need some marker to indicate that they need to sync up.

It's not an easy problem. How does Gettext deal with text which is spelled the same but has different meanings depending on context? Let's say there's a button with the text "Bark" on it. How is it indicated that one wants "dog bark", "bark a shin", or "tree bark" with the default text as a key? I assume there is some way to plumb some kind of intent message through to the interface translators use?

printk() indexing

Posted May 30, 2021 21:17 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

You can just put Bark(Tree) in the message and provide translation for English.

printk() indexing

Posted Jun 1, 2021 13:28 UTC (Tue) by flussence (guest, #85566) [Link]

I've even proposed that as good practice somewhere before. Just in case people need more selling on the idea:

The source code naturally contains LC_MESSAGES=C strings; those don't need to be written for anyone but the author and can shamelessly be optimised for high jargon density and minimum mental context switching. It gives a motive to write a human translation later, and even if that's just en_US with few changes you've now made a map of the terrain for anyone else to work with. There's a long tail of other benefits besides (such as, the one I started the thread with).

printk() indexing

Posted May 30, 2021 21:27 UTC (Sun) by mbunkus (subscriber, #87248) [Link] (3 responses)

The gettext system knows a concept called "Contexts". This is basically a second parameter for the translation functions that works as a disambiguation. You'd have calls to 'pgettext("dog", "bark")' and 'pgettext("tree", "bark")', and both would show up in PO files as two separate entries to translate. How you name contexts is totally up to you.

Contexts are optional. The most basic function, 'gettext("bark")', works without a context.

See here for more details: https://www.gnu.org/software/gettext/manual/html_node/Con...

printk() indexing

Posted May 31, 2021 12:30 UTC (Mon) by eru (subscriber, #2753) [Link] (2 responses)

That requires original authors to use contexts assiduously, because they may not even be aware when a translation depends on it. In Microsoft's web mail, the English word "Empty" was used in two senses (something is empty, or as a command to empty a folder). The Finnish localisation seemed to have been made with a simple string substitution, and used a word that was appropriate only for the first meaning. (This has since been fixed).

printk() indexing

Posted May 31, 2021 14:09 UTC (Mon) by mbunkus (subscriber, #87248) [Link]

Yeah sure, but good knowledge about the pitfalls of translation (or in general: localization) is always required for good translations. The question I answered was "how does gettext deal with terms that have different meanings in different contexts", and in order to solve this in any way, shape or form, requires awareness of the possibility of different meanings. No matter the translation system in use.

Systems that use symbols instead of real text in the source files also require the author to have that awareness so that the author can use two different symbols in different places, even if both symbols' corresponding text in the original language is the same.

Localization is a complex topic. You cannot magic complexity away. All systems add to that complexity in certain ways, some more than others. Some systems only seem to do away with part of the complexity, but at the cost of flexibility and expressiveness, leading to awkward or wrong translations.

printk() indexing

Posted May 31, 2021 18:07 UTC (Mon) by madscientist (subscriber, #16861) [Link]

Localization / internationalization is hard, there's no doubt about it. Successful i18n is an ongoing conversation between the code maintainer and the translator about what is needed vs. what is feasible.

There's no way every maintainer can know enough about all the different languages and their needs to get everything right the first time; that's not the goal. The maintainer's responsibility is to be receptive to the issues raised by the translator to make their job easier and a good translation possible.

printk() indexing

Posted May 30, 2021 21:58 UTC (Sun) by madscientist (subscriber, #16861) [Link]

Thinking about this I'm not sure this variable actually does what you (and I!) were hoping though. I haven't tried it and the documentation is not clear enough.

What I was hoping is that this list is consulted on a per-message basis, so that if a given message was not available in one language it would fall back to another. But maybe this is not the case: maybe it will open only the first catalog of your preferred languages that exist, then stop.

So if the preferred catalog exists but is missing some messages, then you'll get the default message (whatever is printed in the "C" locale) not the fallback.

I could be wrong: I didn't test it.

printk() indexing

Posted May 30, 2021 19:48 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

gettext supports fuzzy matching, so a small typo is not a problem. I also remember a tool that looked at PO files and automatically tried to update the strings for these kinds of changes.

It's not at all different from making a typo in the message name.

printk() indexing

Posted May 31, 2021 2:15 UTC (Mon) by Gaelan (guest, #145108) [Link] (1 responses)

That seems dangerous. If I change "CPU status: fine" to "CPU status: fire", I certainly don't want translations to default to the old translation, even though they're "only" one character apart.

printk() indexing

Posted May 31, 2021 3:57 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Honestly, this is kind of theoretical in practice. I don't remember exactly, but gettext did warn about possible ambiguous matches during PO file compilation.

A modernized gettext() can be designed to raise an error if there are two messages that can be ambiguously fuzzily matched with each other.