Unicode normalization [LWN.net]

Unicode normalization

Posted Jun 1, 2006 21:35 UTC (Thu) by mjr (guest, #6979)
Parent article: GNU grep's new features (Linux.com)

I've yet to need it myself, but for a while it's been on my mind that grep should really be able to normalize Unicode strings for search purposes. After all, often we'd like to get at matches incorporating the given characters, not just the literal octet streams.

Probably not a problem most of the time, and one can always normalize the files separately. A bit cumbersome, though.

to post comments

Unicode normalization

Posted Jun 1, 2006 22:27 UTC (Thu) by tialaramex (subscriber, #21167) [Link] (9 responses)

Once you start thinking along these lines you really want a completely localised search feature. Unicode normalisation only reduces identifiable Unicode characters to a single representation (e.g forcing LATIN CAPITAL LETTER E followed by COMBINING ACUTE ACCENT to LATIN CAPITAL LETTER E WITH ACUTE, or vice versa) in a reproducible manner. That's maybe useful in a search program, but it's not enough to make it suitable for many non-English and particularly non-European languages. IMO it's fine to provide a separate tool for pre-processing text into one of the accepted Unicode normalisations.

Some languages have minor distinctions between characters (think like "case sensitivity" in English) that sometimes need to be ignored when searching. Some have special rules that treat several characters as one in certain circumstances (e.g. imagine if "qu" was not treated as a "q" and a "u" in English, but as a unit "qu", while both "q" and "u" continued to exist separately in other words so that a search for "u" would not match "queen"). Some have the reverse (imagine if English treated W the same as VV so that a search for "veek" would match "week")

On the whole this is a big enough can of worms to deserve a completely new piece of software, one specifically aimed at locale sensitive searching.

Unicode normalization

Posted Jun 1, 2006 22:36 UTC (Thu) by mjr (guest, #6979) [Link]

Fair points. Drawing the line would perhaps be less straightforward than I thought.

Unicode normalization

Posted Jun 1, 2006 23:00 UTC (Thu) by kingdon (guest, #4526) [Link] (2 responses)

Sure it is complicated, but is anyone really doing much work on the problem (either in grep or in a separate tool)?

Google might have something of the sort. I know I've searched for non-ASCII strings but haven't played extensively with things like a-with-an-accent (as one character) versus a plus accent-which-combines (as two characters).

But if Lucene does anything like this, the Lucene FAQ doesn't seem to say so (it just says that Lucene uses Unicode and doesn't elaborate).

Oh, and having the search behave differently based on locale is the wrong approach (IMHO). It is a common case that you have a lot of documents, some in one language, some in another, and some in more than one. Sure, giving up locales might cause you to lose some rules where language A treats character X one way, and language B treats it differently (hopefully obscure, but I'm not expert enough to say). Most of the time it would work to just look at the characters in the document and the search string, and ignore the locale.

Unicode normalization

Posted Jun 2, 2006 6:40 UTC (Fri) by MortFurd (guest, #9389) [Link]

Google does a decent job with that kind of thing - at least for what I do.

German has vowels with the umlaut (the two dots above the character.) The standard way to type these on a key board that doesn't have the umlauted characters is to substitute a two character combination (ae for umlaut a, ue for umlaut u, etc.) Google properly find words containing the umlaut characters, and also find matches to the double cahracter substitute if you give it an umlaut (my home computer has a german keyboard, my work computer has an amercain keyboard, so I get to see both sides of the problem.

Unicode normalization

Posted Jun 2, 2006 9:40 UTC (Fri) by ibukanov (subscriber, #3942) [Link]

> Google might have something of the sort.

It is not necessary for Google to know anything about combined characters etc. since Google search is strictly a word search. So they just need to assemble the list of all forms for particular word and map them to the same index entry.

Unicode normalization

Posted Jun 2, 2006 6:11 UTC (Fri) by dvdeug (guest, #10998) [Link]

Unicode normalization may not be enough for localized searching, but it's the only _correct_ way to search Unicode text. LATIN CAPITAL LETTER E followed by COMBINING ACUTE ACCENT is the exact same thing as LATIN CAPITAL LETTER E WITH ACUTE according to the Unicode standard, and a program that will match one and not the other is not conforming to the standard. It's not unreasonable to ask for grep to at least provide an option to conform to the standard and work the way that users expect? Only a character set geek will understand why those two items don't match, and only such a person should have to understand that.

Unicode normalization

Posted Jun 2, 2006 9:47 UTC (Fri) by stijn (subscriber, #570) [Link] (3 responses)

At one time I worked at PICA in the Netherlands. Together with a colleague (Geert-Jan van Opdorp) I worked on implementing Unicode search with support for wild cards etc. We made extensive use of the icu libraries (IBM Internationalization Code for Unicode), and IIRC Geert-Jan implemented Udi Manber's (et al) search algorithms to work with Unicode. That was quite a feat. All this was of course in the context of indexes and indexes to indexes, and it built on the already existing infrastructure. But it is doable.

One of my previous projects was the development of a macro language + its processor. It is currently byte (and even ASCII) based. Someone once enquired about 'Unicode support'. I still wonder what possible meanings Unicode support could take on in that context, and I wonder to what extent Unicode should permeate the command line. IIRC perl6 might have Unicode tokens. Is that sane (whether it's true or not)? I am attracted to the idea of keeping Unicode for content, but perhaps that assumes a distinction that cannot be maintained.

Unicode normalization

Posted Jun 2, 2006 10:32 UTC (Fri) by arcticwolf (guest, #8341) [Link]

If you use utf8, Perl 5 already allows you to use Unicode in identifiers, for example, actually.

perl

Posted Jun 2, 2006 12:40 UTC (Fri) by niner (guest, #26151) [Link] (1 responses)

Even perl 5 supports unicode in program text (not just string constants, but all identifiers). Just use utf8;

perl

Posted Jun 2, 2006 13:10 UTC (Fri) by stijn (subscriber, #570) [Link]

I was thinking of this:

.. a version with French quotes «» that does interpolation before splitting into words

Which is taking it a little step further. I am sure I don't like it.

(The comment editor does not let me enter «» alas so I cut and pasted French quotes into the comment - apparently that works)