Unicode normalization
Unicode normalization
Posted Jun 2, 2006 9:47 UTC (Fri) by stijn (subscriber, #570)In reply to: Unicode normalization by tialaramex
Parent article: GNU grep's new features (Linux.com)
At one time I worked at PICA in the Netherlands. Together with a colleague (Geert-Jan van Opdorp) I worked on implementing Unicode search with support for wild cards etc. We made extensive use of the icu libraries (IBM Internationalization Code for Unicode), and IIRC Geert-Jan implemented Udi Manber's (et al) search algorithms to work with Unicode. That was quite a feat. All this was of course in the context of indexes and indexes to indexes, and it built on the already existing infrastructure. But it is doable.
One of my previous projects was the development of a macro language + its processor. It is currently byte (and even ASCII) based. Someone once enquired about 'Unicode support'. I still wonder what possible meanings Unicode support could take on in that context, and I wonder to what extent Unicode should permeate the command line. IIRC perl6 might have Unicode tokens. Is that sane (whether it's true or not)? I am attracted to the idea of keeping Unicode for content, but perhaps that assumes a distinction that cannot be maintained.