LWN.net Weekly Edition for September 9, 2016

Untangling character sets and Unicode blocks

At TypeCon 2016 in Seattle, font engineer Richard Fink kicked off the conference program with a talk entitled "Empty Space Characters In Modern Character Sets" that was, ostensibly, a look at the over two dozen space characters defined in Unicode. But that topic was merely a pretext, leading into a broader examination of how the Unicode character blocks fail to align with languages and writing systems, which makes it difficult to define "language support" in a meaningful way.

Space team

There are, as Fink said, a lot of code points defined in Unicode that amount to some form of white space. But some of them behave differently than others—such as the regular space versus the non-breaking space. And some of them have important semantic distinctions for certain writing systems or scripts, such as the zero-width joiner versus the zero-width non-joiner, which (in connected scripts like Arabic) tell the text rendering engine which variants of two adjacent letters should be used.

There are also quite a few other space characters that differ only in their width. Fink got into the type world to do high-quality web design, he said, and he soon realized that most fonts on the market left out these space characters, which made it impossible to make fine-grained adjustments to complicated layouts by inserting, say, a hair space (code point U+200A) or a medium mathematical space (U+205F).

That was rather frustrating to discover, since all of the necessary glyph slots in the font table could easily be automatically generated. Some time later, Fink worked on the font subset-generator tool on FontSquirrel, which lets users select the sets of glyphs they want for a font and build a minimal font file containing just the glyphs of interest. He did, at that point, add a feature that would automatically generate the missing space characters.

But the encounter prompted him to look more closely at the peculiarities of Unicode. Most of the white space characters, he said, are defined in the "General Punctuation" block, which hardly makes sense, since empty space is not punctuation. Then again, the names of the other Unicode blocks are just as obtuse, featuring, for example, the ambiguous distinction between "Latin Extended" and "Latin Supplement."

That ambiguity makes it hard for two people (or two programs) to agree on what it means for a font to "support" Latin, Cyrillic, Devanagari, or any other writing system. It is even more ambiguous to define what characters are required to support a specific language. And whenever a font does not support the characters that need to appear on screen, the user gets garbled text, either "tofu" (an assortment of box symbols ), question marks, or mojibake gibberish.

Software vendors and font buyers alike would prefer to know whether or not a font support the right glyphs before they start using it. Fink recently started working with the Google Fonts project, and set his mind to finding a definitive set of standards the project could use to validate which fonts supported which scripts and languages.

Character sets

The first alleged solution in the type industry is to look outside of Unicode at character sets, which software companies have historically used to establish their own definitions of what it means to support a language or a writing system. Microsoft had one called Windows-1252 to define basic Latin-alphabet support; Apple had MacRoman, Adobe had Adobe Latin 1 through Adobe Latin 5.

But none of these character sets are an official standard; the proprietary OS vendors created their own as a matter of course because they were competitors. Those character sets are also quite old at this point and are not regarded as particularly relevant. In practice, the main alternative is Adobe's character sets, which often get used and referred to by third parties. But Adobe's sets change, seemingly without warning. Fink said that he has found multiple, conflicting versions of the Adobe character sets published as reference material in a variety of places online.

The trouble is that Adobe maintains the lists for use by its designers and developers, updating them when it needs to. But Adobe has started hosting its character-set definitions in a public repository on GitHub, so Fink got in touch with some current and former members of the type team. However, they made it clear, he said, that Adobe is not interested in maintaining the character sets as a standard or specification, either. Officially, the lists are for internal usage only and the company wants the flexibility to alter them at any time. It was not even interested in version-numbering the updates to the lists, although Fink noted that changes could be referenced by Git commits.

Adding to the trouble is the fact that Microsoft, Apple, and Adobe are all US-based companies, Fink noted, and with that comes an English-centric mindset. Languages and scripts that are "foreign" to Americans get short-changed, and the relevant character sets are more likely to contain omissions or errors. And, again, the naming of the character sets left something to be desired. "'Latin 1'? Why would I care about that; I don't do any work for the Roman Catholic Church," he observed.

Design perspective

The other tactic Fink pursued was to ask type designers what they used as their guide for implementing language support. In February 2016, he posed that question on the TypeDrawers discussion forum. He received scores of replies, but the only take away, he said, was "nobody really agrees on anything." But he did learn that type designers grapple with the same problem from a marketing angle; it is hard to advertise a font based on its Unicode-block coverage, after all.

Thus, Fink eventually decided that it was time to sit down and think about defining new character sets—and, hopefully, to define character sets without a strong US-bias. "I'm unhappy with the term 'multi-script'," which is historically used to describe fonts that offer broad language support, he noted. "Because the word 'script' has too many other definitions, particularly on the web. So I prefer 'trans-national' character sets."

Work on the character-set project is being done in the Google Fonts GitHub repository. So far, the Latin sets are defined in four cumulative levels: Core, Plus, Pro, and Expert, with each intended to enable support for a specific set of languages or use-cases. There is also a group of Cyrillic character sets (Core, Plus, Pro, and Historical); discussions are underway for other languages and the process is open to outside contributors.

Each set pulls from a variety of Unicode blocks, including utility glyphs like arrows and geometric shapes in addition to alphanumeric characters. Notably, they also include several glyphs that do not map to separate Unicode code points, such as small capitals, certain ligatures, and small numerals for scientific subscripts and superscripts. As one would expect, the space characters are included. The character sets also suggest a standardized naming scheme for the glyphs within a font's internal tables, although those names are not generally exposed to end users.

Defining support by language (rather than by script) was one of the better decisions made in the development of the FontSquirrel subset-generator, Fink said. It maps better to what the average user thinks of when evaluating the capabilities of a font. In addition, Fink noted that Dutch Type Library's OTMaster program, "without which I don't know if my life would be livable," also considers language support to be the feature of interest.

Fink ended the talk by observing that applications have some catching up to do where non-standard glyphs are concerned. Adobe InDesign is the only application he has found that even allows the user to select and insert the Unicode space characters. Web applications are underpowered in this respect, too, he said. As of HTML 5.0, all of the space characters have a named HTML character entity reference (apart from the ability to enter a character by its Unicode code point, that is). While HTML 4 supported 252 named entities, HTML 5 defines more than more than 2,000, and all major browsers understand them. It remains up to developers to make use of that feature.

Comments (39 posted)

What's next for Apache OpenOffice

Concerns about the viability of the Apache OpenOffice (AOO) project are not new; they had been in the air for a while by the time LWN looked at the project's development activity in early 2015. Since then, though, the worries have grown more pronounced, especially after AOO's recent failure to produce a release with an important security fix nearly one year after being notified of the vulnerability. The result is an internal discussion on whether the project should be "retired," or whether it will find a way to turn its fortunes around.

The current chair of the AOO project management committee (PMC) is Dennis Hamilton, whose term is set to end shortly. He has been concerned about the sustainability of the project for some time (see this message from one year ago, for example), a concern sharpened by the routine requirement that he report to the Apache Software Foundation (ASF) board on the project's status. The board, seemingly, had asked few questions about the status of AOO until recently, when the handling of CVE-2016-1513 (or the lack thereof) came to its attention. Now the board is apparently asking some sharp questions indeed and requiring monthly (rather than every three months as usual) reports from the project. "Retirement" of the project, it seems, has been explicitly mentioned as a possibility.

Pondering retirement

In response, on September 1, Hamilton went to the AOO development list with a detailed description of what retiring the project would involve. He said:

Given that, he said, it is time to openly face the prospect that AOO is not sustainable and needs to be wound down. ASF board member (and corporate vice president) Jim Jagielski added that "it has become obvious to the board that AOO has not been a healthy project for some time." Given that, he said, the important thing is to figure out what is to be done now.

This conversation has been widely reported; the result, unsurprisingly, has been a strong "we told you so" response. There has been quite a bit of rehashing of the history of the AOO project and how the current situation came to be. But Jagielski had a point when he said that most of that does not matter; what does matter is what happens next. Your editor would agree. There may be an opportunity to make things better for developers and users of free office suites, but doing so may require forgiving and forgetting quite a bit of unpleasant history.

Jagielski suggested that, while the project cannot sustain itself as an end-user-focused effort, it may be able to go forward as a framework that others could build applications with. Who those others would be was not specified. He suggested that the OpenOffice.org domain could be redirected to LibreOffice — a suggestion that still seems to be seen as heretical by many in the AOO project.

To top things off, he even said that the project might consider making an apology of sorts for the excesses of one of its advocates in the past. While his tone might be read as being less than fully sincere, one would hope that any such apology, should it be forthcoming, would be taken at its word and accepted fully. LibreOffice developers could even consider making an apology of their own (they have not been 100% perfect either), perhaps ahead of anything from AOO. The sharing of some conciliatory words could do a lot to end the hostilities of past years, enable cooperation, and bring about a solution that is good for everybody involved.

Pondering non-retirement

The above discussion, like much of the conversation on the net as a whole, has an underlying assumption that AOO will, indeed, wind down. The project has been unable to compete with LibreOffice; its commit volume and developer participation are not just lower, they are two orders of magnitude lower. LibreOffice makes regular feature and maintenance releases; AOO has been unable to put out even a single emergency security-fix release. LibreOffice has significant corporate investment driving its development; AOO, seemingly, has none since IBM ceased its involvement. The current AOO development community is too small to even hope to fully understand its (large) code base, much less make significant improvements to it. So, to many, it seems clear that AOO is not sustainable.

There are, however, AOO developers who disagree with this assessment. Some of them clearly think that AOO can be saved and saw Hamilton's message as something close to an act of sabotage. Others saw it as "liberating" and as a way to kickstart the process of bringing in new developers. AOO's remaining community has a clear attachment to the project and a strong lack of desire for any kind of accommodation with LibreOffice. It was said by a few that AOO provides an important competition for LibreOffice, though what form that competition takes when the project has not made a feature release for 2.5 years is not clear.

What is clear is that AOO needs to bring in more developers if it is going to survive. To that end, a new recruitment mailing list has been created to help new developers get started. Plans are forming to try to turn the current round of publicity into a call for developers to join a reinvigorated AOO project. Jagielski has claimed that "AOO has simply been overloaded w/ emails from developers and other contributors offering their help, skills, talents and support" since the discussion started, but that overload is not yet evident on the mailing lists.

The developers involved must surely be aware of just how big a challenge they face. Many office-suite users may not have heard of LibreOffice, but the development community is well aware of the relative health of the two projects; attracting them will continue to be hard. The state of the code base, which has not seen the sort of energy put in by LibreOffice to make development easier, will not help. There is no financial investment going into AOO and, seemingly, no plans to try to attract any; if a company did come in, it would likely end up dominating the project in short order — a situation with its own problems.

AOO also has an interesting problem that is not shared by many other projects: it is subject to the decisions of a foundation board of directors that is concerned about potential damage to the overall Apache brand. Even if the AOO developers feel they are making progress, the possibility of the board pulling the plug on them is real. As board member Greg Stein recently said, "PMCs do *not* want the Board involved ... it rarely turns out well for them". Should such a thing happen to AOO, its developers would still have the source, of course, but would lose access to the ASF's resources and, probably, the OpenOffice brand. With such a cloud hovering over them, it is not surprising that AOO developers are concerned and asking for more time.

So the challenge is real. But one thing should be kept in mind here: the free-software community is famous for proceeding in the face of "you can't possibly accomplish that" criticism and proving the critics wrong. A quick look back at what was said about the GNU project in the 1980s, or Linux in the early 1990s, will drive that lesson home. One should never underestimate what a small group of determined developers can do. If the AOO developers think they can muster the resources to manage their code base, they should probably be given the space — by the ASF board and the world as a whole — to try.

The board, though, would be right to want some specific milestones to demonstrate that the project is back in good health and able to meet its users' needs. Apache OpenOffice cannot continue to distribute software that it cannot fix and cannot keep secure, using a well-known trademark to keep up a user base that may be unaware that there is little underneath. One way or another, that is a situation that needs to be fixed.

It is rare for the community to talk overtly about shutting down a project; usually, projects either just fade away or they are killed by a parent company with little warning. It is the type of discussion that we naturally tend to shy away from. But, like people, projects have a life cycle and do not last forever. Facing up to that cycle can motivate developers to rejuvenate a project or, at worst, can help minimize the pain caused by a shutdown. The AOO project has thus begun an important conversation; quite a bit rides on how it ends.

Comments (146 posted)