LWN.net Weekly Edition for August 29, 2013

Adobe's open source font experience

The fifteenth annual TypeCon conference was held in Portland, Oregon on August 21–25, featuring a mix of session topics that encompassed type design, letterpress printing, and modern font software. Open source and open font licensing have become hot topics in recent years—largely due to the rise of CSS web fonts. But the signs are that open source is gaining even broader acceptance, as evidenced by Paul Hunt and Miguel Sousa's presentation on the Adobe Source Sans Pro and Source Code Pro fonts—and the ripple effects they triggered within the company.

Hunt and Sousa are both part of Adobe's Type Team, which produces both software tools and typefaces. Historically, both categories of work have been proprietary, which is part of what made the 2012 debut of its open font families significant. The pair spoke about the development of the project, from the company's internal motivations to how interacting with the community affected the team's workflow and technical decisions.

After giving a general overview of open source concepts and development processes, Hunt discussed the decision to create an open font in the first place. The company is a type foundry (selling its own fonts), and it runs the web font subscription service TypeKit. But, as Hunt explained it, Adobe created the Source Sans Pro font because the company has been developing more open source software in recent years, and it needed fonts that it could incorporate into its releases. In fact, he said, Source Sans Pro was first used in Strobe Media Playback, an open source web video-player project. Since then, the fonts have also been central to Adobe's Brackets, an open source, browser-based HTML editor, and have been deployed in several other projects.

Initially, Hunt said, the company had considered taking one of its existing commercial typefaces and releasing it under open source terms, but the team eventually decided to create something new. Source Sans Pro was inspired by Morris Fuller Benton's venerable News Gothic and several of its contemporaries, he said, and was optimized to be used in software user interfaces, while still being readable in long blocks on continuous text. The initial release was made in early August 2012, in 12 styles across six weights. In September, a second typeface was added: Source Code Pro, which is a monospaced font family designed for use in coding.

Tooling and retooling

As Sousa explained, the fonts were developed in the (proprietary) FontLab Studio font editor, using Adobe's Multiple Master format (which greatly simplifies adding new font weights), and built with the help of Adobe Font Development Kit for OpenType (AFDKO), which is a suite of scripts and command-line utilities. Internally, the team managed the font sources with the company installation of the Perforce revision control system.

The team initially released the fonts on its SourceForge.net page, providing TrueType and OpenType binaries plus the FontLab Studio source files, together in one downloadable .ZIP archive. The fonts were also made available through a number of font services, including TypeKit and Google Fonts. The reaction was overwhelmingly positive; Hunt commented that the release announcement remains the most-viewed post on the team's blog to this day, and there were a number of stories about the project in the general tech press.

Users even began reporting bugs quite early on in the process. But there were other comments, too—most notably criticism of the location and structure of the releases themselves. Or, as Hunt put it, there were comments of the form "you fools; open source development takes place on GitHub." But that characterization was tongue-in-cheek; such comments were not reprimands, but encouragements. One cited the benefits of moving to GitHub: attracting new users who would report bugs, public forks that tested outside contributions, and the ability to accept or deny pull requests from outsiders.

Those benefits evidently sounded good to the team, because it did indeed migrate from .ZIP file releases on SourceForge.net to a fully hosted repository on GitHub. It took a bit of time to get used to Git's syntax and its differences from Perforce, but the team soon found Git more useful. In addition, because FontLab Studio's native source file format is closed and binary, the team also switched over to using the XML-based Unified Font Object (UFO) format. The text-based nature of UFO worked better with Git, but it also made the sources editable by a wide variety of applications and third-party scripts.

In short order, the team discovered several side benefits to the new workflow. Not only was UFO's format better suited to GitHub's revision control, but its human-readability made visual inspection of changes simple. Likewise, although the team was already well versed in version control, Hunt said that the distributed nature of Git made collaboration much easier—between internal team members as well as with the community. Reports also came in that seeing the document hierarchy in public proved educational for users and contributors, both for how the fonts were designed and for how AFDKO is used.

Contributions

Hunt and Sousa then discussed several types of feedback they had seen from the public project. There were bug reports—including requests for additional character sets—changes to glyphs, comments, and everything down to simple "+1"s, which Hunt told the audience were the least helpful type of feedback. Hunt said that the team had anticipated a significant number of design contributions, based on what the community had said about moving to GitHub, but in reality there have been very few. He chalked this up to a number of factors, starting with the small number of people working in type design, but also including the difficulty of rebuilding and testing fonts. He also speculated that there might be a lot of people who would be interested in contributing but do not know where to begin, and encouraged the open source font community to help educate people.

But there have been other forms of contribution, he said. Logos Bible Software commissioned the addition of small capitals, subscripts, and superscripts to Source Sans Pro. There have also been significantly more bug reports and feedback comments on the open source typefaces than on Adobe's commercial offerings, which Hunt said has led to improvements in the commercial products as well. But perhaps the most fundamental change to come from the project was the rethinking of the font development workflow and toolchain, he said. The team has become more transparent and collaborative in all of its projects, both internal and external. And it has continued to use Git for version control, both on its open source and its proprietary projects.

Sousa and Hunt ended the talk by encouraging everyone to contribute to the fonts. "Although these were started by Adobe, they belong to the community." In the Q&A period, an audience member asked what contributions would be the best to work on; Hunt replied that anything was welcome—there is a roadmap, including several new character sets, but all improvements benefit everyone. Indeed, in one particularly amusing example (perhaps because it incorporated a considerable amount of ASCII art), fellow Adobe employee Frank Grießhammer set out to add the Unicode box-drawing characters to the fonts, an effort about which he delivered his own rather detailed talk later in the conference.

Another audience member asked what the Adobe open font project had learned that had not already been proven by Google Fonts. Hunt responded that the primary drawback of Google Fonts is that it only provides downloadable products. One can get the source files, but the service is not set up to contribute back or collaborate. The same audience member also asked if the team's experience with open source development meant that AFDKO would be released as open source. Sousa replied that they were working toward that goal, but that they still had more internal work to do before an open source release could be made. However, he also commented that AFDKO is free to download and use, and hoped that people would not let the EULA be an excuse to not get involved.

To longtime free software supporters, some of the Adobe Type Team's observations about open source development will come as no surprise. Nevertheless, it was still refreshing to see that the company was willing to listen to the feedback of community members, even in the project's earliest days and even when that feedback recommended a nearly complete overhaul of the project's tool set and workflow. After all, one does not need to look very hard to find examples of proprietary software vendors announcing that they will do open source their way whether the community likes it or not. Those companies usually reap frustration and disappointment, whereas the Adobe Type Team not only found a good user community, but several beneficial changes it can incorporate into its other projects as well.

Comments (21 posted)

Chromatic fonts are coming

Outline fonts historically define their constituent glyphs with just two "colors": the foreground color inside the character's contours, and the background color that lies everywhere else. On a generic document, those colors usually equate to black and white, although both can be assigned to other colors in virtually every application. But this situation is misleading; when applied to digital fonts, the term historically only takes one back a few decades. Typefaces in other media have existed for centuries, and were never restricted to black and white. So it should come as no surprise that there is a push to adapt digital font formats to natively support multiple colors. At TypeCon 2013 in Portland, there were in fact two competing proposals under discussion: one advocated by Microsoft, and one backed by Adobe and Mozilla.

In graphic design circles, the term "chromatic fonts" is used to describe fonts built to support multiple colors. Today, chromatic fonts are generally incorporated into designs by taking a set of fonts with complementary shapes and layering them in software, one color per layer. A good example would be the HWT American Chromatic system from Hamilton Wood Type, a digitized font family based on 19th Century woodblocks. The different layers would be combined in layout software, such as Adobe InDesign, Photoshop, or Inkscape. For the web, there are other approaches, such as Colorfont.js, which we looked at in 2012. But even with software support, layering and aligning blocks of text in multiple fonts is tedious and error-prone. It would be simpler if the font could support regions of different colors out of the box.

There is, however, a use case for multi-color glyph support in fonts that is completely unrelated to multi-color type as graphic designers use, which is what leads to the two competing proposals. The second use case is emoji, the character-sized pictographs popular in text messages and online chat applications. While emoji are relatively popular everywhere, as Microsoft's Michelle Perham explained in her TypeCon talk, they are a far bigger deal in Japan, where they originated with mobile phone operator NTT DoCoMo. DoCoMo and its two largest competitors in Japan, KDDI and SoftBank, standardized on a common set of emoji in the mid-2000s.

While the standard was originally only used in Japan, it was accepted into the Unicode 6.0 specification in 2010, incorporating 722 symbols. The standard emoji set has since spread to many other mobile and Internet platforms, with one of the most notable being Apple's iOS. iOS 5 was the first Apple release to support emoji in every region of the globe, but iOS 6 took things a step further in late 2012 by rendering the emoji glyphs in multiple colors. As would be expected, this proved to be a big hit among the demographics that care about emoji, and, in June, Microsoft announced at one of its developer events that Windows 8.1 would feature color emoji, too.

My emoji

However, while Apple's color emoji implementation was a custom iOS-only solution, Perham said, Microsoft decided instead to implement its color support by adding a set of extensions to the OpenType font format. OpenType already supports a large set of extensions through the use of "feature" tags, which are text tables that usually describe character-matching expressions, substitution rules, spacing adjustments, and so on.

Microsoft's new tags operate a bit differently from the most common OpenType tags. The first is called colr (following the convention of four-letter-or-less tag names). The base emoji glyph is a black-and-white outline character as it is in existing fonts; a colr table is attached to it, which contains a list of additional glyphs, one for each color used, along with an indexed color value for each glyph (these indexes are mapped to colors in another structure, discussed below). The list also indicates the z-order in which the color glyphs are meant to be stacked. If the device or application wants to display the color version of the character, it just looks up the components in the colr table and composites them together. If it does not care about color, it can use the black-and-white base glyph like normal.

The actual color that each of the components should be rendered in is listed in the second addition to OpenType, the cpal (for "color palette") table. The cpal table encodes a color palette as a list of RGBA values. The table must include one such palette (palette number 0), but it can offer alternate palettes as well, leaving it up to the software how best to expose the palette options to the user. Perham pointed out that Apple's color emoji font took a great deal of criticism by rendering all of the human face emoji in decidedly Caucasian-looking colors; supporting multiple palettes offers a relatively simple and space-efficient fix.

At the moment, Perham said, creating these color fonts is a manual process, since no font editing software supports the new tables or rendering in color. But Microsoft is moving forward with the fonts for Windows 8.1, and believes the system is flexible enough to be used for other multi-color symbols or for general typographic usage. Currently, Microsoft's proposal is under discussion on the mpeg-OTspec mailing list; one must join the list to read the messages or the posted files (including the specification), but joining is not restricted or moderated. Microsoft has also announced its intention to formally submit the proposal for standardization by the ISO, which maintains OpenType as ISO/IEC 14496-22.

SVG

While Microsoft's solution is sure to reach users in short order, it is not the only proposed system for adding color glyph support to OpenType. There is one competing extension that can also be tested today: the SVG glyphs for OpenType proposal, which is being developed by a World Wide Web Consortium (W3C) community group with most of the input coming from Adobe and Mozilla. Mozilla and Adobe had each drafted its own proposal, but the two were merged into a single draft specification in July 2013.

The draft adds a single table to OpenType named svg. This table can contain any number of SVG objects, and an index that associates each object with a glyph encoding slot in the font. In that sense, it differs considerably from Microsoft's specification, because it references a completely different data type for the "color version" of each supported glyph. There is also a color-palettes subtable, which lists one or more sets of RGBA colors that can be used to draw the SVG elements.

Just before TypeCon, Mozilla's Robert O'Callahan posted an update on the draft proposal. O'Callahan has added support for SVG-in-OpenType fonts to his personal builds of Firefox, and has also made a web application to add SVG support to existing OpenType font files.

The most frequently-cited advantage of the SVG-based approach is that, by adopting an existing and well-known standard for the color elements, OpenType avoids defining yet another graphics format. The software stack would have to be modified in order to render the SVG glyphs, but it could simply hand that duty off to the system's existing SVG renderer. In comparison, the Microsoft colr approach would require modification to the text rendering process as well, but with entirely new code.

In addition, although at the moment the emphasis of both proposals is on relatively simple, flat-color rendering, the SVG proposal can also include other design features that would be of appeal to graphic designers, such as color gradients, without further alteration. It can also incorporate animated elements, which seems to be regarded by many as the next step in the "evolution" of emoji. The principal drawback, of course, is that SVG itself is a large and complex standard. Incorporating SVG in OpenType fonts would by necessity require defining a "safe" subset of SVG to support; no one is keen on the notion of embedding streaming video elements into font glyphs.

Or at least no one seems interested in such a feature today.

The future

The reality on the ground, though, is that no one is quite sure what type designers will come up with when creating chromatic fonts with either proposal. Gradients, of course, seem like a natural fit; one can already see graphic designers applying gradient effects to plenty of typefaces on the web and in print, and it is a clear analog to the effects that can be produced with ink and a letterpress. Animation is not quite as easy to predict; it is simple to imagine terrible eyesores created with animated fonts, but then again one can already design terrible eyesores with any number of existing web standards today.

Both proposals have a long road ahead before making it into a standard. Microsoft, of course, is marching ahead with its own implementation in the meantime. The SVG-glyphs-for-OpenType group held a meeting during TypeCon (which I did not attend); there is more discussion scheduled for the coming weeks and months. As a practical matter, what the industry's type designers and graphic designers find more useful is ultimately what matters the most, but it could still be many years before chromatic fonts become widespread. On Sunday, type designer Crystian Cruz gave a presentation showcasing many of the advanced effects that are already possible with OpenType's existing features—including randomization, contextual substitutions, and character forms that adapt to fit the available space. But the majority of these features are rarely, if ever, employed by existing fonts, and they have scant support in software. Whether the popularity of emoji is enough to spark a larger revolution in chromatic fonts remains to be seen.

Comments (22 posted)

Calibrating Calibre 1.0

File management does not seem to be a one-size-fits-all proposition. Thus, while general-purpose file managers exist, it often appears that much of the development effort goes into domain-specific management tools. We have a whole set of photo management applications, for example, and even more music managers. When it comes to electronic books, though, there seems to only be one viable project out there: Calibre. LWN last looked at Calibre almost exactly two years ago. The recent version 1.0 release provides an obvious opportunity to see what has been happening with this fast-moving utility.

While installing the new version, your editor quickly learned that some things have not changed. The download page still recommends against using versions of the program packaged by distributors, "as those are often buggy/outdated." Instead, one is supposed to install the project's binary release, done by instructing a Python interpreter, running as root, to fetch a program from a web site and execute it, sight unseen. After all, what could possibly go wrong? For those who might balk at handing the keys to their systems to a remote site in that way, it is also possible to download a simple tarball and run the program from there. Source is also available, naturally, but the build process has a lot of dependencies and is not for those in a hurry.

One unfortunate antifeature that has not changed is the program's habit of phoning home on startup, providing the user's IP address, system type, Calibre version, and an identifying UUID to a Calibre server. One assumes that most distributors have disabled this reporting, but binaries retrieved directly from the site still do it. On the other hand, it's worth noting that Calibre is good about not breaking things with its (frequent) updates. Plugins installed years ago still work without problems (or even rebuilding), for example, and your editor's book library itself has never had trouble with Calibre updates. Calibre is developing rapidly, but some care has clearly gone into avoiding pain for users when they upgrade.

New in 1.0

In one sense, there is not a whole lot in the 1.0 release itself that calls for the big version number bump. Evidently, it was simply time to declare that Calibre had reached that milestone. That said, there are a couple of new features in this release, the most visible of which is the grid view for books in the library. This view, as one might expect, simply shows book covers in a matrix arrangement; it joins the longstanding list view and the relatively useless animated "cover browser" as the third way of selecting a book from the library. The grid view is nice; it can be thought of as analog to arranging one's books with the cover forward, while the list view is like looking at the spines.

The 1.0 release also advertises a much faster database backend for managing book information. With a library containing several dozen books, it is hard to notice the difference; those with much larger libraries may have a different experience. But large libraries present challenges of their own in Calibre; the interface feels increasingly unwieldy as the number of books grows. A set of bookshelves can be organized as the owner wishes; a Calibre library is somewhat less flexible. There is, for example, no mechanism for imposing any sort of hierarchy on books. Yes, this is the 21st century, and we all use tags for everything now, but, as Herbert Simon pointed out years ago in The Sciences of the Artificial, human brains think in terms of hierarchies.

That said, one helpful feature that has been added (in the 0.9.28 release in April 2013) is "virtual libraries," which are a sort of saved-search mechanism. One can create a virtual library containing only books from a specific author, say, or based on tags. Virtual libraries don't really provide much that Calibre's search mechanism didn't already do, but they make frequent searches easier and faster to get to.

Calibre has long been the definitive tool for the conversion of books between formats. Over time it has been gaining features for the editing of books as well. New in 1.0 is an editor allowing the modification of the table of contents; it can perform simple tasks like adding an entry for an interesting location, or it can be used to create a new table of contents from the beginning. The automatic metadata downloading system can look further afield for book covers, descriptions, and more. The embedding of fonts within books is now fully supported. On the input side, it is now possible to import books in the Microsoft Word (.docx) format; for output, the PDF generation engine has been rewritten with, it is claimed, much better results.

Beyond that, of course, the program continues to gain support for more devices, more ebook formats, and more web sites as sources of content. Calibre's ability to download the contents of an online magazine issue and package them into a book on a reader device remains one of its more useful feature; it makes offline reading easy. In general, the addition of features continues at such a pace that many of them are hidden at the outset and must be manually added to a toolbar (by way of the "preferences" screen) to be accessible. Suffice to say that Calibre developer Kovid Goyal does not seem to agree with the minimalist line of thought — with regard to features or configuration options — that seems to dominate some other projects.

All those features and options are certainly welcome, but there is no doubt that Calibre could benefit from some focused design and usability work. The interface is cluttered and useful functionality can be hard to find. Who can say, off the top of their head, why there are two search bars on the main screen? (One of them is actually a place to enter a name to attach to a "saved search"). Or consider a dialog like the one shown to the right. What line of reasoning would lead to the "cancel" button being placed right in the middle of all the other options? In general, Calibre gives the impression of being a large collection of (mostly useful) tools haphazardly thrown into a big box.

Finally, one significant "new" feature (new since the last review, but it was available in 0.9) is proper support for Android devices. Linux systems still struggle to support the MTP protocol in any sort of general way, but Calibre's internal MTP implementation seems to work just fine. Calibre recognizes Android devices when they are plugged in, and can tell multiple devices apart; it can be configured to manage or ignore each device independently. There is a simple rule system that allows different types of files to be directed to different locations on the device. It all works quite well, though interoperability with the Android Kindle app is minimal. The third-party plugins that deal so nicely with books on an actual Kindle device are unable to read Kindle books on an Android device.

DRM

In general, of course, DRM continues to be one of the biggest disincentives for anybody considering the purchase of an electronic book. It should be possible to buy a book and read it using the device — and the software — of one's choice. One should not have to worry about things like having one's books deleted as a consequence of crossing a national border. Books should not be subject to lock-in to a particular device or the continuing good will of a remote corporation. Someday, hopefully, the book industry will free itself of DRM; until then, people who are unwilling to break the DRM on books they buy will not be able to make full use of a library manager like Calibre.

One should note, of course, that there are quite a few publishers selling DRM-free electronic books now. Patronizing those publishers seems like a good way to ensure that they continue to make DRM-free books available. Calibre, of course, has long had a built-in "get books" feature that can search for electronic books from a large number of sellers.

In summary: Calibre remains the definitive tool for the management of electronic books for Linux — or for any other operating system. The needed functionality is there, and it continues to develop at a fast pace. Indeed, the pace is surprisingly fast given that the lead developer is responsible for the vast majority of the work — nearly 75% of the commits in the Calibre git repository. Perhaps, someday, somebody will set out to create something that is prettier, perhaps based on Calibre's book management and format-conversion engine. But, for now, Calibre seems to be the only show in town; happily, it is a highly functional and useful tool.

Comments (26 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: Binary "diversity"; New vulnerabilities in chromium, glibc, kernel, wireshark, ...
• Kernel: flink() flunks; TSO sizing and the FQ scheduler; Device namespaces; Cramming more into struct page.
• Distributions: Debian LTS?; NetBSD, openSUSE, OSTree, Ubuntu, ...
• Development: An updated K-9 Mail; Upstart 1.10; PHP's evolution; GNOME 3.10 sighting; ...
• Announcements: Biotech Startups Join Linux Foundation, Ubuntu Edge, Ada Initiative, ...

Next page: Security>>