Development

Automating font builds

At ATypI 2016 in Warsaw, Marek Jeziorek and Behdad Esfahbod presented the work that the font-tools team at Google has done to automatically build and test fonts from source. The project is increasingly of importance to Google's work on the Noto font project, which is intended to cover every writing system and language on the planet. As one would expect, building and updating fonts that cover so much ground is far beyond what developers can do manually.

Jeziorek started out by describing the team and explaining its mission. There are currently six engineers involved, plus three people who work in a management capacity. The team is responsible for the Noto font family, which means that it must answer to two product teams: Chrome OS and Android. In addition, the team develops all the tools needed to make Noto usable, and those tools are intended to be useful for other projects both inside (for example, to the Google Fonts project) and outside the company.

Noto is an abbreviation for "No more Tofu," Jeziorek explained, a reference to the empty-square fallback character seen when a font does not have the glyphs needed to display the necessary text. The goal is for Noto to provide a quality font for every writing system and language, and to do so in a style that is consistent and looks harmonious across all of those various languages. The main benefit will be to the billion computer users estimated to get online for the first time in the next few years, who do not speak a language written with the Latin, Greek, or Cyrillic alphabets.

The plans right now include more than 100 writing systems and 800 languages, although that could be adjusted. The Noto font family is designed to follow Unicode; if there are additions in future revisions, Noto will add more support to match whatever changes. For each language, there will be a serif font design, a sans serif, and "sans UI" that targets interface design rather than setting long text runs. All of the fonts are open source and are hosted on GitHub.

The big challenge, he said, is coordinating with all of the one-hundred-plus designers that create fonts for their local language. Google works with partners, namely Adobe for Chinese, Japanese, and Korean (CJK) and Monotype for other languages, but those partners frequently employ outside contractors. The result is that the Noto project pulls in work created by a lot of independent type designers.

Each new font project begins with a design proposal sent in as a PDF, Jeziorek said. Several iterations of design feedback tend to follow (focusing on matching the Noto look and feel as well as on readability), after which the designer begins working on the final product. The files delivered when the font is complete are in source form, which is then built and tested for completeness. At the moment, he said, 29 Noto families have been released (each containing several fonts as mentioned earlier, and each including several weight and width variants) that build into 638 TrueType or 638 OpenType font binaries. That amounts to a lot of testing.

Esfahbod then talked about the build pipeline. Three years ago, he said, many existing font-manipulation scripts and tools targeted the Unified Font Object (UFO) format, and he began working on the FontTools Python library to improve its support for UFO. The first task was to build a tool that would create a subset of an existing font, for example. At ATypI 2014, however, he commented that he intended to keep extending FontTools because the other major production pipeline, Adobe's Font Development Kit for OpenType (AFDKO), was closed-source. "Then David Lemon gave me the good news," he said, that AFDKO was being released as an open-source project (with one or two components as small exceptions), as were several related libraries.

So, for the first time, it looked like it was possible to create a full build pipeline using open-source software. When the team next renegotiated its contracts with the type designers, it asked them to provide UFO source files rather than binaries. Surprisingly, though, the replies were "we just switched to using Glyphs." Esfahbod talked to Glyphs developer Georg Seifert about the possibility of providing a scriptable Glyphs-to-UFO converter, and Seifert was receptive to the idea. But, in the end, Esfahbod just wrote the converter himself, since the Glyphs file format is well-documented on GitHub.

That new component made it possible to extract data straight from a Glyphs source file and pipe it into any of several available UFO-related tools, including the ADFKO. New engineers started working on additional pieces of the pipeline. James Godfrey-Kittle developed a library to use UFOs within FontTools; Sascha Brawer wrote a tool to bypass one of the few remaining binary components of AFDKO, which fixed overlapping contours in glyph outlines. Other pieces in the ever-growing Python toolkit include compreffor, which optimizes Compact Font Format (CFF) font tables for size, cu2qu, which converts cubic Béziers from a CFF font into the quadratic Béziers required for TrueType, a tool to do visual diffing between fonts, and much more.

The tool set eventually encompassed everything required to automatically build high-quality binary fonts directly from Glyphs source files. Nevertheless, there was no top-level process to coordinate all of the pieces, until Godfrey-Kittle wrote fontmake. As the name implies, fontmake is a full build system for font sources. The supported inputs include UFOs, Glyphs files, and several auxiliary formats like the designspace files used to define interpolation masters in MutatorMath. That library has been the industry standard to generate large font families from a small set of masters, although Esfahbod noted that the newly released OpenType 1.8 variations font feature may change a lot of workflows.

Up until recently, though, fontmake was useful for the Noto team but was not really friendly to outsiders. "We developers just want to write new code, and maybe fix a bug here and there," he said, not maintain the code. But that has begun to change, he reported. Brawer has developed testing and quality-assurance tools; Godfrey-Kittle and Cosimo Lupo have integrated fontmake with continuous integration, and Godfrey-Kittle has gotten the various dependencies under control, ensuring that fontmake works with a set of "known good" versions of the other libraries.

The result is a fully capable build tool, though Esfahbod noted that it is currently designed only to support Noto's precise build process. Other users will certainly want different or additional options. It is also embarrassingly slow in some places, he added, at least compared to C. "But that's not the point," he said. "The point is that it's faster for development. Font tooling is such a niche that there are not many people interested in developing for it, but Python is what that community wants to use."

More flexibility and stability are still to come, he said, but there are still features to be added as well. Work will need to be done to support the new OpenType 1.8 variations fonts, there are improvements to be made in how CFF operators are used, and there is still one last piece of the AFDKO that has no open-source equivalent: the PostScript auto-hinter used on CFF fonts.

At the pace that the tools team has been working, however, the wait might not be too long at all. For everyone outside of the Noto project, a set of open-source build tools can reap plenty of benefits, from enabling distributions and other font distributors to automate their build scripts to enabling font designers to make use of concepts like continuous integration that other corners of the development community have already found useful.

Comments (1 posted)

Quotes of the week

— John Gilmore

— Justin Kramer investigates npm install hairball. (Thanks to Paul Wise.)

Comments (12 posted)

Bash 4.4 and Readline 7.0 released

The GNU Bourne Again SHell (Bash) project has released version 4.4 of the tool. It comes with a large number of bug fixes as well as new features:"The most notable new features are mapfile's ability to use an arbitrary record delimiter; a --help option available for nearly all builtins; a new family of ${parameter@spec} expansions that transform the value of `parameter'; the `local' builtin's ability to save and restore the state of the single-letter shell option flags around function calls; a new EXECIGNORE variable, which adds the ability to specify names that should be ignored when searching for commands; and the beginning of an SDK for loadable builtins, which consists of a set of headers and a Makefile fragment that can be included in projects wishing to build their own loadable builtins, augmented by support for a BASH_LOADABLES_PATH variable that defines a search path for builtins loaded with `enable -f'. The existing loadable builtin examples are now installed by default with `make install'." In addition, the related Readline command-line editing library project has released Readline 7.0.

Full Story (comments: 30)

Emacs 25.1 released

Version 25.1 of the Emacs editor is available. New features include a dynamic module loader, experimental Cairo drawing, better TLS certificat validation, better Unicode input, a mechanism for embedding widgets within buffers, and more.

Full Story (comments: 9)

LLVM contemplates relicensing

The LLVM project is currently distributed under the BSD-like NCSA license, but the project is considering a change in the interest of better patent protection. "After extensive discussion involving many lawyers with different affiliations, we recommend taking the approach of using the Apache 2.0 license, with the binary attribution exception (discussed before), and add an additional exception to handle the situation of GPL2 compatibility if it ever arises."

Full Story (comments: 32)

CouchDB 2.0 released

The Apache CouchDB database project has announced its 2.0 release. New features include clustering support, a new query language, a new administrative interface, and more. "CouchDB 2.0 is 99% API compatible with the 1.x series and most applications should continue to just work."

Comments (none posted)

GNOME 3.22 released

The GNOME Project has announced the release of GNOME 3.22, "Karlsruhe". "This release brings comprehensive Flatpak support. GNOME Software can install and update Flatpaks, GNOME Builder can create them, and the desktop provides portal implementations to enable sandboxed applications. Improvements to core GNOME applications include support for batch renaming in Files, sharing support in GNOME Photos, an updated look for GNOME Software, a redesigned keyboard settings panel, and many more."

Full Story (comments: 42)

Development newsletters from the past week

• What's cooking in git.git (September 14)
• What's cooking in git.git (September 19)
• This Week in GTK+ (September 20)
• OCaml Weekly News (September 20)
• Perl Weekly (September 19)
• PostgreSQL Weekly News (September 18)
• Python Weekly (September 15)
• Ruby Weekly (September 15)
• This Week in Rust (September 20)
• Wikimedia Tech News (September 19)

Comments (none posted)

Hutterer: Synaptics pointer acceleration

For this week's development horror story, it would be hard to do better than Peter Hutterer's quest to figure out how pointer acceleration works in the Synaptics driver. "Also a disclaimer: the last time some serious work was done on acceleration was in 2008/2009. A lot of things have changed since and since the server is effectively un-testable, we ended up with the mess below that seems to make little sense. It probably made sense 8 years ago and given that most or all of the patches have my signed-off-by it must've made sense to me back then. But now we live in the glorious future and holy cow it's awful and confusing."

Comments (30 posted)

Coghlan: The Python packaging ecosystem

Here's a lengthy piece from Nick Coghlan on how Python software gets to users. "There have been a few recent articles reflecting on the current status of the Python packaging ecosystem from an end user perspective, so it seems worthwhile for me to write-up my perspective as one of the lead architects for that ecosystem on how I characterise the overall problem space of software publication and distribution, where I think we are at the moment, and where I'd like to see us go in the future."

Comments (28 posted)

Garcia: WebKitGTK+ 2.14

Carlos Garcia Campos takes a look at the latest stable release of WebKitGTK+. "[The threaded compositor] is the most important change introduced in WebKitGTK+ 2.14 and what kept us busy for most of this release cycle. The idea is simple, we still render everything in the web process, but the accelerated compositing (all the OpenGL calls) has been moved to a secondary thread, leaving the main thread free to run all other heavy tasks like layout, JavaScript, etc. The result is a smoother experience in general, since the main thread is no longer busy rendering frames, it can process the JavaScript faster improving the responsiveness significantly." This release is also considered feature complete in Wayland.

Comments (8 posted)

The curious case of the switch statement (fuzzy notepad)

The fuzzy notepad blog is carrying a post about the switch statement with just about everything one might want to know about its past, present, and possible future. "As we’ve seen, the switch statement has had basically the same form for 49 years. The special case labels are based on syntax derived directly from fixed-layout FORTRAN on punchcards in 1957, several months before my father was born. I hate it."

Comments (24 posted)