Leading items

Reviving TTX and FontTools

At ATypI 2014 in Barcelona, Google's Behdad Esfahbod gave a report on the work he has been doing to resurrect and reinvigorate the development of a long-dormant free-software project used for font manipulation. In addition to bug fixes and optimizations, he has added several new tools to the original set, and the code is now in use not only at Google, but by several third-party applications as well.

The project in question is known to most users as TTX/FontTools; FontTools is the name of the underlying Python library, while TTX is the most commonly used tool from the included suite of utilities. Esfahbod said he had been "working on a secret project" to revive FontTools for the past few years; he has now made the project public and is inviting users and contributors to get involved. Google started using it internally for font-production work in 2013, at which point he began adapting it and adding new features. His version is available on GitHub, and there is a discussion forum on Google Groups.

By way of background, he outlined the history of FontTools. The project was started in 1999 by Just van Rossum, but in a few years work on it had slowed down to "maintenance mode." Development virtually ceased in 2004, although the code lived on at its SourceForge project page and has continued to be packaged by multiple Linux distributions. The better-known TTX component is a command-line utility that converts font files to XML and vice-versa, since editing and processing XML is considerably easier than performing the same tasks on binary OpenType files. FontTools is the library that powers TTX, he said, but it can do much, much more.

The usefulness of TTX comes from the fact that its XML representation is extremely close to the internal format of OpenType tables. Thus, it is easy to convert a font through TTX and back again, getting a stable, usable output font without losing any information. "I'll make a bet," he said, "no matter what you use to create a font, if you round trip it through TTX, it will be functionally the same." It can look inside of a font file and extract data that other tools cannot. In other words, he said, "it expresses what you want to do with a font much more compactly" than other tools do.

Practical matters

FontTools's XML compiler/decompiler functionality, Esfahbod said, can also be used to optimize fonts—reducing redundancies, for example. The XML it produces is also more compact than that of other conversion tools, so operations are quite fast—fast enough for batch operations on the hundreds of fonts used in the Google Fonts service, for example. In addition, FontTools works with both Python 2 and Python 3, it has no external dependencies, and it introduces minimal abstractions, making it simple to learn.

He then discussed a series of practical usage examples. FontTools can read Type 1, TrueType, and Compact Font Format (CFF) fonts (as well as several less-common formats). It can be used to quickly figure out what a font file includes, such as character coverage, hinting, ligatures, and kerning. That makes it possible to "diff" font versions in a meaningful way, as well as to analyze and generate reports (particularly on large batches of fonts). It can also be used as the final step in producing fonts—for example, adding name-table entries and hints, or even subsetting (such as generating a Japanese-only font file) and merging font files. As a library rather than a graphical application, he said, FontTools is also one of the only options for doing server-side manipulation or reporting tasks, and its XML serialization feature can turn binary font files into a version-control-friendly text representation.

Another major use case for FontTools is "hotfixing" fonts, he said. Converting to XML and back makes it easy to regularize glyph names, fix encoding problems, or add missing characters—for example, he said that Google uses FontTools to make sure that its fonts are not missing the non-breaking space character, which is easy to overlook. As font engineers, he said, there are a number of such things that type designers do not need to worry about, "but are fun for those of us who enjoy writing scripts."

New work

Initially, Esfahbod said, his work on FontTools involved implementing longstanding bug fixes and feature requests that had gone unaddressed in the SourceForge project. He then moved on to ensuring that the XML format was stable, that conversion produced predictable output, and to optimizing the output to be as compact as possible (which he said was vital for Google's web-font service). He then focused on speed improvements, telling the audience that "Python can be slow—if you're not careful about it."

Eventually, though, he began adding new features to the library, and ultimately new utilities as well. Among the highlights are support for Web Open Font Format (WOFF) files and many additional OpenType tables—including bitmap tables and all of the competing chromatic font formats: SVG glyphs and the various formats promoted by Google, Apple, and Microsoft.

There are three new utilities available, he said: pyftsubset, pyftmerge, and pyftinspect. pyftsubset is for generating a subset of an existing font—only certain character blocks, for example, or even a minimal set of glyphs for use in a particular document. It supports both TrueType and CFF glyphs, and it optimizes file output size. pyftmerge is the opposite; it combines multiple fonts into a single file. At the moment it is a "very early prototype," he said, supporting just TrueType. It includes logic to handle conflicting characters and hinting, but in a naive manner at the moment: it preserves the contents of the first input file given. pyftinspect is a previewer, showing the font's table contents in a pop-up window.

In addition to Google Fonts, Esfahbod said that there were already several other users of his relaunched FontTools code. They include several font foundries, Adobe's AFDKO, the free-software font build-service tool Font Bakery, and even Adam Twardoch, one of the main developers at the proprietary font-design-software company FontLab. There are many more features still on the to-do list, Esfahbod said, including support for more input formats, a "lint" tool, better merging options, and better optimization. He would also like to add entirely new functionality, such as the ability to convert between TrueType and CFF, and the ability to optimize TrueType outlines.

There is more work already in development. An intern has been working for the past few months on a CFF subroutiner (which reduces file size by eliminating redundancy) that is expected to be released as free software soon. Esfahbod is also working with a faculty and student team at the University of Waterloo on code to analyze TrueType's bytecode language.

Impact

The font-development community is heavily invested in Python; it is the preferred scripting interface to essentially all font-design applications. But there are relatively few projects that provide a reusable library. In fact, the only other such open-source library that implements a large feature set is Robofab, which is developed primarily to serve as the scripting interface to the commercial font editor Robofont. Nevertheless, there are a lot of small free-software utilities and scripts out there that use Robofab.

But FontTools and Robofab are not direct competitors; they serve two very different purposes. Robofab is a library for working on font design: glyph drawing, spacing, kerning, and so on. The primitives it defines are limited to what is required for these tasks (points, segments, contours, glyphs, etc), and it is designed to work with the Unified Font Object (UFO) drawing format. FontTools, on the other hand, is designed to operate on already built font files: .TTFs, .OTFs, .WOFFs, etc. Its abstractions are not about the drawing of letters and numerals, but about the (arguably) more tedious process of verifying and manipulating all of the tables and embedded features required to make a font work. The fact that it has survived in most major Linux distributions for a decade without real maintainership is a testament to the demand for such a library.

Esfahbod's report on FontTools was also interesting in light of Adobe's subsequent release of AFDKO as an open-source project. Both of these projects attest to the fact that the font-development community is quickly learning the value of sharing tools. Whereas in past years much of the "open source discussion" at ATypI and similar events was focused on the design of open-font licensing, this year open source was more prevalent in conversation—but in a more subtle fashion.

In fact, the first talk of the week was on the subject of what commercial font-makers can learn from open source, and was given by Thomas Phinney, who works at FontLab. He cited advantages that people already enmeshed in FOSS will find familiar: the value of collaboration, the freedom to specialize on certain tasks by not reinventing the wheel for every task, the benefit to everyone of sharing information, and so on. Open source is still only one of multiple development models within the industry, but it has never looked healthier.

Comments (1 posted)

A Dark Room

It's not often that an independent, open-source video game receives widespread mainstream attention and acclaim — let alone a game as quirky as A Dark Room (ADR). The game is a minimalist text adventure that starts in the eponymous room and expands over time into something much greater. In a face-to-face conversation, ADR's developer Michael Townsend detailed the game's origins and development, which may interest those considering common web-development tools (HTML5, JavaScript, and CSS) for game design.

Gameplay begins in a starkly described room. Players will find that soon after they stoke the dark-and-freezing room's dead fire, "a ragged stranger stumbles through the door and collapses in the corner". The stranger's ability to construct buildings and tools provides an early means to grow the game's mechanics, which subtly, gradually, and uniquely paint a chilling portrayal of a post-apocalyptic science fiction world.

The evolution of the gameplay over time, combined with its focus on "ludonarrative cohesion" (the use of gameplay mechanics themselves to progress a story in a video game), would win the game critical praise. A front page post on Hacker News was followed with positive attention from popular sites like The New Yorker and Slate Magazine. ADR would reach the #1 spot on the US and UK iOS App Store download charts for a few months following a proprietary release for Apple's mobile devices.

In 2013, when working full-time at a consulting company, Townsend was inspired by the game Candy Box, which features some similar mechanics: a minimalist text-adventure game with ASCII art that has game mechanics that deepen gradually over time. He felt that this style of gameplay had untapped potential, so he worked on evenings and weekends for several weeks before releasing ADR in June 2013.

Soon after release, the game hit the front page of Hacker News, with several positive comments from software developers who found themselves hooked on the game. A few weeks later, Townsend announced a formal open source release under the Mozilla Public License (MPL) along with a GitHub repository. He had thought that his decision not to obfuscate the game's code would be enough for curious developers, but the clamor for a formal open-source release of the gratis game led him to pick a license.

Townsend released ADR as open source out of a desire to encourage creativity and skill development. The game also gained some improvements following the open-source release:

Written from scratch in JavaScript, HTML5, and CSS, ADR's code was hacked together without Townsend being concerned about optimizing it on a technical level. He was more interested in getting the project done. He had considered using pre-made solutions like Twine, which is an "open-source tool for telling interactive, nonlinear stories" that can be published directly to HTML. But he found that it was too simple for his needs and not as much fun as building the game completely on his own. When ADR got attention on Hacker News, he felt a bit mortified about the code: "Oh no! The nerds are going to tear it apart!"

The game's code is simple. After writing it, though, he discovered the AngularJS web application framework that he now realizes could have provided the technical foundation for the game. ADR connects with Dropbox for handling game save data, and uses Google Analytics to track how players progress through the game to give him feedback about how to improve it. Users can export saved games to continue from a different device; it's a simple matter of generating a unique code that can be entered from a different machine to resume.

"You probably shouldn't look at the code as a good project", Townsend warned, as he was aware that his interview was for a technical news site. There is not much separation between the game's code and its assets (e.g. the strings making up the text of the game). This intermingling of code and assets makes it harder for an interested developer to write a new story using ADR's engine as a base. However, some separation has been made to facilitate translation, which is ongoing; thanks to outside contributions, the game is available in multiple languages. Townsend noted that one could replace certain modules of the game with new ones to create new scenes or game levels.

Although the game did make an impact in the independent video game scene in 2013, it would be a later release on iOS that would garner the game financial success and widespread news attention. A software developer named Amir Rajan, who is a fan of the game, contacted Townsend soon after the Hacker News posting. Rajan wanted to "reboot his life": he had recently quit his job, scaled down his living expenses, and was interested in pursuing a project out of pleasure, using his savings to pay for his day-to-day needs while developing it.

Rajan asked Townsend if he could port ADR to iOS devices (the iOS version remains proprietary; Townsend said it is up to Rajan whether or not that version will be opened up). Townsend agreed, on the condition that he retain a creative veto and receive 50% of any royalties. He did not hear from Rajan for several months, but ended up satisfied with his work. The game was ported to iOS using the proprietary RubyMotion implementation of the Ruby programming language.

Townsend ensured that third-party improvements to ADR's open-source code on GitHub were not incorporated into the iOS app:

Following a release toward the end of 2013, the iOS version would bring in a few hundred dollars a month in royalties, but exploded in popularity in 2014. Both Townsend and Rajan remain mystified as to the causes of the game's financial success; they could not find any correlation between an increase in sales and media coverage.

While the iOS version was its financially successful incarnation, ADR's GitHub repository for the original, open-source browser game remains active. Although Townsend hadn't touched the JavaScript and CSS in a while, he still continues to use the repository to make tweaks and merge pull requests. The bulk of the recent changes are translations, with a recent addition of Polish.

Notwithstanding the success of ADR's proprietary iOS version, Townsend emphasized the importance of open-source game development to him. He opened up the source for ADR's browser version to help other potential game developers learn from his experience. Nonetheless, some third-party use of his code has led to some irritations for Townsend, as quoted by Rajan on his blog:

There are a few versions of the browser-based ADR ported to Android, but they are largely just copies of the browser game's code wrapped in PhoneGap, which is "a free and open source framework that allows you to create mobile apps using standardized web APIs". Without further optimization made for mobile devices, it makes for a sub-par experience; users connecting to the web game through their phones are given a warning page that the game works poorly on mobile devices. No such warning is available on the Android apps.

This has led to some individuals complaining to Townsend about the poor quality of the Android versions, which he had nothing to do with. He contacted some of the developers relaying his concerns; one was apologetic, explaining that they were just trying to learn about game development, while others refused to remove or alter their apps (some of which contain advertisements), citing the game's open-source license as justification.

Townsend has applied for a registered trademark on the name "A Dark Room" in association with video games, and is awaiting approval; if approved, he intends to use the trademark to demand name changes from these developers. He would be content if the game itself remains unchanged, as long as the title is changed to disassociate third-party versions from his authorized ones. Townsend does take comfort in the fact that these clones are nowhere near as popular as the iOS version; that an official Android app is currently in private beta may dull some of the sting as well.

Townsend continues to work a day job (he does full-stack web development for travel website TripAdvisor), while pursuing game development as a side project. His latest game, Gridland, is a fun mash-up of match-three puzzle games like Bejeweled with shelter building and survival mechanics vaguely reminiscent of a simplified Minecraft. It is available gratis online with an open-source release coming soon. Meanwhile, the ADR universe has expanded with Rajan releasing a proprietary prequel to ADR on iOS, The Ensign, and Townsend considering a sequel. Townsend remains committed to open-source game releases under the name Doublespeak Games; it will be interesting following his journey as a game developer in the years to come.

Comments (3 posted)

Adobe releases source code for OpenType font-development tools

Adobe made a surprise announcement at the annual ATypI conference in Barcelona, Spain, releasing one of the company's proprietary font-production tools under an open-source license. In addition, the team convinced another popular font-development project to release its core library as open source, too. Adobe framed the release as a move designed to help improve the quality of fonts produced with any application, but there may be other benefits as well—such as increasing the spread of Adobe's own open fonts. Up until now, those fonts have not been redistributable by many other free-software projects, precisely because the production tools needed to build them remained closed.

David Lemon, manager of Adobe's type team, made the announcement in his talk entitled "A win/win approach to open source." Lemon began the session by recounting Adobe's history with open-source software, as well as his personal involvement. Over the 28 years he has spent at Adobe, he said, he had gone from thinking that open source was "just plain crazy" to becoming one of its biggest advocates within the company.

Standardization and platforms

Lemon noted that free software has its roots in communities like academia that regard sharing information as the default policy and as an ethical position, while "open source" puts the emphasis on the quality of the code produced. Inside Adobe, he explained, both viewpoints have proven important at differing times. The company has long understood the value that comes from publishing a public specification, which it did, for example, with PostScript in 1984. Despite PostScript being given away, he noted, the product still makes money for Adobe today.

Similarly, Adobe published the Type 1 font format in 1990 in the face of competition from TrueType, which was a joint effort of Apple and Microsoft. There are many other reasons why Type 1 has survived, he said, but publishing an open specification was certainly one of them. The PDF specification was another such giveaway, and became a critical platform on which many other products are built. That cemented the idea within Adobe that publishing a standard rather than keeping it secret was a path to success.

At the same time, though, Adobe still struggled to make technologies like Type 1 viable in the marketplace because operating system vendors controlled the platform. The company's initial attempt to circumvent this hurdle was Adobe Type Manager (ATM), a proprietary product that added real-time Type 1 rasterization to Windows and Macintosh systems. Though popular with graphic designers (in part due to its other functions, like managing large font collections), ATM was never a money maker. Eventually, Adobe decided instead to work directly with Apple and Microsoft to get Type 1 rendering incorporated directly into the operating system. The result was that Type 1 gained "equal citizenship on the desktop."

As everyone now recognizes, though, Apple and Microsoft desktop systems are no longer the only platforms that matter—and perhaps not even the most important ones. So, Lemon said, when Adobe realized a few years ago that it was now facing a similar situation with mobile platforms and the web, it decided to take a different approach. The issue at hand was making the Compact Font Format (CFF), the modern successor to Type 1, viable for browsers and mobile devices. On those platforms, TrueType fonts still dominated, but essentially for legacy reasons.

So the company took its CFF rasterizing engine, released it as an open-source product, and donated it to the FreeType project—thus bringing high-quality CFF support to Android and to the most popular web browsers. Adobe still maintains the CFF rendering code to this day, Lemon added. And Adobe has rolled out many other open-source products and projects, he said: the company currently has 99 repositories active on GitHub.

Fonts and tools

One of Adobe's first major open-source products was the online code editor Brackets. Brackets and other such products require fonts with compatible licensing, of course, and Lemon noted that six or seven years ago when the effort began, Adobe did not see any open fonts it was satisfied with. So the company's type team "rolled it own"—first Source Sans Pro, followed by Source Code Pro, and, most recently, Source Serif Pro and the massive pan-CJK (Chinese-Japanese-Korean) font Source Han Sans. The family of fonts has expanded to numerous weights, added scripts like Cyrillic and Greek, and has added features at the request of Adobe product teams—and occasionally outsiders.

That said, Lemon did express some disappointment that there have been few contributions to the fonts from third parties. Adobe had donated fonts to the X Window Consortium years ago, he noted, but the Source Pro project was supposed to be different—a project, not a donation. The magnitude of the response from users was overwhelming, he said, and "although we're quite proud of our QA," the fonts' hundreds of thousands of users "find every little issue" to file a bug report on. To date, though, only one outside designer has contributed a significant number of glyphs.

But one major prerequisite to contributing is access to the tools for building and testing the fonts. This is where Adobe Font Development Kit for OpenType (AFDKO) comes into play. AFDKO is a suite of tools for validating and modifying fonts in the OpenType format (which can encapsulate either TrueType or CFF data). It is also a key part of Adobe's build and release processes. The company regards it as a platform-growing asset intended to spread OpenType adoption; although proprietary applications can work with it and the company provides AFKDO training, it is not a product intended for monetization. Lemon pointed out that it has been available free-of-charge since 2000 (at least for Windows and Mac OS X). The company even released a Linux version in March of 2014.

But free-of-charge simply is not sufficient for all users. Open-source projects could not package AFDKO for license-compatibility reasons, and some projects will not accept binaries (such as fonts) built with any proprietary tools. In addition, Lemon said, there were even some proprietary software companies that would not allow their employees to install and use AFDKO, because the company legal team objected to the end-user license agreement (he named Apple as one such company, but called it a recurring issue).

So, although Adobe wanted AFDKO to improve OpenType support everywhere, it was not having the desired effect. Furthermore, he added, "there is only one open-source font editor, and it sucks"—although he quickly added that was happy to see that there is a lot of recent work going into it, with a number of improvements.

The solution was to make AFDKO open-source software. Lemon said he was excited to announce that as of now, AFDKO is available under the Apache 2.0 license from the Adobe type team's GitHub repository. The existing, pre-packaged version will also remain available under the old terms, "for simplicity's sake," he said.

Around the FDK

Lemon then took a few minutes to detail the contents of the repository. There are around 20 utilities included. Many are Python or shell scripts, but he cautioned that the initial release was "batteries not included." Specifically, installation scripts are missing for the time being, and dependency issues or other hurdles should be expected.

The main use cases of the AFDKO tools are quickly adding the advanced features of the OpenType file format to an existing font, running quality-control and validation checks, and automating build operations. The AFDKO site maintains a description of each tool's capabilities.

Unfortunately, he said, there are two pieces of the original AFDKO that the team was unable to include in the open-source release. Both contain patented code from elsewhere in Adobe, which those other projects were not interested in releasing. One is the checkOutlines tool, which performs a number of validation checks on the Bézier curves in a font—most importantly, removing overlaps. The other is IS—for "intelligent scaling"—which is used to rescale fonts while automatically adjusting the embedded TrueType hints. Lemon noted that the team had identified a possible replacement for checkOutlines already.

Lemon also told the audience that, although he hopes the open-source AFDKO will help improve other font tools, he also hopes that people will contribute their own improvements back to AFDKO. Already the team has begun extending its capabilities, he said. The initial release includes a new set of interpolation functions based on the engine found in Erik van Blokland's highly successful Superpolator application.

Superpolator itself remains closed source, but the AFDKO team convinced Van Blokland to package the core engine into a reusable library, which he also released under an open-source license during the conference. The library is called MutatorMath, and is available from its own GitHub repository under a BSD license. MutatorMath can interpolate entire fonts based on an arbitrary number of interpolation axes (e.g., weight, width, slant, contrast, serif style, etc).

Up next

Lemon concluded his talk by promising that although open source may not take over Adobe, "as long as it continues improving our work, we'll keep doing it." He said more source code was still to come from the type team, starting with some CJK tools built during the development of Source Han Sans.

Outside of the ATypI venue, the AFDKO and MutatorMath releases did not go unnoticed by the free-software community. Shortly after the release announcements (which had been rumored, if not expected, in several circles), a Debian intent to package (ITP) issue was opened up for AFDKO. Bug reports have already started appearing in GitHub, particularly with regard to the dependencies and the build process.

Perhaps most interesting to note is that Debian developers have also announced their desire to package the Source Pro fonts in the wake of the AFDKO release. The software tools may benefit FontForge (the open-source font editor that, presumably, Lemon referred to in his critique) and other free-software applications, but that will take time. The most immediate impact of the release may be that it will, for the first time, be possible to build Adobe's open fonts from source using only free software. Lemon did not delve into the details of that issue during his talk, but being unable to build the Source Pro family with free software has undoubtedly kept at least some potential contributors away.

It is hard to predict whether or not any major contributions might arise as a result, of course, but a fully open-source build chain certainly opens the door to any number of new developments down the line.

Comments (2 posted)