LWN.net Weekly Edition for September 29, 2016

GTK+, version numbering, and long-term support

GTK+ version 3.22 was released on September 21, bringing with it a range of improvements to Wayland support, gesture support for pressure-sensitive tablets, several new widgets, and more. The release also marks a turning point for how stable and development branches of the code will be maintained. Moving forward, the project is adopting a new scheme that allows it to designate certain stable releases for long-term support. The plan also breaks with past releases where version numbering is concerned, though the project is keen to downplay that change in favor of focusing on the support that stable releases will offer to downstream projects.

The new release scheme was announced on September 1 in a Google+ post and an accompanying blog post written by Allan Day. The blog post explains more of the background issues that led up to the decision to adopt a new scheme.

GTK+ has long used the traditional "major.minor.micro" numbering scheme (sometimes called semantic versioning) that was once the approach favored by free-software projects. Bumping the major number indicated a significant API break, breaking backward compatibility. Bumping the minor number to the next even value indicated a stable update, while the odd values designated development branches. Micro (or patch) releases were reserved for bug-fix updates.

But, in the GTK+ 3.x era, Day notes, the project picked up significant development speed and also adopted a strict six-month release cycle. That pace has led to concerns over GTK+'s stability, particularly for projects other than GNOME, which shares many developers and other contributors with the GTK+ project. The GTK+ developers, however, want the project to be useful for a wide range of projects outside of GNOME, which prompted discussions earlier in 2016 about changing the release schedule.

Rethinking

In June, after the annual GTK+ hackfest, Allison Lortie announced the proposed change in a blog post that sparked a fair share of confusion and concern. Commenters were evidently perplexed by the proposal, which included difficult-to-parse statements like this:

One early comment by "Alex" summed up much of the general reaction:

To be fair, though, releasing x.0 versions that were unstable was certainly not the intent of the scheme announced. Rather, the plan was meant to suggest that GTK+ version 4 would continue to evolve over the course of the subsequent 4.y releases. Nevertheless, the confusion was demonstrably a problem. At GUADEC in August, the GTK+ team reexamined the topic with a promise to present an updated plan as soon as possible.

Rethinking again

The September 1 announcements, then, constitute that update, which will hopefully prove clearer to outsiders. In essence, the GTK+ x.0.0 releases moving forward will be designated stable, long-term support versions, with the project planning to release an x.0.0 release about once every two to three years. In between these releases, minor updates will also appear that may introduce new functionality. The minor releases will not be bound to a fixed six-month release cycle, however.

Next, the GTK+ development branches will be numbered x.9, to indicate that they are unstable releases being built in preparation for release x+1. This means that, in the future, there may be (for example) a stable, long-term support GTK+ 5.0 available, a series of updated releases (GTK+ 5.2, 5.4, and so on), and a development branch numbered 5.9.

Furthermore, any features deprecated in one x.0 release will be removed in the following x+1.0 release. This is another area where GTK+ has not historically had a strict policy, so stating and adhering to a regular deprecation formula will no doubt please many outside developers. The new plan also states that minor releases may add new widgets and update the GDK drawing backends used by the various window systems supported, but that no other changes will be made. Finally, micro releases for bug fixes and security updates will be made for three years.

Thus, the total lifespan of the x.0 long-term support releases will be three years. The wording is a bit ambiguous as to whether x.2 and other minor updates will also be supported for three years (potentially several months after the x.0 release), but that does not sound like the intent of the plan.

On a technical note, the blog post notes that future development releases of GTK+ will be labeled with the future stable release's version number in the pkg-config file, in order to make them parallel-installable with the current release. So, for example, the pkg-config file in GTK+ 3.90 will be gtk+-4.0, so it will not conflict with the current stable release, GTK+ 3.22.

Development releases are expected to appear about once every six months, all bearing version numbers in the x.9 range (e.g., x.90, x.92, x.94, etc.). That puts some indirect pressure on the project to release a stable y.0 release once the development version's minor number reaches .98, as Sébastien Wilmet noted on the GTK+ development list.

The new plan sets out a fairly regular numbering and release scheme but, of course, transitioning between the old and new schemes will be a tad awkward. This awkwardness takes the form of the new stable release, GTK+ 3.22, being declared the first long-term support version, even though it is not branded with an x.0 version number. Hopefully, that will be seen as a small price to pay for more predictable releases.

Downstreaming

The hope is that the plan for major and minor releases will better serve downstream project developers and Linux distributions. A guarantee of three years of security fixes should be enough for most Linux distributions, while the promise to make no significant changes to GTK+ internals in minor releases ought to be welcome news for downstream application developers. For distributions that offer their own long-term support releases with a lifespan longer than three years, Day asks that distribution representatives get in touch with the GTK+ project to develop a support plan.

Day's blog post also assures downstream developers that the project is committed to doing a better job of communicating changes—and of doing so in advance:

One of the other criticisms the project has faced in the past was that too many decisions were made within the relatively small set of core GTK+ developers, with that information not always making its way out into the wider GTK+ community in a timely fashion. The project must still deliver on this promise to ensure that changes are well-communicated to the outside world, but acknowledging the concern and making a public commitment to doing better are important steps.

Despite the increased emphasis on meeting the needs of downstream developers, there has not yet been a public statement from GTK+'s largest downstream project, GNOME, on whether (or how) it will adopt the same updated version-numbering and stability plan. In the past, GNOME and GTK+ version numbers have stayed in sync; with the newly announced plan, GNOME would have to adjust its numbering and release schedule as well in order to maintain that relationship.

Then again, perhaps no such change is warranted. A big part of the rationale for GTK+'s change was to better serve non-GNOME projects; enabling those two projects to move at different paces could be just what the developers want.

Comments (29 posted)

The anatomy of a Vulkan driver

Jason Ekstrand gave a presentation at the 2016 X.Org Developers Conference (XDC) on a driver that he and others wrote for the new Vulkan 3D graphics API on Intel graphics hardware. Vulkan is significantly different from OpenGL, which led the developers to making some design decisions that departed from those made for OpenGL drivers.

He started with an "obligatory brag slide" (slides [PDF]) that outlined the progress that had been made on the driver in only eight months, with roughly three and a half people. Ekstrand, Kristian Høgsberg, and Chad Versace, with help from a dozen others, got a Vulkan driver working that was released (as open source) on the same day that the Vulkan specification was released in February. Not everything was written from scratch; the driver uses the same internal representation and back-end compiler that Mesa uses. The driver passed the conformance tests on day one as well, which is not something that everyone in the industry can say, Ekstrand said.

Vulkan is a new industry-standard 3D rendering and compute API from Khronos, which is the same group that maintains OpenGL. It is not simply OpenGL++, he said, as it has been redesigned from the ground up. Vulkan is designed for modern GPUs and software. It will run on currently shipping (OpenGL ES 3.1 class) hardware.

A lot has happened since SGI released OpenGL 1.0 in 1992, which is why a new 3D API is needed. In the 24 years since that first release: GPUs have become more powerful and flexible, memory has become much cheaper, and multi-core CPUs are common. OpenGL has done "amazingly well" over that time, but it is showing its age at this point.

Multi-threaded programs are now commonplace, which makes OpenGL's state machine based on a singleton context kind of obsolete. Off-screen rendering is common as well. Beyond that, GPU hardware has become more standardized, so application developers don't want the API to hide the details of what the GPU is doing as OpenGL does.

Vulkan takes a different approach. It has an object-based API where there is no global state. All state is stored in the command buffer and there can be multiple command buffers. It is more explicit about what the GPU is doing: texture formats, memory management, and synchronization are all client-controlled. Those things are needed to support multi-threading, but also make drivers simpler.

Vulkan drivers do no error checking. There is a set of open-source, vendor-neutral validation layers that do much the same checking as is done in Mesa but they are meant to be disabled at runtime. The idea is for application developers to check their Vulkan code during development, so "why burn 10% of my CPU doing validation" when there are no errors in the Vulkan code?

There is a short distance between the API call and the driver in Vulkan, rather than traversing multiple layers as in Mesa. There is also a short distance between the driver function and actually putting data into the command buffer for the GPU. There are "no extra layers", Ekstrand said.

To handle multiple generations of hardware, each with its own packet format and packing scheme, the Vulkan driver has header files that are generated using Python scripts to process an XML representation of the formats. There is a function that uses that header file information to pack the command data into the buffer in the right way. It has debugging support that can assert() for various problems and the code can be run under Valgrind to find other kinds of problems.

To handle four separate Intel GPU generations, the code is compiled four times to create one version per generation. That allows the driver to keep up with new hardware more easily. The hardware-generation checks for each command function (as in the Mesa driver) are compiled away and the right thing is done for the generation in use. This is one example of where the team got to rethink things because it is a new, from-scratch driver.

One of the challenges faced by the team was in memory allocation. Vulkan provides a collection of heaps where clients can allocate VkDeviceMemory objects. The client can place VkImage or VkBuffer objects at explicit offsets within the VkDeviceMemory object. This doesn't map well to allocation from LibDRM, he said, but it does map well to Graphics Execution Manager (GEM) buffer objects. Other objects have small amounts of driver-allocated memory for state that the driver needs to track. The team had to figure out how to manage all those pieces of memory. Complicating matters was that the Intel hardware has different base addresses for different types of allocations (e.g. shaders, surface states), so the state information needs to be stored with others of the same type.

He and Høgsberg came up with a "crazy" memory allocation structure that they are pretty proud of, Ekstrand said. For device memory objects, GEM buffers are used; there is also a pool of GEM buffers that are used for back buffers. For the state objects, there are block pools that are allocated as a buffer object that grows in both directions as needed. The pools are initialized to provide objects of a specific size. Allocating from either end of the pool is required because of some hardware-specific restrictions.

The block pools are implemented as a 2GB memfd that gets mmap()-ed into the driver. An address in the middle is then turned into a GEM buffer object. The block pool is used to implement both a traditional "allocate and free" style state pool as well as a pool that is used for state that is associated with a command buffer. The latter pool has no free function, it simply gets reset when the command buffer is thrown away. It is a complicated infrastructure, but has worked well, he said.

Most hardware has support for compressed surfaces, but not all parts of the GPU understand all of the different formats. So a "resolve" operation is needed to decompress or recompress the surface at different points in the pipeline. Due to the multi-threaded nature of Vulkan, though, there is no real way to track when the resolves are needed on the CPU side. The Vulkan API provides two features ("render passes" and "layout transitions") that can help. Layout transitions are not currently used in the driver, but render passes delineate where resolves may be needed.

It is easier to write a Vulkan driver than one for OpenGL, Ekstrand said. The lack of error checking simplifies things to start with. The SPIR-V shader language is a bit easier to deal with than OpenGL's GLSL. Also, the Vulkan conformance tests consist of 115,000 tests that the driver developer doesn't have to write. It is a good set of tests, but there are still some holes, he said.

Some things are harder to do for Vulkan than for OpenGL. There is no CPU-side object state-tracking, for one thing. In addition, "applications have a lot more power for stupid". If the application is doing something wrong, which results in a bug filed against the driver, there is a good bit of work—without good tools—needed to track down the problem.

As far as sharing code between Vulkan and OpenGL drivers goes, there are a couple of different approaches. The approach taken was a "toolbox" that provides a number of different parts, from which a driver can be created. That approach has also provided better infrastructure for building other drivers in the future. Those looking for more details may want to view the YouTube video of the talk.

[I would like to thank the X.Org Foundation for sponsoring my travel to Helsinki for XDC.]

Comments (17 posted)

OpenType 1.8 and style attributes

In last week's look at the new revision of the OpenType font format, we focused primarily on the new variations font feature, which makes it possible to encode multiple design "masters" into a single font binary. This enables the renderer to generate a new font instance at runtime based on interpolating the masters in a particular permutation of their features (weight, width, slant, etc). Such new functionality will, at least in some cases, mean that application software will have to be reworked in order to present the available font variations to the end user in a meaningful fashion.

But there is another change inherent in the new feature that may not be as obvious at first glance. Variations fonts redefine the relationships between individual font files and font "families." There is a mechanism defined in the new standard to bridge the gap between the old world and the new, called the Style Attributes (STAT) table. For it to work in a meaningful fashion, though, it must be implemented by traditional, non-variations fonts as well—which may not be an easy sell.

There is no formal definition of a font family, but in general usage the term refers to a set of fonts that share core design principles and, in most cases, use a single name and come from the same designer or design team. The Ubuntu Font Family, for example, includes upright and italic fonts in four weights at the standard width, one weight of upright-only condensed width, and two weights (in upright and italic) of a monospaced variant.

The designers clearly present the fonts as a single conceptual unit, even though (for example) the monospaced version has several characters that use considerably different designs than the proportional version. Some people might argue that the monospace fonts are a separate family, and that together with the proportional fonts, they form a "superfamily." Since no one is in charge of the terminology, such disagreements happen. Similar ambiguities could be found in the Source Code Pro, Source Sans Pro, and Source Serif Pro fonts from Adobe, which were developed separately and take their design cues from unrelated historical typefaces.

An indisputable key to a font family, though, is the fact that the fonts belong together when they are presented to the user. In an OpenType variations font, there is a technical challenge at present but, conceptually, the task is easy: each of the various instances of the font comes from the same source and it can be addressed and otherwise treated as a set of coordinates in the overall design space: (weight=bold, width=normal, italic=no), for example, or (weight=750, width=200, italic=0), to be a bit more numerical. But there has never been a consistent way to map those sorts of design-space characteristics onto standard, non-variations font files. Doing so is the purpose of the STAT table.

Family matters

At the top level, the table lists all of the axes of variation used in the font family. Each axis has a string that can be displayed in user interfaces and an optional axisOrdering number. That ordering has a couple of possible interpretations. One is the order in which the axes should be sorted in a font name. For instance, if width sorts before weight, then a list would look like:

    Foo Condensed
    Foo Condensed Bold
    Foo 
    Foo Bold
    Foo Extended
    Foo Extended Bold
    

and so forth. If weight sorts before width, though, then one would see:

    Foo Condensed
    Foo 
    Foo Extended
    Foo Condensed Bold
    Foo Bold
    Foo Extended Bold

A different interpretation of the axisOrdering numbers would be to specify the order in which the various axes are shown in a font name. That is, whether to show "Foo Condensed Bold" or "Foo Bold Condensed" in the font menu.

Complicating this interpretation is the fact that OpenType already supports several other mechanisms with which to specify a font's name including all of those design attributes, via the name table. The three options are Name IDs 1 and 2, which can be used to specify a Font Family Name and Font Subfamily Name (respectively), Name IDs 16 and 17, which can be used for a Typographic Family Name and Typographic Subfamily Name, and Name IDs 21 and 22, which are for a Weight/Width/Slope (WWS) Family Name and WWS Subfamily Name. Each pair of Name ID entries can take any string, which are intended to be concatenated together in Family Subfamily fashion. The redundancy of multiple such similar options has not escaped the community's notice, of course; it remains for historical reasons.

Complicating matters even more is the fact that different software platforms interpret name table data in their own peculiar ways, such as when parsing and tokenizing the strings in Name IDs. In the OpenType session at ATypI 2016, Peter Constable noted that Microsoft's Graphics Device Interface (GDI) and Windows Presentation Foundation (WPF) each has its own approach to assembling the font name from the Name IDs, and CSS uses a different approach altogether. The obvious question, he said, is why add yet another possible naming mechanism to the mix. The answer is that STAT does not impose a hierarchical solution like the Family/Subfamily options in name do; it defines the variation axes and that is all. Whereas name table entries can be arbitrary strings that may or may not make sense, the thinking goes, at least STAT axes are well-defined and can be reasoned about.

The mapping problem

Confusing though the naming issues may be, a more practical feature of the STAT table is that fonts can provide a mapping between the numeric values defined for OpenType 1.8 axes and the names commonly used by the various font classification systems and shown in user interfaces. The predefined axes each have an expected range. Italic (ital) must be between zero and one; slant (slnt) must be between -90 and 90 degrees; optical size (opsz) and width (wdth) must be greater than zero; weight (wght) must be between one and 1000. During the interpolation process for variable fonts, all of these values get normalized to [-1,1], but these human-readable ranges were selected to better map to how existing font families are described, including the conventions of CSS, GDI, and WPF.

So it's quite simple to specify in the STAT table that the "Regular" weight of a font is 200, the "Semibold" is 450, and the "Bold" is 625, for example. The table even offers an ElidableAxisValueName flag to indicate that "Regular" (for example) can be dropped from the name shown in UIs, which is a nice convenience for end users.

Where things become trickier, however, is when a font family starts out with a few fonts files at first (possibly even just one) and adds more later. For example, consider a variations font that supports the weight and width axes, all in roman (non-italic) style. In that case, there would be no ital axis defined in the font file. But if a matching italic variations font is released later, then the original font's STAT table is suddenly incomplete, because it does not indicate where that first font lies on the font family's new roman-to-italic axis.

The OpenType 1.8 specification offers a fix for this through the STAT table. The newly released italic font should include (naturally) an ital variation axis, and the axis's record in STAT table would include the relevant entries for the new font, plus one entry for the old font as well. The old font's record gets marked with the OlderSiblingFontAttribute flag, which is meant to indicate to the application or operating system where the old font gets mapped into the new, expanded font family on the ital axis. In our simple example, the entry for the old font would be a zero on the ital axis, but lots of other permutations are certainly possible.

So this feature lets one font file supply data about a separate, older font file that software implementations are expected to read and adhere to. The specification does not dictate how a program should go about determining which older font file is the one referenced by entries flagged with OlderSiblingFontAttribute. Presumably, some Name ID(s) from the name table are involved but, as we have seen, there are several of those to choose from. And it is possible that more than one older font might need to be retroactively referenced in such a fashion—consider yet another new font added to our example above that adds an optical-size variant. That font would have to include OlderSiblingFontAttribute information for the older fonts as well.

Assuming that the old and new font files are released by the same (non-pathological) person and there are no naming conflicts with other fonts on the system, there should be no misunderstandings. But it is not quite clear how software should interpret matters when the font name in the new font file seems to match more than one old font file. And the specification recommends that all new non-variations font families supply the STAT table with OlderSiblingFontAttribute flags, too. For a traditional font family like the Ubuntu Font Family, there are lots of individual files to be built and distributed (13 in the Ubuntu Font's case), with a lot of tables that could get confused or out-of-sync as updates are installed.

Practically speaking, it will be quite some time before OpenType variations fonts become the norm on most users' systems. So type foundries can be expected to release variations-font versions of their binaries as well as sets of individual, non-variations font files. Getting the STAT tables right may take some time; deciphering the font-family information that the tables encode, on the software side, may take some time as well.

Comments (4 posted)