A status update on Debian's reproducible builds

Debian's reproducible builds effort has major implications for the trustability of individual software packages and the system as a whole. Implementing reproducible builds is also a complex undertaking, and DebConf 2015 featured several sessions that dealt with aspects of the work. The most comprehensive talk was the team report, led by Jérémy "Lunar" Bobbio and Holger Levsen and joined by Eduard "Dhole" Sanou and Chris Lamb.

The essence of the reproducible-build problem, as Bobbio explained it, is that free software typically provides source that can be studied (and verified) and binaries that can be used for any purpose, but it does not provide a proof that the binaries were created from the verified source. There are proof-of-concept exploits that highlight the dangers of this situation, such as Mike Perry and Seth Schoen's kernel exploit presented [PDF] at the 2014 Chaos Communication Congress.

The solution is to enable anyone to reproduce a bit-for-bit identical package from a given source tree, and that is what the Debian team has been working toward since 2013. The effort impacts a number of parts of the Debian project, including packaging tools, various compilers and build tools, Debian's infrastructure, and quite a few individual software packages.

Reproducibility work so far

The grunt work of testing a package for reproducibility involves building the package, saving the result, and then repeating the build with slight alterations to the build environment. The reproducibility team has a set of Jenkins jobs running a battery of such tests on its own mirror of the Debian archive. The variations tested include hostname, kernel version, username and UID/GID running the build, time zone, and locale. At the moment, variations for some other factors (including CPU type and the exact timestamp) are not tested, although there is work in progress to support more variants.

Within that framework, Levsen explained, well over 75% of the packages in Debian "unstable" can be built reproducibly on the amd64 architecture—but the necessary changes have not been merged into the packages in the main Debian archive, and quite a bit more remains to be done before such merging will be considered. The team recently added armhf to the test pool and will be adding ppc64el soon; Levsen said it would support hardware from other architectures, too, if anyone has hardware to donate for the effort.

The most recent work includes the dh-strip-nondeterminism add-on for debhelper, which normalizes the contents of various problematic file formats. The set of formats handled includes several archive formats, which may record filesystem timestamps and permissions irrelevant to final archive.

The team also wrote a utility program called diffoscope, which shows the differences between two packages (or directory trees). Diffoscope works "in depth," Bobbio explained: it recursively unpacks archives, uncompresses PDFs, unpacks Gettext files, and disassembles binaries. That allows it to look beyond differences in the bytes between two archives to the "human readable difference" in the original files.

In addition, the team has drafted some proposals that will affect build and packaging tools. The first is .buildinfo, which is a Debian package control file. A .buildinfo file will be used to record the details of the build environment for a package so that the same conditions can be recreated later.

The second is SOURCE_DATE_EPOCH, a timestamp environment variable that build tools can use to export the last-modification date of the source. As Levsen explained, once all binaries are bit-for-bit compatible, the "interesting" factor becomes not the build timestamp, but the last time when the source was altered. The SOURCE_DATE_EPOCH timestamp is also useful for packages like help2man or epydoc that are used to process documentation, for which the team has already caught and fixed many bugs. More challenging is the process of persuading the maintainers of some of these upstream tools that SOURCE_DATE_EPOCH is a useful bit of information to report.

Chris Lamb then discussed some common reproducibility bugs and how to fix them. Last-modification-time timestamps embedded in files are a common problem: they change the file without adding substantive value. Some of the fixes border on being trivial; for example, gzip records a timestamp by default when compressing a file, but that timestamp can be suppressed by adding the -n flag. The internal timestamp field in a PNG file, however, must be stripped out with ImageMagick or a similar tool, which is more work. Various programming languages, such as Erlang and Ruby, record problematic timestamps whenever they process a file, he said, while simple configure scripts often record unnecessary information like the current time and hostname.

There are also several issues related to the ordering of files. For example, in an archive, if the alphanumeric ordering used by the filesystem differs (as can happen if the system locale is changed), two tar archives of identical files can produce differing results. The fix is to pipe the list of files through sort before it goes to tar. Perl exhibits a similar problem; the hash order produced by Data::Dumper is nondeterministic.

For many of these problems, Lamb has found fixes, but there are others that will require developers to do some work. For example, code that uses the current time as a lazy form of unique identifier will need to be rewritten. Most of the reproducibility fixes the team has implemented are not "crazy," he said, but the further upstream the fix is needed, the less likely it is to get accepted. Nevertheless, Lamb reported that the team had created more than 600 reproducibility-support patches, averaging about two new patches per day.

A look ahead

So far, many of the reproducibility bugs caught and fixed have been in the source packages themselves but, moving forward, work will have to be done on Debian's packaging tools and even its infrastructure. Bobbio noted that there are several open bugs against dpkg (including the bug to add support for .buildinfo files). Similarly, debhelper, cdbs, and sbuild all need to be patched.

There is not always agreement, though, about where some of the fixes required for reproducible builds should be made. For example, the bug to make mtime timestamps produced through dpkg deterministic could be solved by patching dpkg, debhelper, or tar. More discussion is needed, Bobbio said, and he invited volunteers to join in.

Other fixes will impact the Debian infrastructure itself. For example, reproducible builds need to be performed using a fixed build path, which will mean implementing changes on the Debian build server. Similarly, .buildinfo files would have to be accessible to users anywhere in the world in order for those users to actually perform their own reproducible builds; that means that roughly 200,000 files (for all of the packages across all of Debian's architectures) will need to be published somewhere, perhaps in the Debian archive itself, and perhaps as a new service.

But the final "patch," Bobbio said, will have to be to Debian policy. The reproducible builds team would like to add "source must build in a reproducible manner" requirement to a new section 4.15—but, naturally enough, that is a change that everyone in the project needs to think about and have an opportunity to weigh in on.

The session ended with the team providing some practical steps that Debian developers and package maintainers can take to fix reproducibility problems in their packages. The status of any package can be checked online by visiting reproducible.debian.net. Users can also test some reproducible builds locally. There is a script available for pbuilder, although it only works on the packages in the patched, reproducible-build mirror.

Reproducible builds have benefits beyond security, of course. The speakers listed several during the talk, such as the ability to create debug packages for a binary at any time (including long after the binary was built), earlier detection of failed builds, and better testing of development tools. Given the response to the talk and the questions asked by audience members, this is clearly a project that many in the Debian community see as an important next step—even if it is one that still has many tasks, bugs, and open questions left to address.

[The author would like to thank the Debian project for travel assistance to attend DebConf 2015.]

Comments (7 posted)

Virtualizing the locomotive

Among the more curious talks at KVM Forum 2015 was Mark Kraeling's "Virtualizing the Locomotive" (YouTube video and slides [PDF]) that was submitted in the "end-user presentation" category. The topic is exactly what the title says: virtualization for train software. The presenter works for GE Transportation.

There's a lot of electronics and software in trains, and virtualization doesn't really apply to all applications on a train; GE is not planning to apply virtualization directly affecting the safety of the train, for example. These systems use techniques such as lockstep execution and voting. In addition, the systems have to go through special certification. The hypervisor just gets in the way too much.

However, a train has a lot more software, written by multiple suppliers, each of which used to provide completely separate hardware too. For example, a lot of functionality is related to remote control. It is common for an engineer in the front of the train to drive locomotives at the back of the train, possibly two miles away and on the other side of the hill. Radio-based communication is also used to drive locomotives from the ground at stations or maintenance facilities ("a giant train set", Kraeling called them).

It is very common therefore to have not just a separate processor per application, but even to duplicate components such as cell phone modems. A hardware platform that supports virtualization can go against this trend by encouraging consolidation. Deploying all these systems onto a single multi-core processor saves money and enables more reuse of hardware. And if functionality can be added to an existing system just by dropping a new virtual machine (VM) into it, there is no problem if some applications are developed for a specific Linux version, for a legacy OS, or even for Windows CE. This flexibility is another advantage of virtualization.

However, consolidation requires some level of sandboxing for the different VMs. This is not unlike a data center and Kraeling, in fact, used the analogy of a "data center on wheels" several times; but some of the specific use cases for isolation are interesting. It can be hard to patch locomotive software, because—unlike commodity servers—locomotives cost millions of dollars and customers really do not want them to stay idle while testing updates. The requirements this imposes on the quality of software are obvious, but you also need to make the most out of the testing time you can get in the field. For example, an effective way to validate new software is to place it on production locomotives just for the purpose of collecting logs until the software crashes, as well as for collecting data about the crash itself. But it is not stable, so the code is not really in use: it must not interfere with other VMs and with the control systems. Virtualization helps a lot with this kind of sandboxing, of course.

Hardware and software

After explaining the use cases, Kraeling presented the system's hardware and software. The processors selected are x86 and ARM. x86 processors are used for compute-heavy applications. ARM is mostly used for networking services, though 64-bit ARM has the potential to replace x86 as well. All processors are quite low-end, and they often run a 32-bit hypervisor and operating system in order to save memory. Xen was faster than KVM on 32-bit x86, so Xen was used there; on the other hand, KVM was faster than Xen for ARM hardware. GE wants to use KVM with 64-bit ARM as well when the processors are ready.

The focus on low-end x86 and low-power ARM systems is due to power-consumption concerns, which can be an issue for the locomotive's computer systems. This may be surprising, because on a diesel locomotive you basically have a power plant at hand. However, power is directly related to heat, and a locomotive can easily reach 70°C. If the surrounding environment is as hot as the processor, you cannot easily solve overheating problems by adding fans. And even though there is room for fans at the bottom of the chassis, customers do not like maintaining and cleaning them. The low-end, power-conscious x86 processors (GE uses the BayTrail E3845 and the Broadwell 5500U) can run at those temperatures without fans.

For the management layer, there's no standardized tool yet, but GE is looking into OpenStack. A member of the audience pointed out that the libvirt project was started to bridge the differences between Xen and KVM, so it may help GE as well if it does not need the complexity of OpenStack.

Kraeling has tested containers as well. The isolation and flexibility is, of course, not as good as you can get from virtual machines, but they were faster on ARM, so they are looking into CoreOS and Docker. He hasn't yet looked at why x86 didn't benefit from containers. My guess would be that different applications are running on the two systems; CPU-bound applications are quite efficient in a virtual machine.

While high-level, the talk gave an interesting glimpse into a field that most developers are not familiar with; such "embedded" usage of a data center hypervisor like Xen or KVM is not a well-known topic. But, in fact, a presentation [PDF] from KVM Forum 2010 has some striking similarities with Kraeling's. Embedded virtualization is probably more frequent than one would think, and will probably become even more common in combination with realtime virtualization.

Comments (5 posted)

What's new in TeX, part 1

Development in the world of the TeX document-preparation system is steady, gradual, and solid. This fact reflects the maturity of Donald Knuth's TeX engine, which underlies everything else in the system. TeX has been essentially frozen for about 30 years and is one of the very few large software systems that is generally considered to be bug-free. Development now occurs in those layers that surround the typographic core: formats that supply higher-level control of TeX for the ready production of various classes of documents and packages for drawing, slides, sheet music, poetry, and for tweaking TeX's behavior.

In this two-part series, we will look at recent developments in the world of TeX (including LaTeX and similar systems). Considering the pace of development in the TeX community, the notion of "new" that I have in mind is a time horizon of five years or so, although I might mention things that happened even before then. This first part will touch upon typography, programming TeX, and creating diagrams.

TeX basics

Although TeX is still essentially oriented toward the creation of static, paginated documents—and might seem to be losing some relevance in our online world—it is still widely used, especially by mathematicians and scientists in quantitative fields (physics, computer science, etc.). The core reason for this is the same reason that TeX is also popular with various authors and publishing houses in the humanities—those who publish typographically demanding scholarly editions, perhaps mixing languages that employ a variety of alphabets. TeX's purpose is to achieve the best possible automated typography.

This can be seen not only in its unparalleled rendering of mathematical equations, but in its attention to aesthetics in the setting of prose: TeX contains sophisticated algorithms that adjust line breaking and hyphenation to optimize the appearance of entire paragraphs and pages considered as wholes. This attention to detail becomes critically important in complex documents, where typography becomes part of the expression of ideas.

The official and predominant installation method for TeX is TeX Live, which was traditionally distributed on a DVD and is still available in that form. To get the most recent versions of all its parts, however, you will want to follow the usual procedure and install TeX Live through the network. The versions available through Linux package management systems are usually out of date, but the current release is available from the project site. The release notes for TeX Live 2015 are a list of relatively minor, technical details, so I won't be discussing those changes specifically.

The timeliest and most complete source of documentation for the hundreds of TeX packages will be on your disk after you install TeX Live. Open the file /usr/local/texlive/2015/index.html in your browser for links to manuals and examples in various languages.

For those unfamiliar with TeX, when you process a document with the TeX typesetting system, you do so by invoking one of several TeX engines. The various engines differ in the output format that they produce and in how they implement some of TeX's algorithms—which determines what additional features are available. The original tex engine predates Unicode (so it expected an ASCII file) and produced only DVI (for "device independent") files. DVI was intended to be translated into PostScript or other printer commands with a separate tool. Contemporary TeX engines, though, can produce PDF files (e.g., pdfTeX), can understand Unicode text (e.g., XeTeX), and even incorporate scripting languages (e.g., LuaTeX).

The TeX engines should not be confused with TeX document formats, which are large collections of macros that define a set of higher-level layout commands. The most well-known format is LaTeX; another format (which has become popular for book publishing) is ConTeXt. Formats and engines are orthogonal: the pdftex and pdflatex commands invoke the same engine, but the former will process only plain TeX, whereas the latter supports LaTeX.

Fonts and Typography

You should probably work with the LuaLaTeX or XeLaTeX engines for new projects (or their plain-TeX equivalents, LuaTeX and XeTeX), unless you must use TeX packages that are incompatible with these engines or you require a particular feature that's only available with the traditional, PDF-based engine pdfLaTeX.

The reason for this advice is that LuaLaTeX and XeLaTeX have both feet firmly in Unicode land, and their font handling is far more flexible and straightforward than that of the venerable alternatives. One of the annoying drawbacks of TeX in the past was that it lived in its own font universe, and could only use the typefaces that were designed for it.

Generally, TeX was blind to all the other beautiful fonts that you might have installed on your computer. With XeLaTeX and LuaLaTeX, though, you can now easily use any OpenType or TrueType font on your system. And, as we shall shortly see, the maturing of the fontspec and unicode-math packages in recent years radically improves the font-handling landscape for TeX users.

Here is a minimal LaTeX document that shows how to make arbitrary font changes, selecting from among several OpenType/TrueType fonts—some in the TeX Live directory tree and some in the system font directories:

    \documentclass{article}
    \usepackage{fontspec}
    \defaultfontfeatures{Scale=MatchLowercase,Ligatures=TeX}

    \begin{document}

    {\fontspec{Ubuntu}The }{\fontspec{Fetamont Bold10}quick }
    {\fontspec{Punk Nova}brown {\bfseries fox }}
    {\fontspec{Sawasdee}jumps {\itshape{\bfseries over }}}
    {\fontspec{CMU Serif}{\scshape the }}
    {\fontspec{Overlock}lazy }{\fontspec{Ubuntu Condensed}dog.}

    \end{document}

This file is intended to be processed with the lualatex command, which allows us to use the common names of fonts rather than having to know their actual filenames—as is required by all other engines, including XeLaTeX. This convenience is one of several reasons that lualatex should probably be your preferred typesetting command, unless you need to use a package or feature that only works with one of the others.

The defaultfontfeatures command in the third line of the example selects two options for the fontspec package. The Scale=MatchLowercase option scales the various fonts so that their lower-case letter heights are optically equal: fonts with the same nominal point size can appear to be different sizes, so this option makes them blend better when mixing fonts within a line. The Ligatures-TeX option enables the familiar TeX ligatures, such as "``" for an opening quotation mark.

In the code sample, bfseries select the boldface variant of the currently selected (or default) font; itshape and scshape select the italic and small caps variants, respectively. The code sample also shows how these can be combined to produce, in this example, boldface italics.

Here is the result when you process the file with lualatex.

The image was made by cropping the PDF output and converting it to a PNG. You can see where LuaLaTeX has chosen the appropriate font variants in response to the font attribute commands (bfseries, scshape, etc.), some of which are nested. This works because the example uses fonts with these variants available; if the needed variants are not available, those commands will be ignored.

I've actually used this style of ad-hoc font switching when making posters and name tags but, for the more usual kind of document, you will want to select a harmonious set of fonts at the beginning and use them consistently throughout, switching among them with the standard commands for italic, monospace, etc., as required. Here is how you do this with fontspec:

    \documentclass{article}
    \usepackage{fontspec}
    \defaultfontfeatures{Scale=MatchLowercase,Ligatures=TeX}
    \setmainfont{Overlock}[BoldFont={* Black}, BoldItalicFont={* Bold Italic}, SmallCapsFont={* SC}]
    \setmonofont{PT Mono}

    \begin{document}

    The {\bfseries quick} {\itshape brown fox {\bfseries jumps over}}
    {\scshape the {\tt lazy dog.}}

    \end{document}

Running this through lualatex gives the result in the second figure.

In the options to the setmainfont command (which, unusually, come after the main argument), the asterisks stand for the main font name. This provides a convenient shorthand for selecting font variants. Fontspec is incredibly flexible, allowing you to choose entirely different typefaces for bold, italic, etc., if you want to. You can also choose which font features are activated in every situation; for example, you can decide to use historical ligatures when italics are used, but not in upright text.

LuaLaTeX and XeLaTeX both allow you to use Unicode input without including any additional packages. This lets you replace the traditional TeX commands for accents, and it allows the use of any characters available in the font. This is a Turkish translation of the common English pangram we used in the preceding examples:

    {\fontspec{CMU Serif} hızlı kahverengi tilki tembel köpeğin üstünden atlar}

When inserted into our minimal document example, it is typeset as in this figure:

Note, though, that this approach will fail if you choose a font without the glyphs required. For example, attempting to set the above line using the Overlock font will simply skip the "ğ", which is missing from that font.

With the addition of the unicode-math package, Unicode input can even be used in equations. This package also builds its typeset mathematical output using Unicode glyphs, and it allows you to select any math font without loading additional packages:

    \documentclass{article}
    \usepackage{unicode-math}
    \begin{document}

    Here is the elementary version of Stokes' Theorem:

    \medskip

    XITS (STIX) Math:

    \setmathfont{xits-math.otf}
    
    \[ ∫_Σ ∇ ⨯ 𝐅 ⋅ dΣ = ∮_{∂Σ}𝐅⋅d𝐫 \]

    \end{document}

The results of running this through luatex can be seen in the figure below. A longer example also showing other variations is available by clicking on the thumbnail.

Using Unicode math input clearly leads to source files that are easier to read, but it may not be to your liking if your system or text editor makes the input of Unicode too cumbersome. You can, of course, freely mix traditional TeX math markup with direct Unicode input.

If you do use Unicode characters for math in your source files, you must take care to use the symbols with the correct meaning, rather than merely the correct appearance. In the example file, we've used the uppercase Greek Sigma (U+03A3) to represent the surface of integration. There is, however, another Unicode character that will appear almost identical in the source file, but which is intended to mean the summation operator (U+2211).

When typesetting equations, TeX treats letters (variables) and operators differently, as it must. So, if you accidentally use the operator sigma, the size and spacing of the symbol will be incorrect, and the equations will look quite wrong.

(Note that if you get an error upon loading unicode-math, you may have to reinstall TeX Live 2015. There was a conflict with another package that was only fixed a few weeks before I write this—perhaps a counterexample to my advice to download a recent version rather than settling for the one in your distribution's repository.)

Programmability

TeX is not only a system of declarative markup tags for text and equations. It is also a Turing-complete programming language, meaning that it can express arbitrary computations. Many popular LaTeX packages perform computations in TeX in order to work their magic, but it is an arcane and tricky language to program in, and quite difficult to read.

LuaTeX (which includes the LuaLaTeX engine) is a project that embeds the Lua language within TeX. It's still officially in beta, but over the last few years has become stable and mature enough that LuaLaTeX is now considered the preferred engine for new projects. It is the focus of future development, and the ConTeXt project has adopted it as its official engine.

Lua is a scripting language designed specifically for embedding, and is therefore small and efficient. It has a familiar, imperative syntax and can be immediately understood with no previous exposure to the language. After a few minutes with the documentation, anyone who knows Python or any similar language can write basic programs in Lua. LuaTeX embeds Lua in such a way that it has access to the internals of TeX, and it can be used to manipulate the boxes and other elements that make up the typeset result. It can also make the results of Lua calculations available to TeX for typesetting. A simple example should make clear how this can be useful:

    \documentclass{article}
    \usepackage{luacode}
    \begin{document}
    \pagestyle{empty}
    \begin{luacode*}
      function esn (n)
        return (1 + 1/n)^n
      end
      function etn (n)
        tex.print(string.format('%5d & %1.8f \\\\', n, esn(n)))
      end
    \end{luacode*}

    Convergence to $e$:

    \begin{tabular}{ll}
    \rule[-2mm]{0pt}{4mm}$n$ & $(1 + \frac{1}{n})^n$ \\
    \hline
    \luadirect{
      for n = 10, 110, 10 do
        tex.print(etn(n))
      end
    }
    \hline
    \end{tabular}

    \end{document}

In the first part of this document, we have a luacode environment, where we have defined two functions. The first (esn()) maps a number n to a simple expression that yields the number e in the limit as n goes to infinity. The second function (etn()) prints a string that can be embedded within normal TeX source.

The LaTeX code begins next, with a line introducing a table that will show values of n and the convergents approaching e. Within the table, the columns are built with a luadirect command that immediately executes its argument as Lua code, using calls to one of the functions defined earlier. The typeset result is shown in the figure. The ability to perform calculations and typeset the results in a single TeX file, using a language that is simple to program in, opens up a world of new possibilities, especially for authors of mathematical material.

Another strong use case for LuaTeX is in the automated creation of PDF documents from assorted data sources—for example, consider forms of database publishing, such as the printing of catalogs from product databases. In these cases, there is no TeX formatting in the original data, so some form of flexible mapping from data structures to TeX concepts leading to the final PDF is required. The embedded scripting provided by LuaTeX makes this easier than the alternatives.

Graphics

The PGF/TikZ package, a huge project that provides a complete solution to creating all sorts of diagrams within a TeX document, has learned several new tricks in recent years. A recent article introduced TikZ's new network-graph facilities, including the exploitation of LuaTeX to implement the automated layout of graph diagrams. Here, we'll show how to combine a little scripting along the lines illustrated above with TeX's graphics packages:

    \documentclass{article}
    \usepackage{luacode}
    \usepackage{pgfplots}
    \begin{document}
    \begin{luacode*}
      function esn (n)
        return (1 + 1/n)^n
      end
      function etp (n)
        tex.print(string.format('(%5d, %1.8f)', n, esn(n)))
      end
    \end{luacode*}

    \begin{tikzpicture}
      \begin{axis}[xlabel=$n$, ylabel=$(1 + \frac{1}{n})^n$]
        \addplot coordinates{ 
          \luadirect{
            for n = 10, 110, 10 do
               tex.print(etp(n))
            end
          }
        };
        \addplot[red] coordinates { %
          (0, \luadirect{tex.print(math.exp(1))}) %
          (110, \luadirect{tex.print(math.exp(1))}) };
       \end{axis}
    \end{tikzpicture}

In this example we've replaced the second Lua function with one that prints out a pair of coordinates that we can use in a PGFPlots command. The import of pgfplots in the document preamble creates the tikzpicture environment. The axis environment and \addplot command within the tikzpicture environment invoke the pgfplots subsystem, which provides a specialized language for drawing graphs. The syntax it supports is designed to be more convenient than using plain TikZ for this purpose.

The result is shown in the following figure, where the approach of the convergents to the limit, the transcendental number e (shown as a red line), is illustrated.

With the advent of these easy-to-learn tools, it has become possible to undertake a project such as an entire mathematics textbook, with all calculations and graphing done within a single TeX document.

TikZ and PGF recently received a major update to version 3.0. Because of the power and relative ease of use of PGF and TikZ, much of the action in the past few years on the TeX graphics front has taken the form of new TikZ packages.

A few interesting recent additions or major upgrades are Sa-TikZ, for the automated drawing of switching networks; bondgraph, for making "bond graphs" of physical systems; hf-tikz, which allows you to highlight parts of formulas; the randomwalk package, which calculates and prints 2D random walks; tikzorbital, for drawing colorful pictures of atomic and molecular orbitals; tqft, for topological quantum field theory; and forest, for drawing linguistic trees. All of these packages are documented within the TeX Live installation.

The world of TeX has come a long way since I started using it, when we edited our files on a terminal attached to a remote computer, and checked our output by jogging to the computer room to pick up our printouts.

The next installment of this series will delve into the interaction between the traditional world of TeX, which began as a way to typeset documents for printing, and our current environment of electronic documents that adapt to an assortment of reading devices.

Comments (48 posted)

Python and crypto-strength random numbers by default

There are various types of random number generators (RNGs) that target different use cases, but a programming language can only have one default. For high-security random numbers (e.g. cryptographic keys and the like), it is a grievous error to use the wrong kind of RNG, while other use cases are typically more forgiving. The Python community is in the middle of a debate about how it should be handling random numbers within the language's standard library.

The random module

On the python-ideas mailing list, Guido van Rossum kicked off the discussion by noting that he had been contacted by OpenBSD founder Theo de Raadt about the random.random() RNG that ships with Python. That RNG is based on the Mersenne Twister (MT) algorithm, which is a pseudo-RNG (PRNG) with an enormously long period. The PRNG can either be seeded by the user (with a call to random.seed()) or, by default, it is seeded with 2500 bytes of entropy from the os.urandom() pool (the strong, non-blocking random pool supplied by the operating system). From a given seed, the same sequence of random numbers will be generated, which makes it deterministic.

Using MT-derived random numbers for cryptographic or other security purposes is definitely not recommended by security experts. Given enough output from the RNG, the seed can be recovered, which means that all further random numbers can be perfectly predicted—generally with disastrous results. But for many uses, such as games, testing, or simulations, the random module provides a simple-to-use way to get perfectly adequate random numbers. The Python documentation for the module clearly indicates that it should not be used for security purposes and suggests either os.urandom() or the random.SystemRandom class.

But Van Rossum noted that "as the meme goes -- nobody reads the docs", so De Raadt's concern is that people will simply reach for the default provided by the language—no matter what they are using the random numbers for. According to Van Rossum, De Raadt is advocating the arc4random() RNG that OpenBSD uses (and, in fact, has switched to by default). Some of that advocacy can be seen in a forwarded email message that Van Rossum posted. Though De Raadt was invited to participate in the thread, "he's too busy" to do so, which left Van Rossum in a bit of a quandary:

That set off quite a discussion, which now spans four threads in python-ideas.

Donald Stufft responded that he sees a problem with the current approach, but that he doesn't have any "great solution" for it:

But, as Paul Moore and others pointed out, those with non-security needs for random numbers should be able to expect to access them easily from the standard library. He often uses the random module for games of various sorts, as well as for things like Monte Carlo simulations where millions of results are used. While the reproducibility feature is not particularly important to him, he would expect that a simple API would provide it. People doing crypto or other security programming ought to be looking for an RNG beyond the default:

Steven D'Aprano concurred:

Part of the problem, of course, is that there is a large body of existing code that expects the random module to behave in a certain way. Some of it requires the reproducibility feature and isn't being used for security purposes. Changing the default would break those programs a few years down the road (if the change follows the normal Python deprecation schedule). As Stephen J. Turnbull put it:

Stufft's proposal

That first thread got rather long, so Stufft attempted something of a reboot in a message that was rather long itself. He had talked to De Raadt to try to better understand his concerns, which Stufft said were not being reflected in the earlier thread. He broke down the users of random numbers into three groups: those who need deterministic output (and reproducibility), those who need cryptographic-strength random numbers (and don't want determinism), and those who aren't sure if their use case is security-sensitive or not. The last group generally doesn't need deterministic output, but currently that's what they get by default. It is Stufft's belief (that is shared with De Raadt, he said) that the members of that group would be better served with a crypto-strength RNG.

Those in group #3 might be told to use os.urandom() (or the random.SystemRandom class that is based on it), but they may end up using the standard random functions due to either the performance of the stronger alternative or bad advice (if, indeed, their need is security-sensitive). In fact, Stufft said, the performance of os.urandom() is one of the main reasons not to make it the default:

Stufft looked at StackOverflow and other sites with Python code to see what kinds of advice there was about RNG choices. His post had several snippets of code that used the module-level random functions in dubious ways.

That led him to propose that Python change its random-number handling in several ways. It would provide a means to access a fast user-space RNG like arc4random() and add a new class (random.SomeKindOfRandom) that uses it. The proposal would also move the MT-based RNG to a random.DeterministicRandom class and deprecate the module-level functions in the random module. That would mean that users would (eventually) have to choose the kind of randomness they wanted before they would be able to get random numbers. They would call functions like:

    random.SomeKindOfRandom().random()
    random.DeterministicRandom().random()

SystemRandom would still be available for those who wanted that RNG mechanism. Obviously, SomeKindOfRandom would need a better name, though there are objections to the whole idea of doing user-space RNG. Stufft pointed out that security expert Thomas Ptacek is not in favor of user-space RNGs like arc4random().

As might be guessed, Stufft's suggestions were not met with universal acclaim. There were concerns about backward compatibility and for how today's code could be switched, with minimal changes, to run (correctly) in this new world.

But Nick Coghlan strongly objected to the idea of deprecating all of the module-level functions. As he pointed out, nearly all of the module-level functions could provide any kind of random numbers, though they need to have good performance. Making someone choose what type they want, before they can even get a random number is "a *huge* regression in Python's usability for educational use cases". He said that a common "hello world" kind of program for using random numbers in Python is to roll a six-side die:

He continued:

He noted that there are calls at the module-level that imply the need for a deterministic RNG (like seed() and others that are stateful) and which can be deprecated to help those who really need them to move to an alternative API. Essentially, he is asking that those who don't care about the RNG wars (and those new to Python in particular) be left out—they can still call the functions they call today.

Breaking compatibility "in the name of the public good" is the wrong approach, Antoine Pitrou said. He and others are tired of seeing security trump all other considerations whenever a topic like this is raised. Since a change like Stufft has suggested can't fix the problem of bad advice on the internet, compromising on a change that maintains compatibility is preferable.

Coghlan's proposals

Coghlan started another thread as something of a "pre-PEP" on enhancing the random module. It would create three types of RNGs in Python: seedable, seedless, and system. The seedless version would be the default and guidance would be provided that those who need deterministic random numbers should use the seeded version, while those with strong security needs should use the system RNG.

Coghlan's proposal lays out plans for Python 3.6, which is likely destined for late 2016 (or early 2017), and for 3.7, which is probably due sometime in 2018—a change of this nature for Python can take a while, for sure. Essentially, his idea is to deprecate the stateful methods and module-level functions (seed(), getstate(), and setstate()), except for the SeedableRandom class. Those functions would warn the user, but not actually cause an error—except if the default type has been changed.

The most controversial part of Coghlan's proposal would provide a random.set_default_instance() function to specify which of the three types would be used by the module-level functions. Both Stufft and D'Aprano are concerned that would allow any module to affect all of the random numbers generated from that point on, potentially nullifying the choice another module has made.

Beyond that, though, Moore would like to see some kind of cost/benefit analysis. He is trying to remain open-minded, but is concerned about the disruption for users, particularly those who don't have any security requirements for their use of random numbers. While he recognizes that it is somewhat emotionally stated, the benefits to him seem to be: "Users of code written based on bad advice will be protected from the consequences (as long as the code runs on a sufficiently new version of Python)".

Based on some of that feedback, Coghlan went ahead and drafted PEP 504 that simplifies earlier proposals. It dispenses with the user-space CSPRNG (e.g. arc4random() or something similar) and changes the default to the system RNG (i.e. random.urandom()). If a program uses random.seed() or the other stateful calls (i.e. getstate() or setstate()), the random module will switch to the existing MT-based deterministic PRNG.

He proposes that this be done for 3.6, with a silent-by-default deprecation warning for the fallback to MT. In 3.7 it would instead be a visible-by-default runtime warning. In his announcement of the PEP, he said: "That approach would provide a definite security improvement over the status quo, while restricting the compatibility break to a performance regression in applications that use the module level API without calling seed(), getstate() or setstate()."

Van Rossum's reaction

But it turns out that Van Rossum is not particularly happy with that PEP as the end result of the discussion. He stopped following the "mega-threads", but doesn't see much value in changing things:

Though Stufft reiterated his belief that the existing module "guides you towards using an  insecure source of random numbers rather than a secure one", Van Rossum was not impressed with that argument:

The discussion is continuing at the time of this writing, but there is a sense that adding access to a fast user-space PRNG, perhaps based on ChaCha20 (like arc4random()), may be in the offing. A change in the default would leave some users and programs behind eventually, but it would seem that most believe users who require deterministic random numbers make up a small minority. Van Rossum, though, seems unconvinced that there is much urgency for a solution. He suggested adding the new method for 3.6, but waiting "a few releases" to decide if any change to the default was warranted. "Security isn't served well by panicky over-reaction", he said.

There is plenty of time left in the 3.6 development schedule, so perhaps other proposals and ideas will eventually win the day. But with Van Rossum firmly opposed to the changes that have been proposed so far, a big change seems unlikely for 3.6. Only time will tell.

Comments (29 posted)

Security quotes of the week

— Bruce Schneier

— Sean Gallagher reports on a panel at the Intelligence & National Security Summit in Ars Technica

— Nick Coghlan

— Bruce Schneier (again)

Comments (1 posted)

freetype2: denial of service

Package(s):	freetype2	CVE #(s):	CVE-2014-9745 CVE-2014-9746 CVE-2014-9747
Created:	September 14, 2015	Updated:	October 7, 2015
Description:	From the Mageia advisory: It was discovered that FreeType did not correctly handle certain malformed font files. If a user were tricked into using a specially crafted font file, a remote attacker could cause FreeType to crash or hang, resulting in a denial of service, or possibly expose uninitialized memory (Savannah bugs 41309 and 41590). The CVE numbers were referenced in the Debian LTS advisory.
Alerts:	openSUSE openSUSE-SU-2015:1704-1 freetype2 2015-10-09 Debian DSA-3370-1 freetype 2015-10-06 Debian-LTS DLA-319-1 freetype 2015-09-30 Mageia MGASA-2015-0367 freetype2 2015-09-13

Comments (none posted)

icedtea-web: applet execution

Package(s):	icedtea-web	CVE #(s):	CVE-2015-5234 CVE-2015-5235
Created:	September 14, 2015	Updated:	June 9, 2016
Description:	From the Arch advisory: - CVE-2015-5234 (unexpected permanent authorization of unsigned applets) It was discovered that IcedTea-Web did not properly sanitize applet URLs when storing applet trust settings. A malicious web page could use this flaw to inject trust-settings configuration, and cause applets to be executed without user approval. - CVE-2015-5235 (applet origin spoofing) It was discovered that IcedTea-Web did not properly determine an applet's origin when asking the user if the applet should be run. A malicious page could use this flaw to cause IcedTea-Web to execute the applet without user approval, or confuse the user into approving applet execution based on an incorrectly indicated applet origin. A remote attacker is able to execute a malicious applet without user approval via multiple vectors.
Alerts:	Scientific Linux SLSA-2016:0778-1 icedtea-web 2016-06-08 Oracle ELSA-2016-0778 icedtea-web 2016-05-13 Red Hat RHSA-2016:0778-01 icedtea-web 2016-05-10 Ubuntu USN-2817-1 icedtea-web 2015-11-24 openSUSE openSUSE-SU-2015:1595-1 icedtea-web 2015-09-22 Fedora FEDORA-2015-15677 icedtea-web 2015-09-21 Fedora FEDORA-2015-15676 icedtea-web 2015-09-21 Mageia MGASA-2015-0376 icedtea-web 2015-09-17 Arch Linux ASA-201509-6 icedtea-web 2015-09-14

Comments (none posted)

ipython: cross-site scripting

Package(s):	ipython	CVE #(s):	CVE-2015-6938
Created:	September 15, 2015	Updated:	October 8, 2015
Description:	From the Mageia advisory: In IPython, local folder name was used in HTML templates without escaping, allowing XSS in said pages by carefully crafting folder name and URL to access it.
Alerts:	openSUSE openSUSE-SU-2015:1699-1 IPython, 2015-10-08 Fedora FEDORA-2015-14901 ipython 2015-09-18 Fedora FEDORA-2015-14902 ipython 2015-09-18 Mageia MGASA-2015-0372 ipython 2015-09-15

Comments (none posted)

kernel: privilege escalation

Package(s):	kernel	CVE #(s):	CVE-2015-6666
Created:	September 14, 2015	Updated:	September 23, 2015
Description:	From the Red Hat bugzilla: After fixing Linux's NT flag handling, an optimization was added, making the code vulnerable. A malicious 32-bit program might be able to leak NT into an unrelated task. On a kernel with setting CONFIG_PREEMPT=y, this causes a straightforward DoS. With CONFIG_PREEMPT=n setting, it's probably still exploitable for DoS with some more care. This vulnerability could be possibly used also for privilege escalation.
Alerts:	Fedora FEDORA-2015-15933 kernel 2015-09-23 Fedora FEDORA-2015-15130 kernel 2015-09-11

Comments (none posted)

moodle: information leak

Package(s):	moodle	CVE #(s):	CVE-2015-3177
Created:	September 16, 2015	Updated:	September 16, 2015
Description:	From the CVE entry: Moodle 2.8.x before 2.8.6 does not consider the tool/monitor:subscribe capability before entering subscriptions to site-wide event-monitor rules, which allows remote authenticated users to obtain sensitive information via a subscription request.
Alerts:	Fedora FEDORA-2015-14996 moodle 2015-09-15 Fedora FEDORA-2015-14988 moodle 2015-09-15

Comments (none posted)

onionshare: denial of service

Package(s):	onionshare	CVE #(s):
Created:	September 11, 2015	Updated:	September 16, 2015
Description:	From the onionshare bug report: Some users might be running Tor Browser confined with AppArmor profiles included in Tor Browser Launcher. In this case, OnionShare needs to make sure chooses a hidden service directory in a place that tor has permissions to write to.
Alerts:	Fedora FEDORA-2015-14716 onionshare 2015-09-10 Fedora FEDORA-2015-14715 onionshare 2015-09-10

Comments (none posted)

openldap: denial of service

Package(s):	openldap	CVE #(s):	CVE-2015-6908
Created:	September 14, 2015	Updated:	September 30, 2015
Description:	From the Arch Linux advisory: By sending a crafted packet, an attacker can cause the OpenLDAP daemon to crash with a SIGABRT. This is due to an assert() call within the ber_get_next method (io.c line 682) that is hit when decoding tampered BER data. The following proof of concept exploit can be used to trigger the condition: # echo "/4SEhISEd4MKYj5ZMgAAAC8=" | base64 -d | nc -v 127.0.0.1 389 The above causes slapd to abort as follows when running with '-d3', however it should be noted that this will crash the server even when running in daemon mode. A remote attacker is able to send specially crafted packets that cause the OpenLDAP server to crash, leading to denial of service.
Alerts:	SUSE SUSE-SU-2016:0262-1 openldap2 2016-01-27 openSUSE openSUSE-SU-2016:0261-1 openldap2 2016-01-27 openSUSE openSUSE-SU-2016:0255-1 openldap2 2016-01-27 SUSE SUSE-SU-2016:0224-1 openldap2 2016-01-25 openSUSE openSUSE-SU-2016:0226-1 openldap2 2016-01-25 SUSE SUSE-SU-2016:0778-1 sles11sp4-docker-image 2016-03-15 Oracle ELSA-2015-1840 openldap 2015-09-29 Oracle ELSA-2015-1840 openldap 2015-09-29 Oracle ELSA-2015-1840 openldap 2015-09-29 CentOS CESA-2015:1840 openldap 2015-09-29 Scientific Linux SLSA-2015:1840-1 openldap 2015-09-29 CentOS CESA-2015:1840 openldap 2015-09-29 CentOS CESA-2015:1840 openldap 2015-09-29 Red Hat RHSA-2015:1840-01 openldap 2015-09-29 Ubuntu USN-2742-1 openldap 2015-09-16 Debian DSA-3356-1 openldap 2015-09-12 Mageia MGASA-2015-0374 openldap 2015-09-15 Debian-LTS DLA-309-1 openldap 2015-09-14 Arch Linux ASA-201509-4 openldap 2015-09-12

Comments (none posted)

php-doctrine-annotations: privilege escalation

Package(s):	php-doctrine-annotations	CVE #(s):	CVE-2015-5723
Created:	September 15, 2015	Updated:	August 2, 2016
Description:	From the Doctrine project: We are releasing new versions of Doctrine Cache, Annotations, ORM and MongoDB ODM today that fix a security misconfiguration vulnerability. This vulnerability was assigned CVE-2015-5723. It requires an attacker to have direct access to a user of the server to be exploitable. We consider exploitabilty to be low to medium. Exploiting this vulnerability can allow attackers to perform local arbitrary code execution with privleges of other users (privilege escalation). You are only affected by this vulnerability, if your application runs with a umask of 0.
Alerts:	Fedora FEDORA-2016-8dc0af2c29 php-doctrine-common 2016-08-01 Fedora FEDORA-2016-fa7e683c6e php-doctrine-common 2016-08-01 Fedora FEDORA-2016-7e229134f9 php-doctrine-orm 2016-07-19 Fedora FEDORA-2016-f0c8b7b115 php-doctrine-orm 2016-07-20 Fedora FEDORA-2015-f1e18131bc php-ZendFramework 2015-11-09 Fedora FEDORA-2015-6d70a701bf php-ZendFramework 2015-11-09 Fedora FEDORA-2015-2e7c06c639 php-ZendFramework 2015-11-08 Mageia MGASA-2015-0391 php-ZendFramework/php-ZendFramework2 2015-10-09 Debian DSA-3369-1 zendframework 2015-10-06 Fedora FEDORA-2015-15198 php-doctrine-cache 2015-09-14 Fedora FEDORA-2015-15203 php-doctrine-annotations 2015-09-14 Fedora FEDORA-2015-15204 php-doctrine-doctrine-bundle 2015-09-14 Fedora FEDORA-2015-15206 php-doctrine-doctrine-bundle 2015-09-14 Fedora FEDORA-2015-15199 php-doctrine-cache 2015-09-14 Fedora FEDORA-2015-15201 php-doctrine-annotations 2015-09-14

Comments (none posted)

phpMyAdmin: guessable user credentials

Package(s):	phpMyAdmin	CVE #(s):	CVE-2015-6830
Created:	September 14, 2015	Updated:	October 5, 2015
Description:	From the Red Hat bugzilla: A vulnerability allowing to complete reCaptcha test and subsequently perform a brute force attack to guess user credentials without having to complete further reCaptcha tests was found. This vulnerability only affects installations with reCaptcha test enabled. Affected versions are 4.3.x (prior to 4.3.13.2) and 4.4.x (prior to 4.4.14.1)
Alerts:	Debian DSA-3382-1 phpmyadmin 2015-10-28 openSUSE openSUSE-SU-2015:1674-1 phpMyAdmin 2015-10-05 Mageia MGASA-2015-0366 phpmyadmin 2015-09-13 Fedora FEDORA-2015-15385 phpMyAdmin 2015-09-13 Fedora FEDORA-2015-15386 phpMyAdmin 2015-09-13

Comments (none posted)

qemu: two vulnerabilities

Package(s):	qemu	CVE #(s):	CVE-2015-6815 CVE-2015-6855
Created:	September 15, 2015	Updated:	October 14, 2015
Description:	From the Mageia advisory: Qemu emulator built with the e1000 NIC emulation support is vulnerable to an infinite loop issue. It could occur while processing transmit descriptor data when sending a network packet. A privileged user inside guest could use this flaw to crash the Qemu instance resulting in DoS (CVE-2015-6815). Qemu emulator built with the IDE disk and CD/DVD-ROM emulation support is vulnerable to a divide by zero issue. It could occur while executing an IDE command WIN_READ_NATIVE_MAX to determine the maximum size of a drive. A privileged user inside guest could use this flaw to crash the Qemu instance resulting in DoS (CVE-2015-6855).
Alerts:	SUSE SUSE-SU-2016:1785-1 kvm 2016-07-11 SUSE SUSE-SU-2016:1745-1 xen 2016-07-06 SUSE SUSE-SU-2016:1698-1 kvm 2016-06-28 SUSE SUSE-SU-2016:1318-1 xen 2016-05-17 openSUSE openSUSE-SU-2016:0995-1 xen 2016-04-08 SUSE SUSE-SU-2016:0955-1 xen 2016-04-05 openSUSE openSUSE-SU-2016:0914-1 xen 2016-03-30 SUSE SUSE-SU-2016:0873-1 xen 2016-03-24 Gentoo 201602-01 qemu 2016-02-04 openSUSE openSUSE-SU-2015:2249-1 xen 2015-12-10 openSUSE openSUSE-SU-2015:2003-1 xen 2015-11-17 openSUSE openSUSE-SU-2015:1964-1 xen 2015-11-12 SUSE SUSE-SU-2015:1952-1 xen 2015-11-10 SUSE SUSE-SU-2015:1908-1 xen 2015-11-04 SUSE SUSE-SU-2015:1894-1 xen 2015-11-03 SUSE SUSE-SU-2015:1853-1 xen 2015-10-30 SUSE SUSE-SU-2015:1782-1 qemu 2015-10-20 Fedora FEDORA-2015-d6ea74993a xen 2015-10-14 Fedora FEDORA-2015-4896530727 xen 2015-10-14 Fedora FEDORA-2015-16369 qemu 2015-10-09 Fedora FEDORA-2015-16368 qemu 2015-10-09 Fedora FEDORA-2015-015 xen 2015-10-04 Fedora FEDORA-2015-efc1d7ba5e xen 2015-10-04 Ubuntu USN-2745-1 qemu, qemu-kvm 2015-09-24 Debian DSA-3362-1 qemu-kvm 2015-09-18 Debian DSA-3361-1 qemu 2015-09-18 Mageia MGASA-2015-0368 qemu 2015-09-15 Mageia MGASA-2015-0369 qemu 2015-09-15

Comments (none posted)

qemu: denial of service

Package(s):	qemu	CVE #(s):	CVE-2015-5239
Created:	September 15, 2015	Updated:	September 16, 2015
Description:	From the Mageia advisory: Qemu emulator built with the VNC display driver is vulnerable to an infinite loop issue. It could occur while processing a CLIENT_CUT_TEXT message with specially crafted payload message. A privileged guest user could use this flaw to crash the Qemu process on the host, resulting in DoS.
Alerts:	Debian-LTS DLA-574-1 qemu-kvm 2016-07-30 Debian-LTS DLA-573-1 qemu 2016-07-30 SUSE SUSE-SU-2016:1785-1 kvm 2016-07-11 SUSE SUSE-SU-2016:1698-1 kvm 2016-06-28 SUSE SUSE-SU-2016:1560-1 qemu 2016-06-13 openSUSE openSUSE-SU-2016:0995-1 xen 2016-04-08 SUSE SUSE-SU-2016:0658-1 Xen 2016-03-04 openSUSE openSUSE-SU-2015:2249-1 xen 2015-12-10 openSUSE openSUSE-SU-2015:2003-1 xen 2015-11-17 openSUSE openSUSE-SU-2015:1964-1 xen 2015-11-12 SUSE SUSE-SU-2015:1952-1 xen 2015-11-10 SUSE SUSE-SU-2015:1908-1 xen 2015-11-04 SUSE SUSE-SU-2015:1894-1 xen 2015-11-03 SUSE SUSE-SU-2015:1853-1 xen 2015-10-30 Fedora FEDORA-2015-015 xen 2015-10-04 Fedora FEDORA-2015-efc1d7ba5e xen 2015-10-04 Ubuntu USN-2745-1 qemu, qemu-kvm 2015-09-24 Mageia MGASA-2015-0368 qemu 2015-09-15

Comments (none posted)

vzctl: insecure ploop-based containers

Package(s):	vzctl	CVE #(s):	CVE-2015-6927
Created:	September 14, 2015	Updated:	January 11, 2017
Description:	From the Debian advisory: It was discovered that vzctl, a set of control tools for the OpenVZ server virtualization solution, determined the storage layout of containers based on the presence of an XML file inside the container. An attacker with local root privileges in a simfs-based container could gain control over ploop-based containers. Further information on the prerequites of such an attack can be found at https://src.openvz.org/projects/OVZL/repos/vzctl/commits/...
Alerts:	Gentoo 201701-30 vzctl 2017-01-11 Debian DSA-3357-1 vzctl 2015-09-13

Comments (none posted)

Kernel release status

The current development kernel is 4.3-rc1, released on September 12. "I decided that I'm not interested in catering to anything that comes in tomorrow, and I might as well just close the merge window and do the -rc1 release." In the end, 10,756 non-merge changesets were pulled during this merge window.

Stable updates: 4.1.7, 3.14.52, and 3.10.88 were released on September 13.

Comments (none posted)

Quotes of the week

— Ingo Molnar

— Ted Ts'o

— Eric Biederman

Comments (14 posted)

4.3 Merge window, part 3

Last week's merge window summary ended with a guess that the bulk of the changes for 4.3 had been seen by that point. By the time Linus released 4.3-rc1 and closed the merge window, 10,756 non-merge changesets had been pulled into the mainline repository; that's about 550 since last week. So the patch rate did indeed fall off as expected, but there were still some significant changes that slipped in before the window closed.

Significant user-visible changes include:

• The new file /proc/kpagecgroup can be used to determine which memory control group each page of physical memory is charged to.

• The idle-page tracking feature has been merged. This feature enables the discovery of memory pages that are not in active use; this information can be used to optimize the allocation of memory between containers or virtual machines.

• The membarrier() system call, which has been circulating since (at least) 2010, has been merged. See this commit for the latest description and man page.

• The control-group writeback improvement patches have been merged.

• New hardware support includes Microchip LAN88XX PHYs, NXP LPC18xx/43xx watchdog timers, Atmel SAMA5D4 watchdog timers, Toradex Colibri VF50 touchscreens, and Freescale i.MX6UL touchscreen controllers.

get_vaddr_frames()

With regard to changes visible to kernel developers: there has been one significant addition to the memory-management interface. Certain driver subsystems (media, for example) have long had to reach deep into the memory-management subsystem to implement high-performance I/O to user-space buffers. Some of the resulting code raised eyebrows in memory-management circles; it also stood in the way of efforts to make the mmap_sem semaphore less of a bottleneck.

Jan Kara has been working on reducing mmap_sem use for a while; that effort has extended into improving the memory-management primitives used in the driver tree. In 4.3 he has added a new set of helpers for the easy mapping of I/O buffers. It creates a new type, struct frame_vector, to describe a mapped buffer. That structure lives in <linux/mm.h>, but it's probably best if most users treat it as if it were an opaque structure.

A frame vector can be allocated or destroyed with:

    struct frame_vector *frame_vector_create(unsigned int nr_frames);
    void frame_vector_destroy(struct frame_vector *vec);

Here, nr_frames is the maximum number of pages that will be mapped using this vector. The mapping itself is done with:

    int get_vaddr_frames(unsigned long start, unsigned int nr_frames,
		         bool write, bool force, struct frame_vector *vec);

The beginning virtual address of the buffer to be mapped is passed in start, and the size of the buffer in nr_frames. If write is set, the buffer will be mapped for write access; if force is set, write access will be set up even if the buffer is mapped read-only in user space. After a successful call, vec will contain the results of the mapping, and the return value will be the number of pages actually mapped.

If the buffer lives in ordinary memory, get_vaddr_frames() will take a reference to each mapped page to keep it in RAM. That reference must be released at some point to unpin the pages; to do so, call:

    void put_vaddr_frames(struct frame_vector *vec);

Note that frame_vector_destroy() does not make this call; users must take care to do it themselves.

Once upon a time (i.e. last year), this type of interface would have returned an array of struct page pointers to refer to the mapped pages. The increasing use of not quite real memory in hardware has created pressure to use page-frame numbers (PFNs) instead. As it happens, the contents of the frame vector may be either struct page pointers or PFNs, depending on the type of memory mapped. Driver-level code need not be aware of this practice, but it does have to be explicit about what it wants. To gain access to the mapped buffer (for DMA mapping, for example), use one of:

    struct page **frame_vector_pages(struct frame_vector *vec);
    unsigned long *frame_vector_pfns(struct frame_vector *vec);

The call to frame_vector_pages() can fail if it is not possible to represent the buffer using page structures; the error code will be returned as an ERR_PTR() value, so a macro like IS_ERR() should be used to check the returned pointer before using it. Conversion to PFNs is unconditionally successful in the current implementation.

All of the above functions are exported to modules (with no explicit GPL limitation). Driver code has been fixed up in a number of places (example) to use the new interface; the result is a net reduction in lines of code and, hopefully, an improvement in robustness.

In summary

Meanwhile, the 4.3 development cycle has now entered the stabilization phase. For the curious, Stephen Rothwell's post-merge-window summary gives some statistics about the changes that were merged this time around. It seems that 94% of them were in the linux-next tree at the beginning of the merge window, with the DRM and networking trees being the biggest source of commits that weren't there. Some 587 commits in linux-next didn't make it into the mainline; nearly a quarter of those come from the kdbus tree, which was not proposed for merging this time around. Also not merged was OrangeFS, which ran into some trouble when the pull request went out; chances are good OrangeFS will make it in for 4.4.

Comments (none posted)

Extended system call error reporting

The interface between the kernel and user space is, in places, surprising in its complexity. There are numerous tasks that involve passing detailed information about hardware configurations, process state, and more, in either direction. When something goes wrong, though, that communication channel narrows to a single integer error code, often making it difficult for developers to figure out what is going on. There have been various proposals for widening the error-reporting interface in the past; the latest proposal, from Alexander Shishkin, may not get any further than its predecessors, but it does show that there is ongoing interest in the problem.

As an example, consider the VIDIOC_S_FMT ioctl() call, provided by the media subsystem. Its job is to set the format of images returned to user space from a capture device (such as a camera). There is a mind-boggling variety of possible image formats out there, so the format description passed to the kernel from user space contains a number of complex, interrelated parameters. There are a lot of ways such a description can go wrong — and that's before the vagaries of specific driver implementations are taken into account. Should there be a problem, though, the only thing user space is likely to know is that the VIDIOC_S_FMT call returned EINVAL. The kernel, of course, knows what it was objecting to, but there is no way to communicate that knowledge to user space.

Fixing that problem is not easy; the errno mechanism is clearly inadequate, but it is set in stone by several decades of Unix tradition and cannot be changed without breaking applications. So any extended error information must be carried by a new channel that can be ignored by unaware applications. The addition of error information to the kernel must also be done carefully, so as to avoid slowing down kernel hot paths or clogging the source with an overwhelming set of error messages. Alexander's patch attempts to meet all of these criteria.

How the mechanism works is best illustrated with some examples. In his patch set, Alexander targets the perf_event_open() system call; it takes a perf_event_attr structure as a parameter. That structure has a vast and growing set of parameters describing the events to be captured and, correspondingly, there are a lot of ways in which things can go wrong.

Describing errors

The first step is to create a structure that describes an error site — a place where an error is detected and passed back to user space. That structure should include a field called site that holds an ext_err_site structure; beyond that, it can contain any information needed to fully report the error to user space. In the perf case, that structure looks like this:

    #include <linux/exterr.h>

    struct perf_ext_err_site {
	struct ext_err_site	site;
	const char		*attr_field;
    };

The attr_field member is meant to hold the name of the field in struct perf_event_attr that generated the error.

Then, it is necessary to define a function that can turn any extra information in this structure into a string to be passed back to user space. The perf version is:

    static char *perf_exterr_format(void *site)
    {
	struct perf_ext_err_site *psite = site;

	return kasprintf(GFP_KERNEL, "\t\"attr_field\": \"%s\"\n",
			 psite->attr_field);
    }

Note that this function returns a dynamically allocated string; the extended error infrastructure will free that string when it is no longer needed.

With these two pieces in place, it is possible to define an "error domain" that handles a specific class of errors — perf errors in this case:

    DECLARE_EXTERR_DOMAIN(perf, perf_exterr_format);

The actual reporting of error information is done by way of a rather frightening bit of macro magic called ext_err(). Real users will almost certainly wrap it, though; this is how it is done in the perf code:

    #define perf_err(__code, __attr, __msg)				\
	({ /* make sure it's a real field before stringifying it */	\
	    struct perf_event_attr __x; (void)__x.__attr;		\
	    ext_err(perf, __code, __msg, 				\
	        .attr_field = __stringify(__attr));			\
	})

The parameters to ext_err() are the name of the domain defined above, the (negative) error code, a message to be reported to user space, and a set of initialization strings to initialize the rest of the error-site structure. In this case, the final parameter to ext_err() sets the attr_field of the perf_ext_err_site structure to the name of the erroneous attribute. See this patch for an actual invocation of the perf_err() macro.

There are a couple of other important details. One is that the EXTERR_MODNAME symbol must be set separately before calling ext_err():

    #define EXTERR_MODNAME	"perf"

The other is that ext_err() returns a value, which is a modified version of the error code passed into it. This code can be thought of as an index into an array of ext_err_site structures describing every extended error known to the kernel. The normal way to return the error code to user space will then be through a line like:

    return ext_err_errno(code);

The modified code from ext_err() must not be returned directly to user space, since applications will have no idea what it means. On the other hand, the original error code should not be returned without calling ext_err_errno(); that call is the one that causes the kernel to remember the extended error information. In short, there is a new task_struct field called ext_err_code; the call to ext_err_errno() causes the special error code to be placed into that field. If an ordinary (non-extended) error code is passed to ext_err_errno(), the right thing will happen, so it is safe to use in situations where a support code might return ordinary or extended error codes.

The user-space side

At this point, the kernel is prepared to tell user space about an extended error, but the return value from the system call can still only be the ordinary errno value that it has always been. If the application wants to know more, it can make a call like:

    char message[SIZE];

    len = prctl(PR_GET_ERR_DESC, message, SIZE);

The return value is not just an ordinary message; it is a string in the JSON format containing the file and line where the error was generated, the error code, the module name, the actual message, and any specific information added by the domain format function described above. The changes to the user-space perf tool duly include a new JSON parser to pick this message apart again. The prctl() call will clear the error information on the kernel side, so a second call will return no data.

The patch set has, thus far, not seen much in the way of review comments. In the end, the error-reporting issue is one that most developers recognize, but few feel up to trying to fix. So it is hard to say whether this attempt to widen the error-reporting channel from the kernel will meet with success or not. Ancient traditions are hard to change but, every now and then, somebody succeeds.

Comments (22 posted)

The LPC Android microconference, part 2

The Linux Plumbers Android microconference was held in Seattle on August 20th. It included discussions of a variety of topics, many of which need to be coordinated within the Android ecosystem. The microconference was split up into two separate sessions; this summary covers the second session, which was held for three hours in the evening. Topics were toybox in Android, improving AOSP vendor trees, providing per-task quality of service, and improving big.LITTLE on Android.

The first session's summary can be found here.

Toybox in Android

Rob Landley started the second half of the microconference with some background on the toybox project, which has recently been included in the Android Open Source Project (AOSP), replacing some components of Android's toolbox. Landley described his early attempts to learn how to build a distribution from scratch, starting in 1999, which he would use to create his Aboriginal Linux project. He picked up the BusyBox tool for his project and, after improving it, became the maintainer.

It was after the GPLv3 came out that the trouble started, Landley said. Some code contributed to BusyBox was GPLv2-only, which caused trouble with relicensing. So efforts to audit the code, then remove and replace any GPLv2-only submissions, were made. He became disillusioned with what he saw as the "breaking" of "the GPL", as no longer was there just "the GPL", and it was no longer a universal receiver of code. In the past, most open-source licenses were GPL-compatible and could be up-converted to GPL as needed. With the introduction of GPLv3, along with lots of GPLv2-only code in existence, suddenly this was no longer the case. He handed over BusyBox maintainership to the best person he knew for the job, and started playing with toybox, which he wrote under what he calls a "BSD zero-clause" license—basically public-domain—which would allow the code to be compatible with any other license.

However, soon after starting toybox, he mothballed the project. This was mostly due to the fact that he liked the new BusyBox maintainer and didn't want to undermine the project by leaving and immediately starting a competitor. Also BusyBox had a ten-year head start, which made it intimidating to try to catch up.

Landley then started talking about a paper he had co-written a decade ago that described how big transitions in computing are always opportunities, since big established players are often unseated by smaller upstarts. The paper was focused on the transition to 64-bit systems and how this was an opportunity for Linux to unseat Windows.

Landley realized that the transition to mobile was a similar major transition, and Apple's iOS was likely to eventually unseat existing workstations if it were to become dominant. He saw Android as thus becoming an important vehicle for the preservation of a non-proprietary future. His history with Aboriginal Linux made him interested in trying to make Android self-hosting. Unfortunately, due to Android not accepting any GPL code in user space, BusyBox would not be able to be used, but toybox could be. Thus, he restarted development on toybox.

At this point, Karim Yaghmour had a few questions, the first being: how far is toybox currently away from BusyBox? Landley pointed to a online status page that explains (with a somewhat complicated key) what parts are left to do. There is also a roadmap, which helps show the scope of what the project is trying to achieve. That list includes: POSIX-2008, Linux Standard Base (LSB), a self-hosting environment, as well as fully replacing Android's toolbox and Tizen's core tools.

Yaghmour also asked which commands from toybox have not yet replaced the equivalents in Android's toolbox. Again, he referred people to the status page, but Landley also pointed out that you can look in AOSP at the toolbox Android.mk file, which will clarify which non-toybox tools are still being used. Landley noted that Elliott Hughes of the Android team has been helping with some of the testing of toybox. Hughes syncs upstream toybox to the AOSP tree every two weeks.

At this point the talk ran out of time.

Improving vendor AOSP repositories

The next talk (slides [PDF]) was a discussion led by John Stultz from Linaro on some of the issues he's seen in various vendor trees, and what might be done to improve things. Most vendors utilize a fork-and-try-to-forget model with AOSP, targeting each device with its own tree. Linaro isn't any different, really, as it maintains quite a number of separate AOSP trees for efforts like 96board support, work on Project Ara, as well as for other projects. This behavior is problematic, since it makes operating system updates more complicated, so vendor device updates suffer. That results in real security difficulties, as recently seen with the Stagefright issue.

Additionally, devices do tend to include functionality from a number of different vendors: a system-on-chip (SoC) from one vendor, Bluetooth and WiFi from another, sensors from another, etc. As noted in previous talks in the microconference, integrating vendor hardware abstraction layers (HALs) into an AOSP repository is usually a non-trivial process. HALs do not integrate into the tree in a uniform way and some vendor HALs require tweaks to the framework layer. This all complicates things further when it comes time to handle a release update, since there are now multiple parties that are being depended on for updates, which all have to be integrated together.

Another area of pain is the build system. The device.mk and BoardConfig.mk files used to describe the device usually contain a large set of global build variables, which have no expressions of inter-dependencies (for example, enabling BOARD_HAVE_BLUETOOTH_BCM won't do anything if one forgets to enable BOARD_HAVE_BLUETOOTH), and the device.mk files that list out the PRODUCT_PACKAGES to be included have a ton of duplication from device to device. In fact, it seems most vendors create mid-layer common directories and have logic to try to share some of the standard entries for different devices, just to avoid the heavy duplication required.

AOSP is not really structured to be able to host community contributions, which is another issue. Google has reasonably limited AOSP to hosting device-specific code only for Nexus devices, which the company is able to test and maintain. While this is a understandable and practical decision for Google, it keeps vendors and others in the community working in their own private trees. Since there's no reason to submit code, this results in a limited culture of review, so there isn't necessarily a community sense for what is good or bad code.

Further, there's a missing sense of best-practices. This may not be true for some vendors who are closer partners with Google, but for the wider community that can't attend the private boot-camp meetings, it is. There is a lack of documentation, so things like HAL integration approaches often end up being done in a cargo-cult manner. Improving documentation and having better examples are areas that need work.

After running through the issues, the discussion was opened up to try to see what could be done to improve things, and not just from the "Google should do X" angle, but also what the community-at-large could potentially implement.

One question was "should HALs be submitted to AOSP?", but since AOSP doesn't accept non-Nexus device support, this didn't seem appropriate. Additionally, many HALs are completely proprietary, so licensing issues would prevent that from happening. There is also the fact that vendors are quite often focused on shipping products, so it's not clear that, even if the code was welcome to be submitted for review, many vendors would take the extra time required to deal with the feedback of that code review. So the angle of finding processes to make things easier for the vendors should be considered.

There was also a suggestion that the community create some space where code that couldn't go into AOSP be collected. This might be a possibility, but it would be good to avoid the "creating another fork to solve all the forks" sort of solution.

One idea for the build system is to try to reduce the amount of duplicative code in the device directories with something like Kconfig. This would help express configuration dependencies, reducing the number of options required to be specified, and ideally make it easier to build for multiple devices with only a change to the configuration file. Samuel Ortiz mentioned that Intel basically does this, though it doesn't have the dependency tracking. It uses a configuration file to define the device and some common infrastructure in the tree processes that file. Stultz noted that many vendors have something like this to make it easier to build multiple devices; it points to something that may need to be shared generically.

Dmitry Shmidt of Google's Android team asked how Linaro handles doing validation for devices it doesn't have access to. The answer was that it doesn't, and it's a problem. However, another point was raised that the Linux kernel deals with this all the time, since people don't have test machines for all architectures, and it's handled by delegating testing responsibility to architecture maintainers. It was noted that a perspective from the Chrome OS folks might be useful, as they've been able to delegate testing for a wider array of devices.

Rom Lemarchand from Google mentioned that he would like to see more vendors submit code to AOSP. But some attendees said that it is hard to get patches reviewed on Gerrit. A number of folks in the room agreed, saying they had run into similar problems. The Google developers said that it sounded like maybe the auto-adding of maintainers in Gerrit was broken. They promised to look into it and get it solved quickly.

While Google controls commit access to AOSP, its Gerrit installation allows anyone to review and comment on proposed changes. It was proposed that a group of non-Google folks could make a pointed effort to review patches submitted to Gerrit, which would help build up a better community sense of code taste and might even lead to growing external maintainers. Lemarchand mentioned that it would also be nice because it would allow the Google team to better understand whose reviews could be trusted in the community.

The discussion sort of dwindled at this point, so Stultz suggested moving on to the next talk.

Providing per-task quality of service

Next Juri Lelli from ARM talked (slides [PDF]) about his work on the energy-aware scheduler (EAS) and the potential for use of deadline scheduling in Android. He started with an overview of the EAS, describing how its focus is on per-entity load tracking and per-entity utilization tracking, which try to size up tasks so that they can be properly placed on the right core in an asymmetric multi-processing environment. The goal is to use as little energy as possible while still getting good performance. When the system is not over-utilized, the scheduler will try to pack tasks onto a single CPU, but once a tipping point is crossed and that one CPU is overloaded, it falls back to the conventional approach of spreading the load around.

He described how EAS ties more of the CPU-power logic like dynamic voltage and frequency scaling (DVFS) together into the scheduler, by allowing the scheduler to trigger cpufreq governors directly and make decisions about the speed of each CPU that is being scheduled. This allows for the scheduler to ramp up a processor's frequency to increase capacity if it wants to add a task to that CPU. He also talked a little bit about schedtune, which provides a single sysfs knob to boost performance on a global or per-cgroup basis.

At this point it was noted that most of the EAS discussion focused on energy, while the interactive cpufreq governor in Android is mostly focused on latency, so it was asked: how does latency come into the picture and how can it be controlled? Lelli noted that the interactive cpufreq governor tends to boost frequency quickly to provide interactive latency benefits to the user. The schedtune sysfs knob allows for similar boosting, so when an event occurs Android user space could provide some short-term boosting to improve latency response. Riley Andrews from Google noted that the interactive cpufreq governor gives this benefit without user space having to do anything. Though Lelli pointed out that the governor does it a bit blindly, so this allows a more informed and focused response that could save power.

Andrews was also curious about all the various knobs that were available via EAS. Lelli noted that the out-of-tree HMP scheduler, used by some vendors for big.LITTLE machines, had way too many knobs and was very difficult to tune. So, with EAS, more of the logic has been kept internal in the scheduler so it is easier to get it right. Andrews also wondered about the heuristics for task placement. Lelli noted that there are heuristics and the scheduler needs to make a guess and try to compute the difference in energy used so it can try to give the best tradeoff in performance and energy usage. It was suggested that for more information, folks attend the EAS microconference session that would be going on the next day.

Lelli then switched to the deadline scheduler, which he has been maintaining with others. He mentioned some of the benefits of SCHED_DEADLINE over other policies like SCHED_FIFO, like how it avoids starvation issues that SCHED_FIFO can have. It works using a resource-reservation system that guarantees that a task will get a specific amount of CPU time in a given period. The scheduler just needs to be provided the runtime, period, and deadline values; then it will indicate if it can achieve those constraints or not.

He showed graphs of the performance implications of this model for tasks like movie playback and wanted to know if folks were interested in using this for Android. His thought was that SurfaceFlinger or AudioFlinger might be able to use this policy. One question that came up was how does the deadline scheduler's reservation system handle CPU frequency changes. Lelli replied that it doesn't at the moment and that there needs to be some integration of the policies, so that the deadline scheduler can specify a minimum frequency in order to guarantee that the deadlines are met.

There was also some discussion on how it would interact with SCHED_FIFO tasks. It was clarified that SCHED_DEADLINE runs at a higher priority than SCHED_FIFO, but it can be used so that you can, for example, give a guaranteed limit of 10% of runtime to SurfaceFlinger. That means you don't have to deal with starvation issues, which had been mentioned as being a problem in making the audio pipeline run as a SCHED_FIFO task at Google I/O. Andrews said that he had looked at it for SurfaceFlinger earlier, but had some issues with the interface and the cpufreq issue was problematic as well. There were a few more questions, but it was suggested that they be brought up at the EAS microconference the next day.

Improving big.LITTLE on Android

Todd Kjos (who was standing in for Tim Murray) from Google's Android team then started reviewing the team's experience dealing with big.LITTLE devices (slides [PDF]). For the most part, big.LITTLE issues and the complex tuning required have been left to the vendors to sort out, as Google hasn't directly addressed it so far. However, that's starting to change. He showed a few examples of the complexities of different styles of asymmetric CPUs that they have seen, from more standard big.LITTLE pairings to more complicated ones, where there might be different sets of "little" CPUs running at different speeds, in addition to the "big" CPUs.

He showed some graphs of the power-performance curves for the different CPUs, which may have cross-over points where it becomes obvious it's worth jumping from the smaller CPUs to the bigger ones. However, he also showed charts where there might be a gap between the two curves, making tuning much more complicated, since to gain any performance at that gap, you have to make a much bigger jump in power consumption.

He described some of the changes in Android M to help. Android keeps track of which tasks are foreground and which tasks are background tasks, so it now uses cpusets to help pin all background tasks to the small CPU or CPUs.

Bobby Batacharia from ARM asked if all asynchronous tasks are considered background tasks. Kjos and Andrews responded that not all are, it depends on which tasks they're interacting or running with. Mark Gross then asked how much work is done in background tasks. The Android developers said not much, but it can be intermittent, though they didn't have hard numbers.

Andrews then mentioned that the team really does want to better enforce the notion of foreground and background tasks. Even when background tasks are limited by cpusets and lower priorities, some still cause scheduling interference with foreground tasks. Stultz asked if pushing foreground tasks to SCHED_RR (the round-robin realtime scheduling class) would solve this, but Andrews noted that the team was really avoiding making tasks with non-deterministic runtimes run as round-robin.

Stultz then asked how the cpusets interacted with Android's use of cgroups. Kjos clarified that they help together. Android still uses cgroups to regulate task CPU usage, but the cpusets help pin tasks to processors, so the tasks don't otherwise fan out to the big CPUs. Batacharia noted that EAS should help with that problem.

As the session wrapped up, Kjos indicated that these types of systems will be an area of greater focus for his team in the coming year.

At that point, after over six hours, the microconference came to an end, and everyone quickly left for dinner and drinks.

[Thank you to all the presenters for their discussions, Karim Yaghmour for organizing and running the conference, and Rom Lemarchand for helping get so many of the Google Android team to attend.]

Comments (15 posted)

How Debian managed the systemd transition

Debian's decision to move to systemd as the default init system was a famously contentious (and rather public) debate. Once all the chaos regarding the decision itself had died down, however, it was left to project members to implement the change. At DebConf 2015 in Heidelberg, Martin Pitt and Michael Biebl gave a down-to-earth talk about how that implementation work had gone and what was still ahead.

Pitt and Biebl are the current maintainers of the systemd package in Debian, with Pitt also maintaining the corresponding Ubuntu package. The pair began with a brief recap of the init-replacement story, albeit one that steered mercifully clear of the quarrels and stuck to the technical side. Initial discussions for replacing the System V init system began as far back as 2007, but pressure grew in recent years, included considerable demand from system administrators and upstream projects (typically wanting specific features like support for logind or journald). Once the Technical Committee had made its decision to adopt systemd as the default, Pitt said, "the real work" began.

The jessie release

Only a few months remained between the decision (in February of 2014) and the first freeze for "jessie" that November. Nevertheless, the migration was completed and jessie shipped with systemd 215. In the end, Biebl said, getting systemd into shape turned out not to be as big of a deal as had been feared—the systemd package in "wheezy" was well-tested. On the other hand, the requirement that administrators would be able swap in one of the other init-system packages added quite a bit of complexity. First, the old init system had long been marked as "essential," which meant that removing it required the user to fight quite a few warnings and protests from the package manager and other software. And, just as importantly, the idea of easily swapping init systems in and out was a new one for Debian.

Eventually, the systemd team split the existing sysvinit into several packages, which made it easier to cope with dependent packages that assumed some part of sysvinit would be available. The team also created a new init meta-package, which allowed them to ensure that users would not accidentally remove all of the init-system packages from their system.

Supporting init-system swapping is not merely a package-management problem, however: users are likely to expect some system state to be preserved between any two init systems. For example, if a service is started at boot time by one init system, the user likely expects that service to be started in a comparable fashion under the other init system, too. To ensure this preservation of state, the team re-implemented a subset of systemd's systemctl interfaces in a new package called init-system-helpers.

Along the way, team members also created a package called dh-systemd, which lets Debian package maintainers ensure that their package will be properly configured regardless of which init system the user employs. Finally, they created the systemd-shim package, which supports certain downstream projects like GNOME that have hard-coded dependencies on individual systemd components.

Although the team worked to develop these interface and compatibility packages, it took a different approach when considering what to do with sysvinit's collection of configuration files—such as those in /etc/default or /etc/rcS.d. "The major reason was that we would have to carry a patch for that for all eternity," Biebl said. So they performed a one-time migration of several such settings files.

The integration work consisted of a lot of small changes, Biebl said, "but I think we succeeded. I think that people don't realize that they are using systemd." He noted that people have actually approached him asking how they can start using systemd and been surprised when he told them that they already are.

Along those same lines, he noted that users who upgrade an older Debian system configured with sysvinit to the jessie release are automatically migrated to systemd, and few have noticed. The goal in undertaking this "scary" change, he said, was to have jessie systems be the same regardless of whether they are new installations or updated ones. For upgraded systems, though, there is a fallback sysvinit init binary installed, complete with a bootloader option to boot the system with it if the user encounters a problem with systemd.

There are bugs in every software package, of course. Biebl said that the team braced for a flood of bug reports when jessie was released, but that flood never came. Most of the bugs that have been reported have stemmed from the fact that systemd is stricter in situations where sysvinit tries to hide errors. For example, Pitt said, Debian had a longstanding bug in its ecryptfs-setup-swap package; systems ended up getting configured with unencrypted swap or no swap at all, and thanks to sysvinit, no one noticed for several years. But systemd complained, and now the bug has been fixed.

Looking forward to stretch

Pitt then turned the discussion to the changes that are in store for the next Debian release, "stretch." The first change is to udev, which will begin assigning predictable, stable names for network interfaces (in place of names using the ambiguous "eth0" form). This change has already landed, he said, and should fix problems that many users have encountered in the past. But there are some wrinkles involved: interface names cannot be migrated to the new format automatically when upgrading to stretch, because doing so could break (for example) firewall rules written with the old names in mind.

Another upcoming change is support for networkd, a lean interface for bringing up and taking down network interfaces. Pitt noted that there have been a lot of questions about whether networkd is meant to replace NetworkManager. It is not, he explained; rather, it is akin to ifupdown in its scope. But Debian users will benefit from it because it better handles virtual interfaces and hot-plugging.

Biebl explained another change that is still in the works: the ongoing process of removing sysvinit's rcS.d scripts. There is a wiki page detailing the current status and the roadmap of the process. Because these scripts are executed early in the boot process, they can be difficult to remove, and the removals can trigger dependency cycles. He noted that Felipe Sateler has started working on this task, and in doing so has located quite a few packages that need attention from their maintainers.

Other systemd-related changes that may or may not land in time for stretch include the addition of kdbus support. The Debian systemd team regards kdbus as a beneficial addition, since it would be available at the earliest stages of the boot process, but whether Debian includes it will depend on its inclusion in the kernel. There is, of course, an out-of-tree kdbus module, and Debian users interested in testing it out can do so using the systemd tools packaged for Debian unstable.

There are plenty of opportunities for new volunteers to help out, Pitt and Biebl said. But, just as importantly, Debian developers and package maintainers could start taking advantage of systemd's "shiny new features," including timers, socket activation, or security confinement. Where things go from here, in other words, depends at least as much on how Debian as a whole chooses to use its init system as it does on the team that maintains the init-system packages.

[The author would like to thank the Debian project for travel assistance to attend DebConf 2015.]

Comments (102 posted)

Distribution quotes of the week

-- Andrew Haley

-- Matthew Miller

Comments (2 posted)

OpenWrt "Chaos Calmer" 15.05 released

The OpenWrt 15.05 release is out. This release includes a number of new features, including improved package signing, support for hardened builds and jails, a lot of new hardware support, and much more. (See also: LWN's review of the 15.05 release from July).

Full Story (comments: 51)

Release for CentOS Linux 7 Rolling media

The August snapshot for CentOS 7 is available. "This release includes CentOS Linux 7 iso based install media, Generic Cloud images, Atomic Host, Docker containers, Vagrant images, vendor hosted cloud images and live media."

Full Story (comments: none)

Summary of the Debian CD BoF at DC15

Steve McIntyre presents a summary of the CD BoF and DebConf. Topics include the current state, no new CDs, non-free images, new image types, and more.

Full Story (comments: none)

Bits from the Debian Hamradio Maintainers

The Debian Hamradio Maintainers team share some bits on the things they have been working on. Topics include a new website and future plans, new live images released, and new documentation.

Full Story (comments: none)

GCC 5 in stretch: what next?

The Debian release team reports that gcc-5 is the default in testing (stretch), and looks at ongoing and upcoming transitions. Notably, the team is preparing a Perl transition.

Full Story (comments: none)

Distribution newsletters

• DistroWatch Weekly, Issue 627 (September 14)
• 5 things in Fedora this week (September 11)
• Ubuntu Weekly Newsletter, Issue 343 (September 13)

Comments (none posted)

Libvirt: what went wrong (and right)

At KVM Forum 2015, Michal Prívozník presented a talk entitled "Libvirt: what did we do wrong?" (YouTube video and slides [PDF]). Libvirt is a toolkit to interact with hypervisors and to set up virtual machines and containers. It is used by oVirt and OpenStack, among others. It supports about a dozen hypervisors, either as a bridge to native management tools or (as is the case for KVM) launching and managing guests directly. The purpose of the talk was not only to detail the things that went wrong in libvirt, but also to explain why they happened and to hint at Libvirt's strongest design points.

Libvirt is a complex project. It's a C library with a stable API and bindings for multiple other languages. It has a plugin interface to support multiple hypervisors and container solutions. It also handles the configuration of the host for functionality such as network bridges (with ebtables/iptables), iSCSI connections, and virtual Fibre Channel host bus adapters (with N_Port ID Virtualization or NPIV). Libvirt also supports remote access; API calls can transparently refer to remote machines and the library takes care of establishing a clear text socket connection, a TLS-protected one, or an SSH tunnel to the remote host.

The first thing that went wrong will be familiar to readers of the LWN kernel page: some APIs have no flags argument and thus are not extensible. For example, the virDomainShutdown() function takes a single argument, a pointer to a domain pointer. However, there are multiple ways to shut down a virtual machine. At the very least there are two: through a system interface such as ACPI, and through a remote procedure call (RPC) to a program running in a guest. Therefore a new API function was introduced called virDomainShutdownFlags().

Even when a flags argument was included, new arguments would turn out to be necessary in the future because the APIs were tied to specific hypervisors. For example, virDomainCreateWithFiles() had to be introduced to add the ability to pass file descriptors from the host into a Linux container. One might say that the use case is rather specific and that the new API is also tied to a specific virtualization mechanism.

In general, API names are not generally a strong point of libvirt. virDomainShutdownFlags() was mentioned already; virDomainCreateWithFiles() has an extra "With". It turns out there's also a virDomainCreateWithFlags() function, and virDomainCreateWithFiles() has both a files and a flags argument.

Inconsistency in error conditions was another thing that went unnoticed until it was too late. free(NULL) in C is a no-operation, as is virDomainInterfaceFree(NULL) in libvirt. However, virDomainFree(NULL) is an error. Changing this is difficult at this point, because it is unknown if any user is relying on this.

A different category of mistake is APIs that are inherently subject to time-of-check-to-time-of-use races. Again, this is not a new mistake: the *at() family of system calls was introduced for the same reason in the Linux kernel. In libvirt, the "list active guests" operation was split in two parts: first, it would collect a list of integer IDs and, second, the user would request information for each ID. This API does not make it possible to obtain a consistent view at some point in time.

Some right choices

That said, many other things were done right. In particular, the external API was limited to the minimum necessary for consumers of the library. Libvirt has a large number of internal APIs, which really helps when developing a system library in C. One rather useful subsystem, virCommand, was built to spawn subprocesses with support for file descriptor passing, dropping privileges (UID/GID and capabilities), and more. The developers receive many requests for turning it into a public library. However, the API is good exactly because it was never part of the libvirt ABI, and could grow into a better design over time. Keeping these parts internal, thus, was a good idea, he said.

In a related area, libvirt keeps the internal structure of the objects (domains, networks, storage pools, etc.) hidden to libvirt clients. The clients have to use XML whenever they have to communicate information about the objects. The XML is part of the libvirt API and needs to be kept stable; however, it is much more flexible than C structures and it makes it easy to add new features without having to design tons of APIs. Again, strong separation between internal APIs (which use C structures) and external XML-based APIs simplifies growth.

Some of the XML elements are surely badly named and uselessly hypervisor-specific; these issues come up often enough that Prívozník "could talk for a really long time" about them. In addition, for a long time Libvirt was not even checking the XML against a schema, preventing the API from failing fast. However, the use of XML has become a signature feature of the Libvirt API and Prívozník considers it one of its strong points.

What was the source of the mistakes? Mostly, the lack of a global view of what was happening with multiple hypervisors. Some of the mistakes above can be explained by Libvirt's beginnings as a C wrapper for the Xen RPC. Even today, communication is insufficient about the requirements of different hypervisors and plans for future development. The same problems are solved at different times for different hypervisors; if whoever comes second has additional or different requirements, the resulting API is often suboptimal.

To some extent, this is a consequence of the open-source model, he said. Collaborative development doesn't encourage traditional approaches to requirements design; "agile methodologies" are not easy to apply, either, for a project with a large number of individual contributors scattered around the globe.

Some of the principles of agile methodologies and extreme programming are indeed applied to Libvirt, however. The project developers are not scared of refactoring internal APIs whenever it becomes useful. They write a large number of unit and integration tests and have a monthly release cycle so that consumers do not have to use the Git repository.

A lot of this can be achieved simply by using some discipline and, especially, some common sense, he said. And even if you get some things wrong, it is much more important to get the important points right. In the case of libvirt, the strict boundary between internal and external interfaces, the separation of concerns between its C and XML APIs, and the focus on tests from the beginning were enough for the library to grow at a fast pace and to remain maintainable after almost ten years.

Comments (1 posted)

Quotes of the week

— James Hague, in his list of "Computer Science Courses that Don't Exist, But Should."

— Bjarni Einarsson on naming Mailpile releases.

Comments (5 posted)

KDE Frameworks 5.14.0 released

Version 5.14 of the KDE Frameworks libraries are available. Changes in this release include a number of renamed private libraries, the refactoring of many settings in KActivities, and numerous fixes in the Plasma and KIO libraries.

Full Story (comments: none)

Python 3.5.0 released

The Python 3.5.0 release is out. "Python 3.5.0 is the newest version of the Python language, and it contains many exciting new features and optimizations." See the what's new page and this LWN article for details on the new features in this release.

Full Story (comments: 4)

Bassi: Who wrote GTK+ (Reprise)

At his blog, GNOME's Emmanuele Bassi has published some statistics on developer contributions to GTK+, dating back to the 2.0 release cycle. He also provides some context for interpreting the raw numbers. "Disparity in the length of the development cycles explains why the 2.12 and 2.14 cycles, which lasted a year, represent an anomaly in terms of contributors (148 and 140, respectively) and in terms of absolute lines changed. The reduced activity between 2.20 and 2.24.0 is easily attributable to the fact that people were working hard on the 2.90 branch that would become 3.0." This historical analysis is a follow-up to Bassi's development statistics about GTK+ 3.18, also published this week.

Comments (3 posted)

Development newsletters from the past week

• What's cooking in git.git (September 14)
• LLVM Weekly (September 14)
• OCaml Weekly News (September 15)
• Perl Weekly (September 14)
• PostgreSQL Weekly News (September 13)
• This Week in Rust (September 14)
• Wikimedia Tech News (September 14)

Comments (none posted)

Reavy: WebRTC privacy

At her personal blog, Mozilla's Maire Reavy addresses some common privacy concerns users have brought up regarding WebRTC. The most basic issue is that engaging in a WebRTC chat session can reveal one user's public IP address to the other user in the chat session. As it turns out, there are ways to mitigate the risk. "We’ve added several new privacy controls for WebRTC in Firefox 42. These controls allow add-on developers to build features that give users the ability to selectively disable all or part of WebRTC, and which allow finer control over what information is exposed to JS applications, especially your IP address or addresses. None of these features are enabled by default due to the considerable cost of enabling them to most users (most of them can be also enabled via about:config)."

Comments (none posted)

FSF seeks nominations for the Free Software Awards

The Free Software Foundation is seeking nominations for the Free Software Awards. "The Free Software Awards include the Award for the Advancement of Free Software and the Award for Projects of Social Benefit. The awards are presented each year at the LibrePlanet free software conference, and at the same time nominations for the next year's awards open."

Full Story (comments: none)

FSFE: "Freedomvote" and the Swiss national election

FSFE Switzerland has started the "Freedomvote" campaign, in cooperation with the "Swiss Open Systems User Group“ (/ch/open). "The campaign offers an online portal that lists those candidates who will run for election, and their opinion towards Internet policy, Free Software, and open data formats." Swiss national parliament elections (Nationalrat- / Ständeratswahlen) will be held on October 18, 2015.

Full Story (comments: none)

Italian Defense migrates to LibreOffice

The Document Foundation reports that LibreItalia Association NGO and the Italian Defense have adopted LibreOffice for Office Automation. "Signed by Sonia Montegiove, President of Associazione LibreItalia NGO, and Ruggiero Di Biase, Rear Admiral and General Manager of the Italian Ministry of Defense Information Systems, the collaboration agreement is focused on the adoption of LibreOffice as open source productivity application for Office Automation. With the adoption of LibreOffice, the Italian Ministry of Defense complies with Law 83 of June 22, 2012, which provides a framework for the adoption of open source software by Italian public administrations, to replace proprietary software."

Full Story (comments: none)

Presidential candidate Lawrence Lessig goes one on one with Ars Technica

A bit far afield, perhaps, but Lawrence Lessig is the co-founder of Creative Commons and a proponent of reduced legal restrictions on copyright and trademark. Ars Technica talks with Lawrence about his bid for the US presidency. "Ars: Does your copyleft past help or hurt your presidential bid? Lessig: Whatever you call it, I have the right position on copyright—namely, that it is essential, but needs to be updated to the digital age. If people want to challenge that position, then I’d have to make fair use of the words of Harry Callahan: “Go ahead, make my day.”"

Comments (13 posted)

Library Suspends Tor Node After DHS Intimidation (EFF DeepLinks)

The Electronic Frontier Foundation (EFF) is running a story on its DeepLinks blog that the Kilton Public Library in Lebanon, New Hampshire has suspended its Tor node deployment—at least temporarily—due to criticism by the local police department (we covered the launch of the Kilton library's Tor node in August). The EFF post says that the criticism originated when "a regional Department of Homeland Security office contacted the local police to spread fear, uncertainty, and doubt about Tor. The police got in touch with the library board, who suspended the program until they could vote on it on September 15." The EFF has set up a page at which interested parties can sign a petition showing support for the library, and has written its own letter of support to the Lebanon library board. The Library Freedom Project, which is handling the details of running Kilton's Tor node, has also written about the incident and promises further updates after the library board meeting.

Comments (39 posted)

Library’s Tor relay now restored (Ars Technica)

Last week we reported that the Kilton Public Library in Lebanon, New Hampshire suspended its Tor node deployment due to criticism by the local police department. Ars Technica now reports that the Tor relay has been restored. "As Ars reported earlier, the goal of the Library Freedom Project is to set up Tor exit relays in as many of these ubiquitous public institutions as possible. As of now, only about 1,000 exit relays exist worldwide. If this plan is successful, it could vastly increase the scope and speed of the famed anonymizing network. For now, Kilton has a middle relay but has plans to convert it to an exit relay. A middle relay passes traffic to another relay before departing the Tor network on the exit relay."

Comments (12 posted)

The DevConf.cz 2016 Call For Participation is now open

DevConf.cz will take place February 5-7, 2016 in Brno, Czech Republic. The call for participation deadline is November 30.

Full Story (comments: none)

CFP Deadlines: September 17, 2015 to November 16, 2015

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
September 20	October 26 October 28	Samsung Open Source Conference	Seoul, South Korea
September 21	March 8 March 10	Fluent 2016	San Francisco, CA, USA
September 25	December 5 December 6	openSUSE.Asia Summit	Taipei, Taiwan
September 27	November 9 November 11	KubeCon	San Francisco, CA, USA
September 28	November 14 November 15	PyCon Czech 2015	Brno, Czech Republic
September 30	November 28	Technical Dutch Open Source Event	Eindhoven, The Netherlands
September 30	November 7 November 8	OpenFest 2015	Sofia, Bulgaria
September 30	December 27 December 30	32. Chaos Communication Congress	Hamburg, Germany
October 1	April 4 April 6	Web Audio Conference	Atlanta, GA, USA
October 2	October 29	FOSS4G Belgium 2015	Brussels, Belgium
October 2	December 8 December 9	Node.js Interactive	Portland, OR, USA
October 15	November 21	LinuxPiter Conference	Saint-Petersburg, Russia
October 26	April 11 April 13	O’Reilly Software Architecture Conference	New York, NY, USA
October 30	January 30 January 31	Free and Open Source Developers Meeting	Brussels, Belgium
October 30	January 21 January 24	SCALE 14x - Southern California Linux Expo	Pasadena, CA, USA
October 31	November 20 November 22	FUEL GILT Conference 2015	Pune, India
November 2	December 4 December 5	Haskell in Leipzig	Leipzig, Germany

If the CFP deadline for your event does not appear here, please tell us about it.

SFLC Annual Conference

Software Freedom Law Center's Fall Conference, "FOSS and Global Entrepreneurialism", will be held October 30 in New York City, NY. There will be a panel on license compliance dispute resolution, a presentation of new work on designing non-profit organizations, and much more.

Full Story (comments: none)

Mini-DebConf in Cambridge, UK

There will be a mini-DebConf November 5-8 in Cambridge, UK. There will be a two-day mini-DebCamp followed by two days of conference. "I'm expecting that we will end up discussing and working on the arm ports and other ARM-related topics at the very least, but there's also plenty of scope for other subjects for both sprint work and talks."

Full Story (comments: none)

Events: September 17, 2015 to November 16, 2015

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
September 16 September 18	PostgresOpen 2015	Dallas, TX, USA
September 16 September 18	X.org Developer Conference 2015	Toronto, Canada
September 19 September 20	WineConf 2015	Vienna, Austria
September 21 September 25	Linaro Connect San Francisco 2015	San Francisco, CA, USA
September 21 September 23	Octave Conference 2015	Darmstadt, Germany
September 22 September 23	Lustre Administrator and Developer Workshop 2015	Paris, France
September 22 September 24	NGINX Conference	San Francisco, CA, USA
September 23 September 25	LibreOffice Conference	Aarhus, Denmark
September 23 September 25	Surge 2015	National Harbor, MD, USA
September 24	PostgreSQL Session 7	Paris, France
September 25 September 27	PyTexas 2015	College Station, TX, USA
September 28 September 30	Nagios World Conference 2015	Saint Paul, MN, USA
September 28 September 30	OpenMP Conference	Aachen, Germany
September 29 September 30	Open Source Backup Conference 2015	Cologne, Germany
September 30 October 2	Kernel Recipes 2015	Paris, France
October 1 October 2	PyConZA 2015	Johannesburg, South Africa
October 2 October 3	Ohio LinuxFest 2015	Columbus, OH, USA
October 2 October 4	PyCon India 2015	Bangalore, India
October 5 October 7	LinuxCon Europe	Dublin, Ireland
October 5 October 7	Qt World Summit 2015	Berlin, Germany
October 5 October 7	Embedded Linux Conference Europe	Dublin, Ireland
October 8	OpenWrt Summit	Dublin, Ireland
October 8 October 9	CloudStack Collaboration Conference Europe	Dublin, Ireland
October 8 October 9	GStreamer Conference 2015	Dublin, Ireland
October 9	Innovation in the Cloud Conference	San Antonio, TX, USA
October 10	Poznańska Impreza Wolnego Oprogramowania	Poznań, Poland
October 10	Programistok	Białystok, Poland
October 10 October 11	OpenRISC Conference 2015	Geneva, Switzerland
October 14 October 16	XII Latin American Free Software	Foz do Iguacu, Brazil
October 17	Central Pennsylvania Open Source Conference	Lancaster, PA, USA
October 18 October 20	2nd Check_MK Conference	Munich, Germany
October 19 October 22	Perl Dancer Conference 2015	Vienna, Austria
October 19 October 22	ZendCon 2015	Las Vegas, NV, USA
October 19 October 23	Tcl/Tk Conference	Manassas, VA, USA
October 21 October 22	Real Time Linux Workshop	Graz, Austria
October 23 October 24	Seattle GNU/Linux Conference	Seattle, WA, USA
October 24 October 25	PyCon Ireland 2015	Dublin, Ireland
October 26	Korea Linux Forum	Seoul, South Korea
October 26 October 28	Kernel Summit	Seoul, South Korea
October 26 October 28	OSCON	Amsterdam, The Netherlands
October 26 October 28	Samsung Open Source Conference	Seoul, South Korea
October 27 October 30	OpenStack Summit	Tokyo, Japan
October 27 October 29	Open Source Developers' Conference	Hobart, Tasmania
October 27 October 30	PostgreSQL Conference Europe 2015	Vienna, Austria
October 29	FOSS4G Belgium 2015	Brussels, Belgium
October 30	Software Freedom Law Center Conference	New York, NY, USA
November 3 November 5	EclipseCon Europe 2015	Ludwigsburg, Germany
November 5 November 7	systemd.conf 2015	Berlin, Germany
November 5 November 8	mini-DebConf	Cambridge, UK
November 6 November 8	Dublin blockchain hackathon	Dublin, Ireland
November 6 November 8	Jesień Linuksowa 2015	Hucisko, Poland
November 7 November 8	OpenFest 2015	Sofia, Bulgaria
November 7 November 9	PyCon Canada 2015	Toronto, Canada
November 7 November 8	PyCON HK 2015	Hong Kong, Hong Kong
November 8 November 13	Large Installation System Administration Conference	Washington, D.C., USA
November 9 November 11	PyData NYC 2015	New York, NY, USA
November 9 November 11	KubeCon	San Francisco, CA, USA
November 10 November 12	Allseen Alliance Summit	Santa Clara, CA, USA
November 10 November 11	Open Compliance Summit	Yokohama, Japan
November 10 November 13	Black Hat Europe 2015	Amsterdam, The Netherlands
November 11 November 13	LDAP Conference 2015	Edinburgh, UK
November 14 November 15	NixOS Conference 2015	Berlin, Germany
November 14 November 15	PyCon Czech 2015	Brno, Czech Republic
November 15 November 20	Supercomputing 15	Austin, TX, USA

If your event does not appear here, please tell us about it.