Some improbable 2017 predictions

The discussion on eliminating Debian maintainers faded out with no clear conclusions. But, over the years, it has become increasingly clear that the single-maintainer model used in much of the free-software community creates a great many single points of failure, some of which are guaranteed to fail over any reasonable period of time. Thus, we have seen interest in group maintainership models grow, and that process will continue into 2017. We will certainly not see the end of single-maintainer fiefdoms in 2017, but it seems likely that there will be fewer of them.

Another important single point of failure is Android. It has brought a lot of freedom to the mobile device world, but it is still a company-controlled project that is not entirely free and, by some measures at least, is becoming less free over time. A shift of emphasis at Google could easily push Android more in the proprietary direction. Meanwhile, the end of CyanogenMod has, temporarily, brought about the loss of our most successful community-oriented Android derivative.

The good news is that the efforts to bring vendor kernels closer to the mainline will bear some fruit this year, making it easier to run systems that, if not fully free, are more free than before. Lineage OS, rising from the ashes of CyanogenMod, should help to ensure the availability of alternative Android builds. But it seems likely that efforts to provide free software at the higher levels of the stack (microG, for example) will languish.

The security mess will only get worse, as it becomes clear that vast numbers of severely insecure systems have been deployed, many in important infrastructural roles. Those who have been paying attention have understood the scope of this problem for some time; the rest of the world will catch up quickly. By the end of the year, expect to see attempts at legislative solutions in various countries, many of which will be less than helpful at addressing the real problems.

With luck, the free-software community will have a somewhat better security story by the end of the year, as understanding of the gravity of the problem sinks in. Fuzz-testing and hardening efforts will increase, and some notable progress will be made. But it is certain to be far short of what is needed.

The disintermediation of traditional Linux distributors will continue, with more software being distributed via containerization mechanisms, language-specific repositories, and more. But it will also become clear that this process has a cost; distributors perform a useful service by providing an integrated system, and some users will find that they miss that service. Episodes like the Node.js left-pad fiasco will highlight the risks of performing one's own software integration from multiple repositories under varied management.

We should see another ruling in the appealed VMware GPL-infringement lawsuit. Whether that ruling will bring any joy or any clarity in the area of derived works and the GPL is rather less certain.

The problems of protocol ossification and widely deployed ancient kernels will push more big content providers toward protocols like QUIC or TCP over UDP. Both protocols allow moving a lot of networking logic into user space. There are some advantages to such schemes, including the ability to more quickly deploy protocol enhancements and pushing network intelligence back to the endpoints. But they also will facilitate the creation of private low-level protocols that need not be open source. We could be seeing the beginning of the end of the era where widely used protocol implementations had free reference implementations.

Much of the kernel's core memory-management and filesystem code was designed around a fundamental truth of computing: the latency associated with I/O operations is orders of magnitude larger than memory-access latency. As solid-state memory devices replace traditional storage, that "truth" is increasingly untrue. Fundamental changes will be required for the kernel to perform well on near-future hardware. Those changes will not (for the most part) be made this year, but our understanding of what needs to be done will be much improved by the end of the year.

Containers will be big. Really big. You'll be fed up with hearing about containers by the end of the year. You heard it here first.

2016 was a year that saw many triumphs by divisive forces that would set people against one another. Those forces will certainly not rest in 2017, but they will also be more strongly confronted by those with a vision of a more integrated and cooperative world. The free-software community has, over the last few decades, shown that it is possible to bring together people from all over the planet to work toward shared, long-term goals. We have our own inclusiveness problems to solve and, hopefully, will make progress on those this year. But, if we work at it, we can also serve as an example of how a vast number of people and companies, while working toward their own goals, can build something that serves everybody and makes the world a better place.

As always, the usual disclaimers apply to everything written here. Attributing any level of trustworthiness to your editor's predictions is hazardous at best, if not downright foolhardy. The safest prediction of all is that much of the above will be proved wrong.

With 2017, LWN begins its 20th year of publication, something that we certainly did not predict in 1997 when the idea of starting a Linux-oriented newsletter was being discussed. Let us start this year by wishing the best to all of our readers — newcomers along with those who have been with us the whole time. While we thought, 20 years ago, that Linux was bound for success, few of us could have predicted the world we find ourselves in now. It is probably safe to say that Linux and free software will continue to go from strength to strength, and that 2017 will be another successful year. We look forward to being there with you.

Comments (7 posted)

Symbolic mathematics on Linux

This article is an introduction to the world of free and open-source applications for symbolic mathematics. These are programs that assist the researcher or student through their ability to manipulate mathematical expressions, rather than just make numerical calculations. I'll give an overview of two large computer algebra packages available for Linux, and a briefer sampling of some of the more specialized tools aimed at particular branches of mathematics.

This category of software is traditionally called a "computer algebra system", but that description can be misleading. These systems can find analytic solutions to algebraic and differential equations; solve integrals; sum infinite series; and generally carry out nearly any kind of mathematical manipulation that can be imagined. At the least, symbolic mathematics software can replace the bulky handbooks of mathematical information that have been lugged by generations of graduate students.

Over decades, mathematicians have honed these programs, encoding within them the accumulated mathematical knowledge of centuries: information about special functions, for example, that's so difficult (for some of us) to remember. They have learned to reduce such things as algebraic simplification and calculating derivatives to patterns of symbol manipulation ripe for automation. The earliest of these systems, developed in the 1960s, were based on Lisp, the obvious choice at the time, but development of later systems used a variety of languages.

Fortunately, most of the best of this software is free and open source, which allows us to look under the hood and examine or alter the algorithms employed.

Maxima

The ancestor of all symbolic mathematics systems is Macsyma [PDF]. It began as an academic project at MIT in the 1960s, but it was eventually licensed to Symbolics (a company founded by former MIT people), which sold it throughout the 1980s. A fork of an earlier version became Maxima, which remains the predominant free-software symbolic mathematics solution. Maxima is actively developed, and has attained a high degree of sophistication and completeness; source is licensed under the GNU GPLv2.

On Ubuntu, at least, the packaged version is quite close to the latest development version; if your distribution lags, you can build it from the sources in the Git repository on SourceForge. If you intend to do any real work with the program you should also install the "maxima-share" package (after installing Maxima itself). This will install a large number of extra mathematical libraries that are used transparently by Maxima and make it far more capable.

The traditional way to use Maxima is in a terminal. Just type "maxima", and you will be presented with an interactive command prompt. You can enter mathematical expressions in the customary computerized dialect (x^2 for x squared, etc.), followed by a semicolon, and Maxima will respond with a simplified or "solved" version of the expression. If Maxima can't do anything with the expression, it usually just repeats it: tell it "foo" and it will respond with "foo". Readline support allows you to recall previous inputs and edit them, which is particularly convenient for the type of exploration that Maxima seems to inspire, at least for this user.

But if you tell it:

    integrate(%e^(-x^2), x);

Maxima will respond with something that looks like:

    sqrt(%pi) erf(x)
    ----------------
           2

This example is here to illustrate three things: First, special numbers such as π and e are represented, in input and output, as %pi and %e. If you say, for example, %e, Maxima will just say %e back; to see the numerical value of the constant, say float(%e).

Second, mathematical expressions in the output are, by default, rendered in an ASCII approximation of mathematical notation, making their structure easier to grasp than the computerese that we are obliged to use for the input. You can enter tex(%) to get the result in a form ready for pasting directly into a TeX or LaTeX document. (The symbol % refers to the immediately preceding output; each input and output is numbered, and can be referred to directly, as %o6 for output number six, %i6 for input number six, etc.)

Third, there is a wealth of mathematical knowledge baked in to Maxima. Notice the erf in the numerator of the answer: this refers to the error function, well known to statisticians. Maxima will provide the results of integrals and the solutions of differential equations in terms of the special functions that it knows about, when appropriate.

If you'd like to know what that error function looks like, you just need to say:

    plot2d(%, [x, -2, 2]);

Remember, the % notation refers to the last result returned. The three terms in square brackets are the variable on the horizontal axis and its range. Maxima uses gnuplot for its plotting. If you're familiar with gnuplot, you can add options to the plot command to tweak the plot's appearance, or set global options that will apply to every plot in the session. If you have a high-resolution screen you may want to apply these global options to make the plots easier to see:

    set_plot_option(
        [gnuplot_preamble, "set termoption font 'courier,24';
         set termoption lw 4"]);

In fact, these options were used for the example plots in this article.

Maxima also provides an interface to gnuplot's 3D plotting commands. Here's a simple example of a surface plot:

    plot3d(sin(x)*cos(y), [x, 0, 2*%pi], [y, 0, 2*%pi]);

By setting the gnuplot preamble, either globally or per plot, you can access contour, parametric, or any of gnuplot's other plotting modes. The plots use the x11 gnuplot terminal. When you interact with them, you are interacting directly with the gnuplot subsystem, so you can use the mouse to zoom 2D and 3D plots and rotate 3D surfaces.

Using Maxima at the terminal is convenient because it starts instantly, is responsive, and there is nothing extra to install. However, the console interface has some disadvantages: the ASCII output is not easy on the eyes, especially after a long session; there is only one plot window, which gets reused for each plot; and there is no convenient record of your session.

There are a handful of alternative interfaces that solve one or more of these problems. None of them are ideal, nor as capable as the solution that we'll see in the next section; but I'll briefly describe them here, to provide an idea of what's available, and in case any one of them hits a sweet spot for the reader.

Two slightly more graphical interfaces to Maxima are Xmaxima and wxMaxima. The first of these is based on Tk and the second on wxWidgets. They both allow plots to be embedded with the input and output, and allow for various options to save the session, which is useful for creating notebooks or documents from Maxima explorations. Neither one seems to provide true typeset output, although wxMaxima can use the jsMath fonts to make the results somewhat more attractive. wxMaxima can serve as a gentler introduction to Maxima for those who prefer to get started without frequent trips to the manual, as it bristles with menus and dialogs that expose some, but far from all, of the program's options.

As we saw above, Maxima knows how to create TeX versions of its output. Therefore it should be possible to simply run TeX behind the scenes and display math that looks like real math. There are at least two interfaces that follow this strategy. The WYSIWYG editing platform TeXmacs can interface with Maxima and display typeset output, but this is probably mainly of interest to those who are already using TeXmacs. Perhaps of more general interest, especially to Emacs users, is the imaxima mode of that editor, where you can embed plots and fully typeset output directly into the editing buffer. A readline-like functionality is simulated by typing M-p instead of up-arrow.

Installation of imaxima can be an adventure. Several files need to be placed in the correct places, and variables set in the .emacs configuration file. In Ubuntu this can all be done automatically by installing the maxima-emacs package. The downside here is that this package will install large chunks of TeX Live. As the typical user of Maxima is likely to already have installed a more up-to-date version of TeX Live than most Linux distributions' package managers provide, this will create some redundancy — but it does save time. Some alternatives for those who already have TeX installed are to download the Maxima source, which includes the most critical required files, or to try to find recent versions of the imaxima mode files on the web. Either way, smooth operation is likely to require some customization in your .emacs file, some of which can be set through the Emacs user options interface, which exposes some imaxima settings; see the imaxima link above to get started.

The next figure shows part of a Maxima session in emacs using the imaxima interface. The regular plot commands can be used as in the terminal, and separate gnuplot windows will pop up. To get embedded graphs, use the wxplot2d() and wxplot3d() analogues. With the use of comments, for which Maxima uses C-style syntax, the Emacs buffer can become a notebook in the Jupyter style (in fact there is also a Maxima kernel for Jupyter). The notebook can be converted into HTML or LaTeX with the Emacs commands imaxima-to-html and imaxima-latex, respectively, making it a convenient way to generate lecture notes or parts of papers. The HTML export is serviceable out of the box, but the current state of LaTeX export fails to create correct graphics insertion commands, requiring some extra ad hoc post-processing steps.

The Emacs session depicted also illustrates a few additional Maxima features, such as the use of infinity, sums, and defining functions.

In the space available, I can't even scratch the surface of all the math that Maxima can do. It has grown by accretion over the decades, and continues to grow, as mathematicians take advantage of its extensible nature to teach it the secrets of their discipline's sundry specialties. Maxima is especially congenial to the Lisp programmer, who can drop into its Lisp subsystem at the interactive prompt and make contact with the internal representation of its mathematical expressions.

Sage

In 2006 a single developer, Ondrej Certik, started a project called SymPy, which is a symbolic mathematics program, like Maxima, written entirely in Python. It grew quickly, and is now a mature project with scores of contributors. SymPy can be used from a specially wrapped IPython, providing an experience similar to Maxima in the terminal, or as a library. It would deserve its own section in this article were it not easier to discuss it in the context of Sage.

Sage is an enormous system for symbolic and numerical mathematics. It is unique in presenting a unified interface to 90 distinct components. Behind the scenes, Sage will automatically use the appropriate component to perform the calculation or manipulation that the user desires; or, when there is more than one way to do something, Sage can be directed to use a particular library or algorithm.

Because of this, Sage can handle optimization problems, astronomical calculations, elliptic curves, number theory, interval arithmetic, networks, cryptography, statistics, Rubik's cubes, ray tracing, and much else that I don't understand. It can display 2D and interactive 3D graphics, including visualizations of molecular structures. It includes embedded versions of Python (with NumPy and many other Python libraries), R, SciPy, SymPy, Maxima, and many other subsystems. Sage even claims to be able to outsource computations to the Wolfram Alpha computational engine, but in my testing this did not work.

Naturally, Sage is a big download. The usual way to install it is to download a 1GB archive from headquarters and expand that to a 3GB directory tree. Start it by typing ./sage in a terminal from the SageMath directory. This approach is convenient, but may result in some redundant installations on your system. On Ubuntu, at least, there is also a PPA that will allow installation through its package manager.

Sage is based on Python rather than Lisp. It has two interfaces, each of which presents a version of Python with superpowers. After typing the sage command you will be faced with a wrapped IPython prompt, as when using SymPy. You can interact with Python normally, but you will find a few alterations in syntax; for example, ^ is used for exponentiation rather than **. You can perform symbolic math manipulations (and numerical calculations) from the prompt just as with Maxima, but with a somewhat different syntax (no semicolons required). There is no need to import any libraries, as everything has been set up within the special IPython environment. You can plot your results, but rather than gnuplot Sage uses matplotlib for 2D graphics and the Jmol Java molecular viewer to put 3D graphs in a window where you can interact with them.

Where Sage really shines is in its notebook interface. If you type notebook() at the IPython prompt, Sage starts a web server and connects it to a new tab in your browser. Here you can talk to Sage just as in the IPython interface, but the responses will be typeset by jsMath, which uses an embedded JavaScript version of the TeX mathematical typesetting algorithms. Thanks to AJAX and other JavaScript magic, interaction using the notebook is smooth; there are keyboard shortcuts for evaluation and for manipulating the "cells" into which the notebook is organized. Graphics are embedded into the page, and you can interact with 3D plots directly, zooming and spinning with the mouse. With a single click, the notebook can be converted into a neat HTML version suitable for printing or for use as a web page. You can even share the live notebook online.

The figure shows an extract from a Sage notebook in Firefox, while playing with the included graph theory packages. It gives some idea of the appearance, but cannot convey the fun of using Sage. This project has succeeded in gathering a carefully curated set of components and presenting them with a unified interface that is powerful and enjoyable to use.

Specialized packages

Maxima and Sage, powerful and wide-ranging as they are, are the general practitioners of the symbolic math world. Sometimes one needs to consult a specialist. Here I survey a few of the more narrowly focused mathematics systems, some of which are research projects, such as a system for symbolic computations in general relativity.

A related project is Cadabra, designed to help with the tensor manipulations and other math used in field theory. Cadabra is unusual in that the input language as well as the output are a subset of TeX. It's available as a reasonably up-to-date Ubuntu package.

While on the subject of physics, the FORM project is popular with particle theorists. Interaction is through text files that define the computation to be performed, and which can be processed in a multi-threaded, multi-processor, or sequential style.

Fermat, the personal project of Robert H. Lewis of Fordham University, specializes in polynomial and matrix algebra over the rational numbers and finite fields. It's the best in class for the specialized set of algebraic problems that it's designed for.

The CoCoA (Computations in Commutative Algebra) System specializes in polynomial systems and commutative algebra. It can employ Gröbner basis computations, unlike Fermat. It can also do map coloring and logic problems.

Macaulay2 concentrates on algebraic geometry and commutative algebra. It's been supported by the National Science Foundation since 1992. You communicate with Macaulay2 through a its own specialized, interpreted language.

TRIP specializes in perturbation series computations, specially adapted to celestial mechanics. In development since 1988, TRIP does both numerical and symbolic work, and is integrated with gnuplot.

I hope this incomplete roundup of a handful of specialized mathematics packages in this section provides a general impression of the range of activity in this field. For anyone with even a nascent interest in mathematics, systems like Sage, in particular, can make exploration of this world a great deal of fun.

Comments (29 posted)

A look at darktable 2.2.0

In what is becoming its annual tradition, the darktable project released a new stable version of its image-editing system at the end of December. The new 2.2 release incorporates several new photo-correction features of note, including automatic repair of distorted perspectives and the ability to reconstruct highlights that are washed out in some color channels but not all—a type of overexposure that other editors can miss. There is a new image-warping tool that lets users edit image pixels (a first for darktable, which has historically focused on image-wide tasks like color correction). And there is at least one new tool that may prove intriguing even to users who prefer editing images in some other program: a utility for inspecting and editing color-mapping look-up tables.

Source code bundles are available for download through the project's GitHub repository and binary packages are already available for a wide variety of popular Linux distributions. Users of the 2.0 series should note, however, that opening existing darktable edit files with the 2.2 release will automatically migrate them to the newer format and render them subsequently unopenable with darktable 2.0.

New perspectives

For many users, the most useful new addition in this release will be the automatic perspective-correction tool. Imagine having an image like the one below, where straight lines appear curved due to distortion:

Using an algorithm from the Windows-only free-software program ShiftN, this new tool automatically detects off-vertical and off-horizontal lines in an image and computes the transformation needed to create a perfectly aligned, perpendicular output image.

If the algorithm does erroneously mark lines that should not be used when transforming the photo, the user can simply deselect them before generating the correction.

The user can choose whether to apply vertical correction, horizontal correction, or a combination of both, and the parameters are adjustable (although only in truly pathological cases is the algorithm likely to need much adjustment). There are also parameters available for correcting rotation and for applying a lens shift if either of those functions is necessary to align or re-center the image.

Highlights in 2.2

Washed-out highlights in an image usually occur when the light hitting the photosensor meets or exceeds the maximum value that the sensor can measure; the pixel's value then gets clipped and detail is lost. But each pixel in a color image sensor consists of separate red, green, and blue monochrome sensors (at least, for the most common camera types), and it is possible that only one or two of those subpixels has been maxed out. Darktable deals with these cases in a different manner than the case where all three colors are washed out: it reconstructs luminance information from the non-overexposed color channels. That produces grayscale pixels that have at least some texture (as opposed to being flat fields of plain white). So, while the clipped region is still nearly white, it can still exhibit some gradation and reveal some shapes.

This option is the "reconstruct in LCh" mode of darktable's existing highlight-reconstruction tool but, in prior releases, it simply did not work—despite how nice it sounds in theory. In the 2.2 release, "reconstruct in LCh" has been rewritten from scratch and now works as advertised. Moreover, darktable now offers an on-canvas indication of which subpixels are clipped and which are not: a checkerboard pattern utilizing red, green, and blue checks in the washed-out image region.

This clipping information can be computed from the original raw camera file, even before standard transformations like white balancing are performed. When this indicator is activated, it enables the user to spot instances where it is possible to recover more image data from a washed-out image than would otherwise be possible. In the case where all subpixels are clipped, no detail can be recovered with the "reconstruct in LCh" mode. As was mentioned in our look at the 2.0 release, darktable's highlight-reconstruction tool also offers a way to fill in these entirely washed-out areas by copying colors from the surrounding pixels, but that option is something of a last resort. The now-working "reconstruct in LCh" mode is likely preferable.

Strictly speaking, of course, one should not take overexposed images in the first place (perhaps, instead, bracketing exposures if one is unsure of how to get the best shot), but "only take perfect pictures" is hardly practical advice. And overexposed pixels can result from image processing itself; any time the user applies exposure compensation in darktable or some other program, clipped highlights can be among the side effects.

Speaking of bracketing exposures, darktable 2.2 adds another tool designed to help with difficult-to-capture situations. In scenarios where the dynamic range of a scene is too wide to be captured in a single shot, the photographer can shoot multiple exposures (e.g., one to capture the highlights and one for the shadows). Those exposures can then be combined via darktable's new "exposure fusion" module. In essence, the two frames (or however many were taken) are stacked together, then the lighter portions of the darker image and the darker portions of the lighter image are blended to produce one image that looks more-or-less correct everywhere.

This is same the technique found in the Enfuse utility, and similar approaches are available in other "high dynamic range" (HDR) image-processing tools like Luminance HDR (both of which LWN looked at briefly in 2012). Sadly, neither Enfuse nor Luminance HDR has made much progress toward a new stable release since 2012. So the addition of the exposure fusion module in darktable is a welcome sight. Perhaps that is the better option in the long run, since using a single-purpose application like Enfuse makes for a more complicated workflow than most users want. But all of the usual caveats about HDR images apply to darktable's new module: it is easy to overdo the blending process, creating distracting artifacts like halos or producing weird, washed-out looking results. Proceed with caution.

Distort all the things

A key distinction between photo-manipulation programs like darktable and RawTherapee and raster image editors like GIMP is that the former are intentionally restricted to "post-production" image-tuning operations. That means they focus on making adjustments to the hue, saturation, and lightness of the pixels in the image as a whole (or in large portions of the whole), as opposed to painting or drawing with tools onto a canvas. Various names are trotted out for such applications: "raw photo editor" was one of the first; "photo workflow application" is more recent, although neither really seems to communicate the idea unambiguously.

On the plus side, "workflow" adjustments can easily be implemented as non-destructive transformations, and the operations used on any particular image can be saved in a compact file format. The downside is that, in real life, one eventually runs into some situations where on-canvas editing is required.

The darktable 2.2 release adds what could reasonably be considered the application's first on-canvas tool for image manipulation (with possible exceptions going to special cases like flipping and rotating). It is called Liquify and it lets the user deform the image by pinching, stretching, and warping pixels.

The name "Liquify" is seemingly borrowed from Adobe Photoshop, though similar functionality can also be found in GIMP's IWarp filter.

Darktable lets the user add a single-point warp node that will stretch, compress, and twist the surrounding pixels within a user-defined radius, or the user can connect several such nodes along a vector path and warp a larger portion of the image. To be sure, the Liquify tool will not suffice for every possible operation, but it is an important first step. If bending or reshaping one portion of a photograph is all that is needed, doing so in darktable is more convenient that exporting the image to a separate raster file and opening it with GIMP.

Darktable is looking up

Whereas the Liquify tool manipulates pixel positions in space, the vast majority of darktable's other functions manipulate pixel colors in some specified fashion: adjusting the white point, brightening the dark regions, shifting certain hues, and so on. The final new tool in the 2.2 release is another color-adjustment feature, but it is a feature that can best be thought of as a generic color-mapping function. The color look-up table (CLUT) tool supports remapping all of the colors in the image based on some predefined set of transformations.

Remappings of this type are how "film emulation" features work: when someone takes the time to measure the output of known sample images exposed on Kodachrome 64, Fuji Velvia 50, or some other film stock, they can produce a mapping that transforms a generic RGB image into something that looks more vintage. Photographer and GIMP team member Pat David, for example, has created several hundred open-source film-emulation CLUTs that are already usable in GIMP's G'MIC plugin and in RawTherapee. But there are other important CLUTs found in the wild, such as those that digital cameras apply when converting a raw image into a JPEG to be saved on the memory card. Being able to apply the same CLUT to a raw image in darktable would ensure that its output matches the camera's JPEG exactly.

This is indeed what the new tool in darktable can do—at least, if the CLUT of interest is available. At present, the process of creating those CLUTs is only beginning, with a handful of mappings available.

Creating a new CLUT can be a time-intensive task. Color spaces are three-dimensional constructs—regardless of whether those dimensions are red-green-blue, hue-saturation-lightness, or something more exotic; a 3D-to-3D transformation matrix thus involves looking at a lot of sample points. But the new darktable release adds a utility called darktable-chart that can assist with the process. To match a camera's output, the user will need to purchase a calibrated IT8 target and take a reference photo, but nothing stops interested parties from fiddling around to create interesting CLUT mappings of their own. In fact, the popularity of products like Instagram suggests that it is hard to stop people from fiddling with CLUT transformations.

In the long run, CLUT support will let darktable users easily apply a large variety of color styles to their photographs, whether those styles are designed to emulate film or not. But darktable-chart may prove useful in its own right, because—as David has mentioned on his blog—the existing process for creating a CLUT is complex.

From the shadows

While the aforementioned tools are the biggest changes in the new release, there are scores of smaller improvements to be found as well. The keyboard support introduced in version 2.0 is expanded upon: arrow keys can be used to adjust sliders and controls, and modifier keys can be used to change the rates of adjustment (Shift is a 10x increase in the adjustment increment, while Control is a 0.1x reduction in the increment).

The script-friendly command-line version of darktable, darktable-cli, can now take entire directories as an input argument. Watermarks generated by darktable can now include geolocation information. And, on Linux, the application now tells the window manager when it has completed a batch operation, thus enabling the user to be notified when another application has focus.

There is also initial support for undo and redo on image-adjustment operations. That may seem like a fundamental missing piece, but one must remember that darktable's adjustments are non-destructive and most can only be applied once (e.g., you would not adjust the white balance of a photograph twice). So in most cases, undoing or changing an adjustment was a simple matter of deactivating it or resetting it to the default parameters. Still, this is a nice improvement for how most people think of editing images.

At a lower level, the project has implemented initial support for some new photosensor subpixel patterns, including CYGM and RGBE arrays. OpenCL support has also been improved, with OpenCL now used to demosaic raw images, and the first builds for ARM devices are available. Finally, the program now uses Pango for text rendering, which will enable it to be used with right-to-left languages.

Interfacing

The new feature set boasts a lot of useful additions. But, if there is any criticism to be leveled at the 2.2 release, it is that darktable still hides so much of its functionality behind user-interface choices that are invisible while simply using the program.

A good example is the substantial set of options in the perspective-correction tool that are only made available by holding down modifier keys when clicking on the "automatic fit" buttons. The existence of the modifiers is only discoverable by hovering the mouse over the buttons long enough to bring up the tooltip hints; for an application offering as many functions as darktable does, it is simply not feasible to expect users to memorize every combination of Shift, Control, and left- and right-clicking for every tool.

Other applications—for example, GIMP—find a way to present such options in a persistent message-bar area or in a field within the toolbox itself; there is no reason darktable cannot do the same. Instead, having to pause and hover the mouse over every tool before you can see how to use it is reminiscent of darktable's early days, when the filters and controls had no text labels and users had to guess at the meaning of cryptic icons. The application has come a long way since then; it would seem that it still has some distance to travel.

Comments (5 posted)

Fuzzing open source

Fuzz testing finds bugs, so it stands to reason that continuous fuzz testing will find more bugs and find them sooner. That is part of the premise behind the OSS-Fuzz program announced by Google on December 1. Since many of the bugs found by fuzzing have security implications, discovering more earlier can only be a good thing for the security of our systems.

OSS-Fuzz is meant to apply fuzzing power to free and open-source software projects, especially those that are part of the critical internet infrastructure. As might be guessed based on that, the Core Infrastructure Initiative has worked with Google to develop OSS-Fuzz. The underlying technology used comes partly from the ClusterFuzz project that has been successfully employed to find a large number of bugs in the Chrome browser.

The basic idea is to continuously fuzz test the latest source version of the projects that have signed up (and been approved) for the OSS-Fuzz beta test. Initially, OSS-Fuzz will be using the libFuzzer coverage-guided fuzzing library in conjunction with AddressSanitizer (ASan) to try to find various types of memory misuse that could be exploited by attackers. That process is parallelized across thousands of virtual machines such that some four trillion test cases are run per week (or were at the time of the announcement, that number may have grown since then).

The announcement describes an early success story for the project:

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa
READ of size 2 at 0x615000000ffa thread T0
SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)
   #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31

Many bugs have been found, 150 at the time of the announcement, though there are 200 entries (with 177 verified) in the bug tracker as of this writing. Roughly half of those are marked as security bugs. All of the bugs found will be disclosed within 90 days of their discovery in keeping with Google's own disclosure policy.

ClusterFuzz does more than just parallelize the fuzzing process, it manages test cases to both whittle them down to something small that still reproduces the problem and to continue to run them to detect when the problem is fixed and, after that, whether regressions bring it back. It also tries to determine which change (or set of changes) introduced a problem by doing a bisection.

Projects interested in joining the OSS-Fuzz effort will need to add some fuzz targets to their code. These targets are functions that accept a byte array from the fuzzing engine and use that input in an "interesting" way using the project's API. These fuzz targets are not libFuzzer-specific and can be used by other fuzzers (there is talk of adding support for american fuzzy lop, for example). Those fuzz targets must be integrated with the build and test system for the project.

After that, a corpus of both good and bad inputs for the fuzz target should be created. That gives the fuzzing engine a starting point that has the proper structure for the input data (and the bad inputs will help show ways to "break" the format), which eliminates a whole bunch of wasted effort on inputs that won't get past the first tests in the code. In coverage-guide fuzzing, the binary is instrumented to provide information on the code that a given input has caused to be executed. Changing the input data to cause new paths through the code to be taken is the underlying mechanism for coverage-guided fuzzing.

To set up a new project, a directory needs to be created under projects in a GitHub clone of the oss-fuzz Git repository. It needs a Dockerfile to describe the container environment for building the project and its fuzz targets, a build script (build.sh) that will run in the container to generate a build of the project, and a configuration file (project.yaml) with some project metadata. A pull request can be made to the oss-fuzz project and if it is accepted, the project will be tossed into the ClusterFuzz hopper.

The FAQ lists two main criteria for a project's inclusion in the beta: exposure to remote input and the number of dependent users (people and projects). Right now, the goal is to add "established projects that have a critical impact on infrastructure and user security", though expanding the reach of OSS-Fuzz is in the plans. The current project list has over 50 entries that reads a bit like a "who's who" of open-source projects that fit the criteria including libreoffice, curl, libarchive, pcre2, libpng, openssl, postgresql, sqlite3, tor, strongswan, and so on. There is also a build status page where the most recent build log for each project can be accessed.

Fuzzing takes a lot of resources, but it is an inherently parallelizable process, so it is a perfect match for Google and others with enormous clusters of computers. Though it has taken some time, the security-testing story for open-source projects has certainly gotten better over the years. The lessons of Heartbleed (and, seemingly to a lesser extent, some of the larger vulnerabilities of yesteryear) have not gone completely unheeded. Beyond that, it is good to see other projects that are applying fuzzing to the kernel. Fuzzing is no panacea, but it can certainly help find the next "zero day" before it actually becomes one.

Comments (1 posted)

Security quotes of the fortnight

Comments (13 posted)

How to find Android apps that respect user privacy (opensource.com)

Comments (none posted)

The Year Encryption Won (Wired)

Comments (35 posted)

Bottomley: TPM2 and Linux

Comments (1 posted)

bash: code execution

Comments (none posted)

borgbackup: two vulnerabilities

Comments (none posted)

botan: integer overflow

Comments (none posted)

chicken: two vulnerabilities

Comments (none posted)

curl: information leak

Comments (none posted)

curl: buffer overflow

Comments (none posted)

cxf: two vulnerabilities

Comments (none posted)

cyassl: multiple vulnerabilities

Comments (none posted)

dovecot: denial of service

Comments (none posted)

exim4: information leak

Comments (none posted)

firejail-lts: denial of service

Comments (none posted)

gdk-pixbuf2: unspecified

Comments (none posted)

graphicsmagick: denial of service

Comments (none posted)

gstreamer-plugins-good: code execution

Comments (none posted)

httpd: three vulnerabilities

Comments (none posted)

imagemagick: code execution

Comments (none posted)

imagemagick: code execution

Comments (none posted)

irc-otr: information disclosure

Comments (none posted)

js-jquery: cross-site scripting

Comments (none posted)

kernel: denial of service

Comments (none posted)

kernel: three vulnerabilities

Comments (none posted)

libarchive: denial of service

Comments (none posted)

libcrypto++: denial of service

Comments (none posted)

libphp-phpmailer: code execution

Comments (none posted)

libpng: NULL dereference bug

Comments (none posted)

libvncserver: two vulnerabilities

Comments (none posted)

mariadb: multiple unspecified vulnerabilities

Comments (none posted)

msgpuck: two denial of service flaws

Comments (none posted)

nagios-plugins: multiple vulnerabilities

ldap_bind: Can't contact LDAP server (-1)
	additional info: TLS error -8172:Unknown code ___f 20
Could not bind to the LDAP server

Comments (none posted)

openfire: multiple vulnerabilities

Comments (none posted)

openjpeg2: multiple vulnerabilities

Comments (none posted)

openssh: multiple vulnerabilities

Comments (none posted)

pcsclite: privilege escalation

Comments (none posted)

php-zendframework-zend-mail: parameter injection

Comments (none posted)

pillow: heap-based buffer overflow

Comments (none posted)

postgresql-common: file overwrites

Comments (none posted)

python-crypto: denial of service

Comments (none posted)

python-wikitcms: code execution

Comments (none posted)

qemu: denial of service

Comments (none posted)

smack: TLS bypass

Comments (none posted)

spip: two vulnerabilities

Comments (none posted)

springframework: directory traversal

Comments (none posted)

squid: two vulnerabilities

Comments (none posted)

tracker: adding sandboxing

Comments (none posted)

xen: two vulnerabilities

Comments (none posted)

xen: denial of service

Comments (none posted)

xen: denial of service

Comments (none posted)

Kernel release status

Stable updates: none have been released since December 15. The 4.9.1, 4.8.16 and 4.4.40 stable updates are in the review process as of this writing; they can be expected on or after January 6.

Comments (none posted)

Finishing out the 4.10 merge window

That list includes:

• The PA-RISC architecture has gained support for kernel address-space layout randomization.

• The cache-allocation technology patch set has been merged. These patches provide access to a new mechanism in Intel processors by which the processor's memory cache can be partitioned between processes. It can be used to keep one group of processes (a container, say) from dominating the cache, or to set aside a portion of the cache for a set of privileged processes.

• There are new drivers for QLogic QEDI 25/40/100Gb iSCSI initiators and Loongson1 SoC hardware watchdogs.

• The cycle_t type used for clock values inside the kernel has been removed; a plain u64 type is now used instead.

The 4.10 stabilization period got off to a slow start due to the holidays; only 27 non-merge changesets were applied between 4.10-rc1 and 4.10-rc2. The pace of change can be expected to pick up, though, as developers return to work and the final 4.10 release date (probably February 12 or 19) approaches.

Comments (10 posted)

Tracking functional dependencies between devices

Some device dependencies are inherent in the architecture of the system. For example, a USB peripheral will not be usable if the appropriate USB host adapter is unavailable, and that adapter is probably connected to some other system bus that must also be up and running. Dependencies based on the connection topology of the system are relatively easily represented in a tree structure; that is what the kernel's device model was created to do. Using this model, the kernel can, for example, suspend devices in the system in the correct order, keeping intermediate devices operational until all the devices that depend on them have been shut down.

In a modern system, though, the dependency graph can be rather more complicated. A camera "device", for example, is likely to be a set of interconnected devices that look independent to the kernel. Actually operating the camera requires a sensor device, which is probably controlled via a connection to an I2C bus; it probably also depends on a couple of GPIO devices for its power and reset lines. The sensor is connected to a separate bridge device that acquires the image data; that bridge may need a DMA controller to move that data into memory. There may be other devices for various hardware-implemented image transformations (rotation or color-space conversion, for example) in the mix as well.

The point is that each of these components looks like a separate device to the kernel. These devices are on separate controllers and, perhaps, on separate buses; they do not appear to be related from a look at the topology of the system. For the most part, a top-level driver marshals these devices together and makes them function together; the information it needs to do this task is, in current systems, often found in the device tree structure. But the kernel's driver core can break things if it shuts down one of the subdevices because it doesn't understand that other devices depend on that subdevice.

Drivers have tended to work around this problem with one-off device-specific code. As one might expect, that leads to a fair amount of code duplication and a lot of inadequate solutions. It would be better to have a single solution in the driver core code that works for all devices. Moving toward that solution is the objective of the functional dependencies infrastructure merged for the 4.10 kernel.

The interface to this mechanism is relatively simple, consisting of a single function to indicate that a dependency exists:

    struct device_link *device_link_add(struct device *consumer,
				    	struct device *supplier,
					u32 flags);

This call informs the driver core that the consumer device depends on the supplier device. So, for example, the system will not suspend supplier unless consumer is already suspended, and it will not probe or resume consumer until supplier is up and functional. Additionally, if the supplier device is unbound, the consumer device will, by virtue of no longer being able to function anyway, be unbound automatically.

By default, device links are persistent and will remain in place for as long as the system is running. If, however, the DL_FLAG_AUTOREMOVE flag is provided when the link is created, that link will be automatically removed if the driver of the consumer device is unbound. These non-persistent links can be useful in situations where the hardware can be configured in multiple ways, creating varying dependencies over time. The DL_FLAG_STATELESS flag can be used to create a link that is used for suspend/resume ordering, but which is not otherwise managed by the driver core.

If there is a need to explicitly remove a device link, that can be done with a call to device_link_del():

    void device_link_del(struct device_link *link);

As of 4.10-rc2, there is only one user of this new infrastructure (the Exynos I/O memory-management unit code) in the mainline kernel. There will certainly be others that will show up in future development cycles, though. With luck, they will be accompanied by a reduction in driver-specific dependency-handling code and an improvement in kernel quality overall.

Comments (1 posted)

Context information in memory-allocation requests

Memory-allocation complexity in the kernel is driven by constraints on what the kernel can do in any given situation. It is often the case, for example, that the kernel is running in a context where it is not allowed to block waiting for an event, so allocation requests must be satisfied without acquiring any sleeping locks. Sometimes a request should be given access to the last-resort pools of memory; this is usually the case when the request itself is part of the process of freeing more memory in a system where memory is tight. There can be constraints on where the allocated memory must be located. And so on.

The approach taken to track these constraints is to add a "GFP flags" argument to every memory-allocation function. So, for example, the prototype of kmalloc(), used to allocate relatively small chunks of memory, is:

    void *kmalloc(size_t size, gfp_t flags);

The flags argument describes the constraints on the request. A value of GFP_ATOMIC indicates that the request is running in atomic context and cannot sleep, for example, while GFP_DMA32 says that the allocated memory must be placed in a physical location reachable by devices limited to 32-bit DMA operations. There is a long list of these flags; <linux/gfp.h> has the whole set.

Two types of flags

The point of interest here is that some of these flags (like GFP_DMA32) describe attributes of the needed memory — they apply to a specific allocation request. But others, like GFP_ATOMIC, instead describe the context in which the allocation request is being made. This context is often not known at the point where the allocation function is called, since that often happens in low-level code that can be invoked in many contexts. So higher-level code must inform the low-level code about the context in which it is running; this is generally done by adding GFP-flags arguments to functions all the way up the call chain. To pick a random example, consider the function that submits a request to a USB device:

    int usb_submit_urb(struct urb *urb, gfp_t mem_flags);

This relatively high-level function must track the given mem_flags and pass them to any function it calls that might allocate memory; it must also adjust the flags if its own context changes. This interface has been made to work for many years, but it is somewhat prone to errors. One could argue, as some have over the years, that it is fundamentally wrong; information tracking the context in which a thread is running might be better attached to the thread directly rather than dragged along through a chain of function calls.

GFP_NOFS

One flag in particular that describes the calling context is GFP_NOFS, which instructs the memory allocator to avoid calling into any filesystem code. In particular, that means that the allocator cannot start writeback on dirty pages to make more memory available. There are (at least) a couple of reasons to impose this constraint. One is that the allocation call itself is coming from filesystem code; in that case, calling back into the filesystem risks deadlocking the system. The other is that adding filesystem calls to a lengthy call chain could overflow the kernel stack, an outcome cherished by attackers but otherwise unloved by Linux users.

Given those possibilities, it is unsurprising that kernel developers have tended to take a "better safe than sorry" approach to the GFP_NOFS flag; as a result, that flag appears in a great many allocation calls — a quick grep shows over 1,300 instances in the 4.10-rc2 kernel. At the Linux Storage, Filesystem, and Memory-Management summit in April 2016, Michal Hocko called out use of this flag as a problem. It appears in many places where it is not really needed, unnecessarily constraining what the memory-management code can do and, as a result, worsening system performance. He suggested that this flag should be phased out in favor of a flag in the task structure that could be used to accurately track the allocation context.

More recently, he has proposed a new API that implements these changes. A new flag (PF_MEMALLOC_NOFS) is defined for the flags field of the task_struct structure. Then, whenever a thread enters a context where filesystem calls should not be made, it should call:

    unsigned int memalloc_nofs_save(void);

This call will set the PF_MEMALLOC_NOFS flag and pass the previous flags value back as its return value. Exiting from the "no filesystem calls" context is done with a call to:

    void memalloc_nofs_restore(unsigned int flags);

The flags value passed in should be the value returned from memalloc_nofs_save().

Between the two above calls, all memory-allocation requests executed in the current thread will behave as if the GFP_NOFS flag had been passed, regardless of whether it is actually present or not. Since each caller saves the previous context, these calls can be nested to any level and the right thing will happen. For now, the GFP_NOFS flag remains (there is the matter of those 1,300 users, after all), but one can see its eventual removal in the cards. The patch set begins that process by fixing some callers in the XFS and ext4 filesystem code. The resulting code should be clearer, and it eliminates the chance of a stray allocation calling into the filesystem code in the wrong place.

Developers familiar with the memory-management code may think that this interface looks familiar. Indeed, it is inspired by a set of already existing functions:

    unsigned int memalloc_noio_save(void);
    void memalloc_noio_restore(unsigned int flags);

These functions were added to the 3.9 kernel in 2013 by Ming Lei; they move the GFP_NOIO flag (which inhibits the initiation of I/O operations) into the task structure in the same way.

The memory-allocation interface is, thus, clearly evolving in a direction where context-related information is stored with the rest of the thread's context rather than being passed through function arguments. This evolution can only be described as a slow process, though; there are nine memalloc_noio_save() calls in the 4.10-rc2 kernel, compared to nearly 500 uses of GFP_NOIO. Increasing the pace of change may be hard, though; switching to the new API requires a fairly deep understanding of the code involved and cannot be done with a simple script.

One could imagine taking this work further by, for example, tracking atomic context explicitly. But that is work for the future; completing the task for GFP_NOIO and GFP_NOFS should arguably be done first. Once all that is done, the kernel's memory-allocation API may better match the uses to which it is put. Given that we have only had 25 years to work on it so far, it is not entirely surprising that we have not gotten there yet.

Comments (12 posted)

Moving on from net-tools

The package in question is net-tools, the home for many familiar network-configuration utilities. If you are accustomed to using commands like ifconfig, arp, netstat, or route to make network changes, you are a net-tools user. Many of these tools have a long pedigree, at least in spirit, having originally been written before the first Linux kernel. Anybody who has been administering Unix-like systems for any period of time will certainly have learned how to use the net-tools utilities to get things done.

The only problem is that net-tools is considered obsolete; indeed, it has been so considered since early this century. The modern replacement is iproute2, which is actively developed and, unlike net-tools, has support for all of the kernel networking stack's fancier features. In theory, we all should have transitioned over to iproute2 at least ten years ago.

In practice, many of us have not done that. Indeed, "us" in this case includes distributors who still use net-tools utilities in a number of packages. Surprisingly, the net-tools developers, who have not made a new release since 2011, have recently resumed working on those utilities; even more surprisingly, they made output changes, apparently breaking scripts that parse that output. These changes led Debian developer Marco d'Itri to call for the project to "kill" net-tools, and, in particular, to stop depending on it in other packages. He would also like to see the project stop installing net-tools by default.

The list of Debian packages depending on net-tools is not that short, but it would appear to be a manageable list if the project were to decide to fix all of those dependencies. The net-tools maintainer is not opposed to the idea of deprecating and eventually phasing out the package; indeed, he tried to do so in 2009. It seems to be generally agreed that the configuration scripts shipped with Debian packages should use iproute2 rather than net-tools to ensure that the examples seen by administrators are using the current tool set. So there would appear to be little controversy around the idea of phasing out net-tools usage and dropping the priority of the package for installations.

There is less consensus around removing the package entirely, or even just dropping it from the default install. There are, it seems, quite a few users who have ifconfig and netstat burned into their muscle memory; those tools work fine for most use cases, so users don't see a strong reason to shift to iproute2. As Ted Ts'o put it:

The fact that the iproute2 commands are seen as more verbose and, by some at least, as having less readable (and less machine-parseable) output doesn't help either. So it's not surprising that various participants expressed their preference for the net-tools utilities, but Russell Stuart suggested that it might be time to move on:

(It is worth noting that the "unnecessary" part is not universally accepted; it's not clear that net-tools could have been evolved to handle modern networking configuration without incompatible changes.)

Debian, as it happens, is having this discussion a bit late. OpenSUSE discussed removing net-tools in 2009, but has not done so. Red Hat and Fedora got serious in 2011, and the RHEL 7 release no longer installs net-tools by default. The fact that this change is not universally popular shows how reluctant users can be to let go of their long-used tools.

None of these distributors have removed the net-tools package, and none are likely to as long as supporting them is relatively easy and users depend on them. But, when they do break, or when they fail to support new networking features that users need, there is not likely to be a lot of interest in fixing them. Distributors have to choose where to expend their energy, and there will come a point where dragging along obsoleted tools that the old folks want falls off the list.

For now, users who are accustomed to typing commands like ifconfig are probably safe; at worst, they will need to install the net-tools package explicitly. But anybody who is using these commands in scripts should probably have updated those scripts some time ago. There will come a point where those scripts break; it seems that could even happen as a result of attempts to restart development on net-tools, rather than by explicit deprecation. This cheat sheet is likely to prove helpful for anybody wanting to make the transition to the new tools.

Software transitions like this are invariably an unwanted distraction for users who are uninterested in whatever new features are available and would prefer that their systems (and their habits) just continue to work. But the world we live in does not stand still, so such transitions are simply going to happen, and distributors will find themselves caught in the middle. As those distributors strive to keep everybody happy, we should not be surprised to see more of these transitions take a decade or more.

Comments (90 posted)

Distribution quotes of the week

Comments (3 posted)

Alpine 3.5.0 released

Comments (none posted)

The end of CyanogenMod

Comments (13 posted)

Announcing FreeDOS 1.2

Comments (1 posted)

OpenELEC 7.0 released

Comments (none posted)

Talks between OpenWrt and LEDE

Full Story (comments: 10)

Release for CentOS 7.3.1611 on ARM64/AArch64

Full Story (comments: none)

Bits from the DPL -- 2016Q4

Full Story (comments: none)

openSUSE Board elections 2016/2017 - Phase 1

Full Story (comments: none)

Removing 32-bit powerpc architecture from future Ubuntu releases

Full Story (comments: none)

FreeBSD 9.3, 10.1 and 10.2 EoL

Comments (none posted)

Distribution newsletters

• DistroWatch Weekly, Issue 693 (January 2)
• Lunar Linux weekly news (December 23)
• Lunar Linux weekly news (December 30)
• This Year In Solus (2016)
• Ubuntu Weekly Newsletter (January 1)

Comments (none posted)

LXLE: A Linux Distribution Light on Resources But Heavy on Function (Linux.com)

Comments (none posted)

NethServer: Linux without All That Linux Stuff (Linux Journal)

Comments (none posted)

New features in Python 3.6

The Python 3.6 release occurred on December 23, only one week later than planned all the way back in October 2015. Python 3.6 adds a number of new features, including more support for asynchronous operations (generators and comprehensions), a filesystem path protocol, a new literal string formatting option, two random-number-related features, a frame evaluation API for debuggers and just-in-time (JIT) compilation, and more. Some of these features have been described in LWN articles along the way, but many haven't, so an overview of the highlights of the new release is in order.

As always, the release schedule (PEP 494) and, especially, the "What's New In Python 3.6" web pages provide an excellent overview of the release as well.

String formatting

Two string-formatting changes made their way into 3.6. The first adds yet another way to format text strings in Python using f-strings (which are described in PEP 498). This creates a new literal string type in the language (f'string') that can be used to directly interpolate variables (and full expressions) into strings:

    >>> num = 42
    >>> f'November 20{num}'
    November 2042
    >>> f'{num} * {num} = {num*num}'
    42 * 42 = 1764

In addition, PEP 515 was adopted, which extends the language's syntax to allow underscores into numeric literals for readability purposes:

    a = 1_000_000
    b = 0x_abcd_ef09

A path protocol

The addition of a filesystem path protocol (PEP 519) will allow easier handling of path names in the language and standard library. The pathlib module was meant to help solve the problem of different representations for path names when it was added (on a provisional basis) for Python 3.4. But it suffered from a lack of widespread adoption, even by other parts of the language (e.g. open()), at least partly because it was provisional. Some discussions in the early part of this year led to the development of the PEP and the removal of pathlib's provisional status.

All of that paved the way for open() and the standard library to start accepting pathlib.Path objects. The protocol allows objects to declare their path-like nature by implementing the __fspath__() method. The os.PathLike() function can be consulted to determine if something is path-like and there are explicit operations in the os module to get either str or bytes representations of paths. The "What's New" page fills in some details as well:

Frame evaluation

JITs were a hot topic at the 2016 Python Language Summit; one of the presentations on the Pyjion project introduced a plan to add an API to allow alternative ways to evaluate Python code (i.e. a frame-evaluation API). That idea has come to fruition with the adoption of PEP 523. It adds hooks to the CPython interpreter (the reference Python implementation) that will allow JIT engines as well as debugging tools to intercept the evaluation of frames. That will allow other programs to provide alternate ways to evaluate and execute the code in the frame objects.

Asynchronicity

Two PEPs that extend Python's asynchronous feature set have been adopted for 3.6. They both extend the async/await concept into more contexts. The addition of the async and await keywords in Python 3.5 furthered the move toward better support for asynchronous programming into the language itself. That largely started with the provisional adoption of the asyncio module back in 3.4 (see our coverage of Guido van Rossum's 2013 PyCon keynote for some background).

The asynchronous generators feature (as described in PEP 525) is effectively just a shortcut, but also provides improved performance—roughly twice as fast. Currently, in order to have an asynchronous generator that can be used in async for statements, a class must be defined that implements the __aiter__() and __anext__() asynchronous protocol. An alternative might have been to allow yield statements inside functions defined with async def, but when support for async/await was added, the yield statement was not allowed for async def coroutines. Changing the language to accept that made asynchronous generators easier to write and roughly twice as fast. The PEP gives an example of how that might be used (and compares it to how it would currently need to be written):

    async def ticker(delay, to):
	"""Yield numbers from 0 to `to` every `delay` seconds."""
	for i in range(to):
	    yield i
	    await asyncio.sleep(delay)

PEP 530 allows async and await to be available to comprehensions (i.e. list, dict, and set comprehensions as well as generator expressions) so that async can be used in coroutines defined with async def. The PEP gives an example of an asynchronous list comprehension that uses an asynchronous generator (agen()):

    [ i async for i in agen() ]

In addition, await can be used in comprehensions in coroutines to wait for the completion of a asyncio.Future object. One of the dict examples given in the PEP is shown below, but more intricate comprehensions are possible:

    result = { fun : await fun() for fun in funcs }

Random numbers

As was described in an article back in July, Python will return to having a blocking os.urandom() call. That change was added to Python 3.5, but had an unexpected side effect: Python initialization could block waiting for enough entropy to initialize its seed for dictionary randomization (to avoid hash collision attacks). In 3.5.2, Python stopped using the blocking call for its own initialization, but also partly reverted the change to os.urandom() to give users some time to adjust. For Python 3.6 and beyond, though, the call will block until there is sufficient entropy (i.e. will call getrandom() without the GRND_NONBLOCK flag) in the kernel's entropy pool.

Another change that came about as the project looked at how Python generates random numbers is the advent of the secrets module, which has been specified to provide crypto-strength random numbers. The default random number generator provided by the random module is documented to be unsuited for cryptographic purposes, but was seen as an "attractive nuisance" (as PEP 506 called it) that would draw in new programmers based on code snippets on various web sites. It is hoped that adding secrets will provide users with an obvious standard library alternative that does not suffer from the shortcomings of random.

Variable annotations

Type annotations (or type hints) have been fast-tracked into Python over the last two years or so. The first support for the optional annotations was added to Python 3.5 (PEP 484), but that did not address annotations for variables (just function arguments and return values). PEP 484 suggested using comments as a stopgap, but it was clear that variable annotation would eventually come (see, for instance, Guido van Rossum's 2015 Python Language Summit talk on type hints).

As Van Rossum predicted in that talk, variable annotations will make an appearance in Python 3.6. The optional annotation will be allowed in variable initializations or just as a type declaration. From PEP 526:

    some_number: int           # variable without initial value
    some_list: List[int] = []  # variable with initial value

As with the earlier annotations, there are no language semantics associated with the annotations, but they will be placed into the __annotations__ ordered dictionary for the class, module, or function where they occur. They can also be retrieved using the typing.get_type_hints() API. It is important to note (and the PEP emphasizes) that: "Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention."

Compact dictionaries

The internal representation of the dict type has been changed to use far less memory (20-25% less compared to Python 3.5). This is a feature that came from the PyPy project. As a side effect of the new representation, dict now preserves the order in which items were added. This side effect should not be relied upon going forward, but choosing collections.OrderedDict when preserving the order is needed will benefit from the new dict implementation.

And more

There are more features listed for the release, including the removal of the provisional status for the asyncio module, updating the SSL modules to support OpenSSL 1.1.0, adding a PYTHONMALLOC environment variable for debugging memory allocation issues, and more. All in all, it is a solid release that brings some useful facilities, as well as rounding out some features that have appeared in earlier releases. It is far too early to guess what might appear in Python 3.7, but it is currently scheduled for a mid-2018 release.

Comments (9 posted)

Development quote of the week

Comments (none posted)

darktable 2.2.0 released

Comments (2 posted)

Inkscape 0.92 released

Comments (3 posted)

LedgerSMB 1.5.0 released

Full Story (comments: none)

Python 3.6.0 released

Full Story (comments: none)

sed-4.3 released [stable]

Full Story (comments: 2)

Development newsletters

• Emacs News (December 26)
• Emacs News (January 2)
• What's cooking in git.git (December 21)
• What's cooking in git.git (December 22)
• What's cooking in git.git (December 27)
• Koha Community Newsletter (December)
• OCaml Weekly News (December 27)
• OCaml Weekly News (January 3)
• Perl Weekly (December 25)
• Perl Weekly (January 2)
• PostgreSQL Weekly News (December 25)
• PostgreSQL Weekly News (January 1)
• Project Gutenberg Newsletter (October-December)
• Python Weekly (December 22)
• Ruby Weekly (December 22)
• This Week in Rust (December 27)
• This Week in Rust (January 3)

Comments (none posted)

Grumpy: Go running Python

Comments (47 posted)

Ringing in 2017 with 90 hacker-friendly single board computers (HackerBoards)

Comments (25 posted)

Larsson: A stable base for Flatpak: 0.8

Eulogy for Pieter Hintjens

Comments (3 posted)

Free Software Supporter Issue 105, January 2017

Full Story (comments: none)

EFF: The Patent Troll Abides

Comments (2 posted)

7 notable legal developments in open source in 2016 (opensource.com)

Comments (none posted)

FOSSASIA'17 Kernel Track: Call for Speakers

Full Story (comments: none)

CFP Deadlines: January 5, 2017 to March 6, 2017

If the CFP deadline for your event does not appear here, please tell us about it.

LibrePlanet 2017 keynote announcement

Comments (none posted)

Events: January 5, 2017 to March 6, 2017

If your event does not appear here, please tell us about it.