Git for design projects

At Libre Graphics Meeting 2016 in London, Julien Deswaef presented a talk exploring how Git can aid graphic designers and other artists in their daily processes, as well as pinpointing Git's shortcomings for that user community. The crux of the issue is that, for Git to be useful for visual design, it would need to provide ways for users to inspect and compare changes to media files as media, not as raw data. That is, to provide "visual diff."

Deswaef talks to a lot of designers, and an increasing number are familiar with Git and GitHub (although they are not always clear on the distinction between the two), but they rarely take advantage of it. Most design applications today provide their own file versioning, but of a rather limiting form, such as simply saving multiple local copies of a project file with numbers tacked somewhere onto the file name. Essentially no design tools implement anything close to Git's more advanced features like branching, even though they would be just as useful for design as for code.

The question of how to convince designers to use modern version-control tools like Git has been brewing in the back of his mind for some time, Deswaef said. At LGM 2013 in Madrid, he reminded the audience, he had presented a demo of his first "design with Git" experiment, which provided a "visual diff" for SVG files kept in a Git repository. The goal was to make tracking changes in a graphics file as convenient as it is for text, and he implemented a variety of visualization options, getting positive responses from designers. But his efforts to get companies that provide Git services (like GitHub) interested in the tool proved fruitless.

Last year, he said, he found himself living in a new country and with excessive free time, so he revisited the question—first, by cataloging all of the similar work he could find that is already used by designers. Some designers work with "digital asset management" (DAM) systems, although "if you don't work in a 500-person company, you probably don't have access to one." Then there are the web-based software services. Some of the most popular are targeted at specific sub-disciplines, like InVision and Pics.io for interaction design. There is a service called Gravit that lets users fork and remix projects; ironically, it used to be open-source software, then it was taken proprietary. Another, LayerVault, went bankrupt. Its closest competitor, Pixelapse, is still around, but was purchased by Dropbox (which Pixelapse had used as its storage back-end) and has stopped receiving updates. Not that the loss is great, he added: Dropbox's idea of versioning is just appending numbers onto the name of the file.

In practice, though, the market leader is Adobe's cloud service, which he described as "Adobe Captive Cloud;" it is a Dropbox clone that comes built into Adobe's desktop applications. But even then, it is a terrible versioning tool, he said. For starters, it treats every file format other than Photoshop's PSD as an opaque binary, giving the user no previews or thumbnails. Worse still, as the fine print reveals, the service may look like a folder on your desktop, but it actually only retains copies of files for ten days, then it deletes them. "Who wants to access something they worked on more than ten days ago?" he joked. "Nobody, I guess."

But if it seems like the proprietary offerings are not serving designers well, the free-software world is not doing much better. Anecdotally, it seemed clear that many designers and other Git users store image and media files in GitHub, so Deswaef decided to get a clearer picture of how common the practice is. Using the site API, he spent several weeks querying public projects hosted on the site, retrieving each project's file listings and logging the file types in a database.

He sampled one percent of GitHub's available projects altogether: 500,000 projects (including forks), which contained about 130 million files. Of those, he said, about 12% are some form of media. More than half of the media files are PNG images, followed by JPEG and GIF at around 14% each. About 9% of the media files fell into the "other" category, which included everything from PDF to font files to audio to 3D formats like OBJ. What stood out, though, was that many of those media formats are directly viewable in browsers, but GitHub does not display them. For example, "GitHub likes to promote that people use it for font design," he said, pointing to the site's promotional material, "but it's not helping them." A few rather arbitrary formats are displayed directly. The site renders STL files (for 3D printing) to WebGL, but does not do the same for OBJ, which is five times more popular.

Alternative GitHub-like offerings such as GitLab offer the same story, but with the added drawback of not being nearly as popular of a service. Self-hosting a package like Gogs is a non-starter for most users who are not experienced with server administration. There is one free-software Git system familiar to many in the LGM community, he said: visualculture, which is the project-publishing front-end used by the non-profit Open Source Publishing (OSP) project, many members of which are long-time LGM presenters and active free-software developers. But visualculture is highly customized to tie into OSP's internal structure and release processes. While it provides visual display of many types of media, it omits other features like issue tracking, and Deswaef said he was unable to get anything more than the home page running on his own machine.

One might consider telling designers to use Git locally, he said, except that doing so with any sort of visual diff means providing a GUI front-end. And writing a usable desktop front-end for Git still seems to elude developers—although not for lack of trying. "There are as many Git GUIs as there are people upset about Git GUIs," he noted. A few standalone tools for "visual diffing" do exist, he said. P4Merge is freeware, and does not even come with an end-user license agreement (EULA), which is a whole other issue on it's own, he said. There is also webdiff, although it only presents side-by-side views of the changed file, which turns the diff process into more of a "can you spot the changes" puzzle than anything else.

He concluded by turning the questions back to the audience. "Where is my open-source Git desktop?" We have isolated web tools that need further polish, he continued; is that the future? Deswaef said he maintains a growing list of bookmarked projects that might prove helpful for designers who want to use Git. For most of the media files that GitHub cannot render directly in the browser, there is some open-source project that could provide a solution, such as three.js, a JavaScript library to transform OBJ and other 3D file formats into WebGL. But it remains up to GitHub or other services to deploy solutions of that nature.

In the question-and-answer period at the end of the session, one audience member asked if perhaps designers needed to be more open and embrace other kinds of software interfaces. Deswaef replied that he does not think giving up on visual diff is necessary. "There are a lot of people interested in this problem; big companies as well as small shops. But everyone is waiting for someone else to find the recipe that will fix the whole thing."

Eric Schrijver from OSP also spoke up, first concurring with Deswaef's assessment of the visualculture tool. "We have tried finding funds to do the work of un-tying it from us," he said, but without success. He then asked whether Git was truly a good fit for design projects, considering that the workflow of an artist or design team can be so variable, depending on the project and the participants. Deswaef replied that he did not think that was an impediment. "Git supports thousands of different workflows already," he said. Schrijver then asked whether it might be worth setting an attainable goal, like "let's implement media-file display in GitLab by LGM 2017."

Deswaef did not object to that idea, but he re-emphasized the importance of GitHub. The company has already done work to standardize project presentation, such as making Markdown-formatted README files the entry point, he said, and it has rolled in other features like GitHub pages, issues, and announcements. "What I want to see is something that you can set up for your project when you don't know in advance who you will be collaborating with." Realistically, a self-hosted solution likely does not fit that bill, given that so many designers already have trouble distinguishing between Git, the tool, and GitHub, the service.

Discussion on the subject continued well after the session was over, as one might expect at a free-software event filled with artists and designers. There is clearly a lot of room for improvement in how Git tools and services handle media files. Whether GitHub takes the hint or someone else from the free-software development community beats the company to the punch remains an open question.

[The author would like to thank Libre Graphics Meeting for travel assistance to London for LGM 2016.]

Comments (14 posted)

Refactoring the open-source photography community

Generally speaking, most free-software communities tend to form around specific projects: a distribution, an application, a tightly linked suite of applications, and so on. Those are the functional units in which developers work, so it is a natural extension from there to focused mailing lists, web sites, IRC channels, and other forms of interaction with each other and users. But there are alternatives. At Libre Graphics Meeting 2016 in London, Pat David spoke about his recent experience bringing together a new online community centered around photographers who use open-source software. That community crosses over between several applications and libraries, and it has been successful enough that multiple photography-related projects have shut down their independent user forums and migrated to the new site, PIXLS.US.

The impetus for the project, David said, was his annoyance with bad photography tutorials—specifically those for open-source applications. Bad tutorials exist for proprietary software, too, but they are disproportionately common for open-source programs. The majority of the photography tutorials for applications like GIMP, he said, address only "really basic" functions like "how to apply an unsharp mask." Worse still, they have a tendency to be overly "monetized" in the advertising sense, "giving you ten pages of four lines each filled with as many ads as they can fit on the screen."

While that would be irritating for any reader, it particularly bothered David because, as a serious "semi-professional" photographer, he knew how capable GIMP and other open-source applications are for real photographic work, but it seemed like that word was not getting out. The high-quality photography tutorials were "pretty much proprietary-only," and at online photographer user communities like Luminous Landscape or DPReview, open-source software is a subject non grata. "If they talk about it at all, it's just in passing or else it's somebody telling me GIMP can't handle high bit-depths."

Then, one day in 2011, everything changed when a discussion about a particularly exotic image came up on a GIMP discussion forum. The presiding opinion was that it was probably too complicated to reproduce with GIMP. David chimed in and said "no, that's easy to do in GIMP," to which someone else replied: "Okay; so show me." Though initially taken aback, David took the challenge seriously and, four months later, came back with a detailed tutorial. The author of the "show me" comment turned out to be Alexandre Prokoudine, himself a photographer and a GIMP contributor.

David then spent a few years writing photo tutorials on his personal blog but, by 2014, he decided that a dedicated community site was still missing. While there are several good GIMP user sites and blogs (he highlighted Meet the GIMP and GIMP Chat as examples), they focus on GIMP in general, rather than photography. When the majority of a forum's discussions are about "making chat-room avatars," he said, one cannot build a community of photographers.

Thus, PIXLS.US was born. David planned the site to have two halves: a collection of articles and blog posts on one side, plus a discussion forum on the other. He developed the article side (with the help of Marcus Rückert) using the static-site generator Metalsmith, in order to best support a (hopefully) high traffic load for image-heavy pages.

For the forum side, he spent time evaluating the options, and was displeased with most of them. Most open-source options are self-hosted (like phpBB) and require users to create yet another username and password—which only increases the "friction" of getting involved in the site. "We know comments on the Internet suck," he said, and, for their solution, more and more sites seemed to be trending toward embedding page widgets from the proprietary discussion service Disqus. But that was a non-starter. "Do you really want to do that to your users?" he asked, "have them tracked by Disqus?"

Eventually, he settled on Discourse, the open-source, embeddable discussion framework started by Jeff Atwood and Robin Ward. It runs on a Digital Ocean droplet that costs David a few dollars a month, but it supports dedicated discussion threads as well as threads attached to posts on the article side, serving as post comments. Moreover, it allows users to log in through any of a number of OAuth2-capable web services if the user chooses (such as Facebook or Google), but permits email-based account creation, too.

The site launched in April 2015 and has grown to 475 active users thus far. But, while the site started out as a photographer's discussion forum, in July 2015 the G'MIC project decided to move its official discussion forum from its own site to PIXLS.US. Then in October, the RawTherapee project followed suit. The PhotoFlow project became the third to join in January 2016, providing the biggest jump in numbers the site has seen to date. At the same time, the number of tutorial contributors has continued to grow as well.

Having users from all three projects share a common discussion site solves the "separate forum" problem, David said. Users make use of multiple applications in their workflow, so it is an easy step from there to discussing workflows and techniques in a single place. Even better, he said, the "cross-pollination" effect has already had a noticeable impact on two new projects. Carlo Vaccari's Filmulator started out as a stand-alone utility, but after soliciting input from PIXLS.US users, Vaccari was persuaded to re-implement it as a plugin for darktable. And Damon Lynch got valuable early design feedback from PIXLS.US discussions when he launched Rapid Photo Downloader.

Moving forward, David said his emphasis for now is on writing more content for the article side. "There is a trade-off between frequency and quality, and my tendency has been to err on the side of quality." Nevertheless, he said that PIXLS.US would be happy to have other projects join the discussion-side of the site.

In response to an audience question, David said that the only step he took to lure existing projects over to the PIXLS.US forum was to make the offer to the development team and see if they replied. For some users, he said, the process of moving to a different site was painful, since the normal "migration" involved the development team announcing the transition and locking the old discussion forum. But that is part of why he made an effort to provide multiple, low-friction ways to create an account at the new forum.

Another audience member asked what the comment-moderation experience has been like. Thus far, David said, the community has had very little trouble with disruptive posters or trolls. Discourse includes a robust set of built-in reputation and moderation features, and so far, no one who works on the forum has had any need to intervene personally. "But who knows," he said, "if some of the GIMP forums were to migrate over, we might have a different story, given how wild and polarizing they can be. But so far, it's been great."

David's posts on the article side of the site are, indeed, in-depth and focus on high-quality image work like one would see on any other dedicated photography site. He has written about luminosity masks, wavelet decomposition, and color-curve matching, just to name a few; topics the casual GIMP user may never had heard of. But it is hard to argue with the results—as many other open-source projects would no doubt agree, having a skilled set of users to write tutorials can reap large benefits.

That said, there may be lessons worth learning from the discussion-forum side as well. Intentionally creating an online community based around a type of user, rather than a specific application or a Linux distribution, is rare. But cross-pollination is a facet of most (if not all) disciplines, and it hard to imagine that there are no other communities that could find similar benefits from consolidating their discussions in one place rather than dividing them up on a project-by-project basis. For everyone not grappling with such philosophical ideas, though, PIXLS.US continues to work on its own terms: demonstrating that high-end photographic work is right at home with open-source software.

[The author would like to thank Libre Graphics Meeting for travel assistance to London for LGM 2016.]

Comments (1 posted)

Book review: Designing with LibreOffice

Designing with LibreOffice (DWL) is a new book by Bruce Byfield exploring the discipline of typographical design and how eye-pleasing results can be achieved with the LibreOffice (LO) office suite. Well over half of the considerable 500 pages are dedicated to creating documents with Writer, but slide shows, drawings and even spreadsheets need some consideration given to design and they are not ignored.

The target audience is anyone who creates with LibreOffice and wants their creations to look better, whether as a novice or a seasoned traveler wanting to make better use of a complex tool. Byfield provides a mix of objective practical advice and subjective personal preferences. Even when those personal choices seem different than what you might prefer, you will find them presented with sufficient clarity and justification that you'll be able to identify and name the things you want to change and the things that can be safely preserved.

Jack of all trades

DWL weaves together three distinct goals, each serving to support the other two. Primarily, as the title indicates, the book is about the style of a document, particularly the visual typographic style. The general approach to style that is encouraged is one of simplicity and uniformity. These abstract ideas are taken into various corners of the design process, with space given to font choice, use of white space, and page color, just to name a few.

With so many fonts available today, making a choice can become bewildering. To assist, Byfield spends time on both historical and structural details of fonts so that the reader will know the correct terminology and be able to ask the right questions to focus in on which of the available fonts might be most appropriate in a given setting.

On the question of white space, the reader again has options reduced through the dictum that all spacing, whether margins, indents, or vertical gaps, should be multiples of, or occasionally submultiples of, the baseline-to-baseline line spacing. It may seem strange to correlate horizontal and vertical space so closely, but to achieve the larger goal of uniformity some sort of rule is needed and this one certainly seems as good as any.

Page color may at first seem a confusing term as it has nothing to do with hue or saturation and is purely about shades of grey. Depending on details of the font, how thick various strokes are, and of spacing, what inter-character and inter-line gaps are used, the page as a whole can look lighter or darker. DWL encourages the designer to think carefully about this and to adjust spacing and font weight if necessary to get the result most fitting for the document. This is a simple example of how the design guidelines go well beyond thinking about what LibreOffice happens to let you do and very much into the realm of appearance independent of a particular enabling technology.

The second theme is of a gentle introduction and tutorial in using LibreOffice to create these well-designed documents. Throughout the description of what a document should look like there are step-by-step instructions on how to coax LibreOffice to produce that appearance. Sometimes, the steps given seem overly simplistic or obvious but, just as often, there are pointers to functionality that might be hard to find but is useful. This ensures that the text is useful to a broad range of readers. To get the most value out of the tutorial side of the book, it would be a big help to have LibreOffice open on a non-trivial document for experimentation. As the Open Document source for the book is available for download, it would likely make a good sample to experiment with.

Finally, DWL gives the impression that it could be quite useful as a reference guide, not only for referring back to those step-by-step instructions, but also for the various summary lists of points to consider when making a decision, such as choosing a font, making use of tab stops, or even the perfect positioning of superscripts and subscripts. Unfortunately, the lack of an index impedes this usefulness somewhat. The table of contents is thorough, but I had cause to look back to be reminded how to update the color palette and all the references to "Color" in the table of contents were to entirely the wrong sort of color, as discussed above.

Some things I learned

A persistent theme throughout the book is the importance of making full use of styles. The position presented is quite uncompromising:

While a great many useful details are presented, I quickly developed the feeling that I was missing something. It was as though I couldn't see the forest for the trees, probably because I had some preconceived ideas about styles that were not a perfect match for LibreOffice and were not being directly challenged by the presentation in DWL. Eventually, all the pieces fell into place and, looking back, I can identify two ideas that would have been more helpful to me had they been introduced earlier and more explicitly.

The first is that styling can be introduced into a document either by value or by reference (though DWL doesn't use these terms). If we accept styling as a generic concept for giving a name to a collection of one or more specific styling decisions, then one of the simplest examples would be the creation a color palette: giving specific names to selected colors. These could provide corporate branding, or maybe there is a single color called "highlight" used to provide uniform highlights. These colors can then be included as appropriate into the document, but they will always be included by value, not by reference. If, after creating your document, you go back to your color palette and change "highlight" be a slightly darker color, the colors in your document will not change at all.

This styling-by-value also applies to the styling tables using "AutoFormat" and is an approach that can be very effective when creating drawings. If you want a number of elements in a drawing to be the same size and shape, this uniformity can only be achieved by creating an original and copying that value around.

Styling-by-reference is the more traditional form of styling. A "style" can be defined that specifies various attributes for a character, a paragraph, or a list item, etc. Each such object references a style from which it inherits attributes, and that style can in turn inherit from a parent style, and so on. It would be possible to create a "highlight" style that sets text color, rather than a "highlight" color as mentioned earlier, and have all styles that require highlighting inherit from this. That would make changing the highlight color easy.

There is a blurry line between by-value and by-reference styling when it comes to the use of external templates, and this might have caused part of my vague feeling of uncertainty. A "template" is a LibreOffice document much like any other, though with a slightly different name extension, and with no document content. What it does contain is styles. When you create a new document based on a template, the new document gets both a copy of the styles in the template and a reference to the template. If the template is subsequently changed, the reference allows new styles to be copied into the document with little effort, but it is always the copy of the styles in the document, not the ones in the template, that are used. So a template is quite a different thing than a CSS style sheet used with HTML or a LaTeX .sty file.

The other fundamental aspect of LibreOffice styling, which is only indirectly described in the text, is that it is really all about paragraphs. Without wanting to belittle characters and frames, or ignore pages completely, it is necessary to realize that when thinking about the structure of a document there are no chapters, sections, subsections, or lists. There are just paragraphs, some of which may look like chapter or section headings, and several of which might combine to create the appearance of a list. This aspect of LO styling was made particularly apparent in DWL in the various break-out passages such as Tips and Cautions.

Each of these is structured with two paragraph styles, the first being a list item style, despite the fact that there is no list in sight. The need to place a graphic just before the introductory word "Tip" is easily met by defining a bullet list style with the chosen graphic as the bullet. A "Note - Tip" style chooses that bullet and sets the font, color, and positioning for the word "Tip". Then a "Note" style for subsequent paragraphs ensures that the body of the Tip appears in the desired place with the desired appearance.

There are two particular properties of a paragraph style that help to synthesize larger-scale structure out of what is really a list of paragraphs. A paragraph style can identify a "Next style" which will be the default for the next paragraph. "Note - Tip", as well as "Note - Caution" and others, identify "Note" as the "Next style". When the note is finished, the style must be explicitly reset to "Text Body". A paragraph style (typically for a chapter or section heading) can also be given an "Outline Level" which allows a hierarchical structure to be extracted from the document and presented in the "Navigator" dialogue. These two allow multiple paragraph styles to work together to provide the appearance of larger structures, but those structures are not imposed in the way they might be by an XML DTD (Document Type Definition).

While all this information is contained in DWL, the big picture is left for the reader to deduce rather than being spelled out.

A moment of self reflection

Being a book that discusses the style of (among other things) books, it seems unavoidable that the metrics given in DWL should be used to measure the book itself. On the whole it passes with flying colors, being pleasant to read and possessing a visual style that is distinctive without being distracting.

There are just two fonts, one for the main body of text and one for various break-out sections, and there is one alternate color, a muted green, used for various highlights such as section headings, page numbers, and list bullets.

The design guidelines repeatedly encourage the use of white space for separating distinct elements rather than more direct methods like borders or alternate backgrounds. While this seems to work well for the most part, there is one area where it falls down. The captions for the various screen-shots are set in the break-out font, are slightly closer to the screen-shot than to surrounding text, and do not have the first-line indented the way that body-text does. Nonetheless, several times I found myself reading those captions as though they were part of the regular flow of text, which was quite distracting. For me at least, a stronger contrast would help, even if it was just centering the caption text rather than left-aligning it.

Chapter 16 begins by acknowledging, and then discussing, the considerable difficulties involved in finalizing a document — correcting those issues that a word-by-word focus won't see, but which require a much broader perspective. In what can only have been a serious breakdown of the editorial processes, the book provides clear examples of what can go wrong. Chapter 5, "Spacing on all sides" contains two sections (at different nesting levels) named "Spacing between paragraphs". It is not just the name that is duplicated but also most of the text and even the caption of a screenshot, though it is a different screenshot in each case. While this is by far the worst lapse, there are two or three other places where text is needlessly duplicated. These problems don't really detract from the value of the book, but they do distract from focusing on the important issues.

For your bookshelf?

There is no doubt that DWL contains a wealth of information gained over years of experience. This information is conveyed in a coherent and approachable style. Even when it is not able to provide the complete and definitive answers we might want to get on with creating a document, it provides so many hints, pointers, and perspectives that it will doubtless set you on the right path to finding what you need to know.

DWL is licensed under the Creative Commons Attribution-ShareAlike license and can be downloaded in source or PDF formats, or purchased from a print-on-demand service. "Thank you" contributions from $2 to $25 can easily be made through the book's web site. For anyone who regularly works with nontrivial documents and wants to lift their game, or anyone with a general interest in improving their LibreOffice skills, this book would be a worthwhile investment.

Comments (17 posted)

SMTP Strict Transport Security

It is no secret that email has been a weak point in the security of the standard Internet protocol suite for quite some time. While improvements to HTTP and TLS have resulted in more robust web security, mail transfer has lagged behind. Now, a new draft RFC has been submitted to the Internet Engineering Task Force (IETF) that may make SMTP connections substantially more secure, closing loopholes that have made mail transfer vulnerable to man-in-the-middle and cryptographic-downgrade attacks.

The draft is called SMTP Strict Transport Security (SMTP STS), and it was first posted on March 18. The authors work for a variety of well-known technology companies and ISPs, including Google, Yahoo, Comcast, Microsoft, LinkedIn, and 1&1 Internet. It sets out a standard by which a mail server can publish an STS Policy that defines how mail transfer agents (MTAs) can connect using TLS, how they can validate the server's TLS certificate, and what they should do in the case that a TLS session cannot be correctly established. As the name implies, SMTP STS is similar in design to HTTP Strict Transport Security (HSTS), defined in RFC6797.

SMTP was originally defined in RFC821, long before TLS (or SSL) were invented, and sent all messages in the clear. The STARTTLS extension (RFC3207) added support for TLS transport in 2002. Although reports are that STARTTLS has proven popular with mail providers, it suffers from some severe shortcomings.

First, the TLS session is subject to downgrade attacks. In the SMTP session-establishment handshake, the STARTTLS command is sent from one machine to the other, in the clear. Upon receipt, the remote machine is intended to respond by initiating a TLS handshake before the SMTP session continues. But an attacker sitting in between the two machines can simply intercept and drop the STARTTLS command (or overwrite it with garbage), in which case the SMTP session will continue in plaintext.

The second shortcoming is that, in a STARTTLS handshake, client machines do not attempt to verify that the TLS certificate presented by the server actually belongs to the expected domain. Here again, an attacker in the middle can intercept the genuine certificate and replace it with a fraudulent one, without the client knowing the difference.

SMTP STS defines a way for a mail server to publish a public record that TLS is available and that SMTP clients should thus use it when making connections. This record is the STS Policy, and has seven mandatory fields:

• v - the SMTP STS version. Only a value of STS1 is supported.
• to - a boolean indicating whether the server is "TLS Only." If true, then clients who encounter errors in the TLS setup must not deliver any mail to the server.
• mx - a comma-separated list of domains for which the policy applies.
• a - the authentication mechanism clients should use to verify that the policy is authentic. The supported mechanisms are DNSSEC and an HTTPS URI at which the client can separately request the policy and verify that the two copies are identical.
• c - the validation method clients should use to verify the authenticity of the server's TLS certificate. The supported methods are DANE TLSA (RFC7672) and verifying the PKI trust chain on the certificate back to a root certificate authority (that is, traditional TLS identity checking as specified in RFC6125).
• e - the maximum lifetime of the policy.
• rua - the address to which feedback and errors may be reported by the client.

A server can publish its STS Policy at https://example.com/.well-known/smtp-sts or via DNS as a TXT record under the name _smtp_sts.example.com. The draft document further specifies that any errors reported to a server by clients should contain details about where unexpected results were encountered: an expired certificate, a DNSSEC failure, a certificate that does not match the server's domain, and so forth.

As Lucian Constantin at InfoWorld pointed out, one possible security loophole is that the first time a client connects to a server, it has no cached copy of the server's STS Policy to compare, so a man-in-the-middle attacker could, in theory, intercept the first request and return a forged policy instead. That forged policy would, by necessity, include an a field similarly doctored to direct the client to an attacker-controlled server when authenticating the policy.

The draft's support for DNS-based policy publishing provides the means for some additional robustness against man-in-the-middle attacks over what is offered by TLS identity checking. If a server's STS Policy is published as a TXT record in a DNSSEC-protected zone, clients can verify its authenticity. Furthermore, the c field can be used to specify that a DANE TLSA record for the server's TLS certificate is required for validation. Thus, an attacker would have to compromise the DNSSEC server in addition to intercepting the SMTP handshake.

The draft also notes that clients are not obligated to accept STS Policies the first time they retrieve them from a server. Instead, they are permitted to use the rua reporting mechanism to tell the server's administrators what they saw. That report can be sent "offline" or during a different session, potentially allowing failures and suspicious reports to be routed around a man-in-the-middle attack. Such a reporting feature is not found in HSTS; nevertheless, SMTP STS is still a "trust on first use" system.

The authors of the draft list two other options to be considered for future inclusion in STMP STS as well. They are certificate pinning, in which a policy could specify specific certificates that must appear in the certificate-validation chain, and policy distribution, in which sites' policies could be recorded and publicly tracked.

The draft also speculates that it could be worthwhile for a policy to specify the list of cryptographic ciphers and TLS versions that the server accepts, and perhaps to specify that all mail received from the server's domain should be expected to arrive over TLS. The latter feature would, in theory, allow a mail server to check for an STS Policy for incoming mail domains and, thus, reject mail that is "supposed to" arrive over TLS but does not. The draft notes, however, that such a policy would come with its share or potential man-in-the-middle attacks to be guarded against, and may not offer significant gains.

STARTTLS has, no doubt, improved the security of e-mail transfer, but it is far from complete. SMTP STS hopefully closes some of the more egregious holes.

Comments (15 posted)

Security quotes of the week

— Mark Shuttleworth

— The attacker responsible for the Hacking Team data breach

Comments (5 posted)

Gone In Six Characters: Short URLs Considered Harmful for Cloud Services (Freedom to Tinker)

Over at the Freedom to Tinker blog, guest poster Vitaly Shmatikov, who is a professor at Cornell Tech, writes about his study [PDF] of what URL shortening means for the security and privacy of cloud services. "TL;DR: short URLs produced by bit.ly, goo.gl, and similar services are so short that they can be scanned by brute force. Our scan discovered a large number of Microsoft OneDrive accounts with private documents. Many of these accounts are unlocked and allow anyone to inject malware that will be automatically downloaded to users’ devices. We also discovered many driving directions that reveal sensitive information for identifiable individuals, including their visits to specialized medical facilities, prisons, and adult establishments."

Comments (25 posted)

How Badlock was discovered and fixed

This post on the Red Hat Enterprise Linux blog describes the discovery and repair of the "Badlock" vulnerability. One begins to understand a little better why it took as long as it did. "The code was rewritten; in March 2016 the changes needed to fix all eight CVEs amounted to about 200 individual patches against a development version of Samba, with about half of those responsible for fixing CVE-2015-5370. When backported to previous stable Samba versions, they needed additional hundred patches. To oldest supported Samba version — about four hundred patches. What started as an individual snowflake became an avalanche but it wasn’t finished yet."

Comments (7 posted)

The Android Security 2015 Annual Report

Google has announced the availability of the Android security 2015 year in review [PDF]. "Android’s open source model has also allowed device manufacturers to introduce new security capabilities. Samsung KNOX, for example, has taken advantage of unique hardware capabilities to strengthen the root of trust on Samsung devices. Samsung has also introduced new kernel monitoring capabilities on their Android devices. Samsung is not unique in their contributions to the Android ecosystem. Blackberry has worked to enhance the security of their devices by enabling kernel hardening and other features in the Blackberry PRIV. CopperheadOS has both introduced security improvements to their own version of Android and made significant contributions to the Android Open Source Project. These are just some of the various contributions made possible through open sourcing that improved the Android ecosystem in 2015."

Comments (25 posted)

apparmor: profile updates

Package(s):	apparmor	CVE #(s):
Created:	April 20, 2016	Updated:	April 20, 2016
Description:	From the openSUSE advisory: This update for apparmor updates some profiles. It is specifically required for the Samba security update.
Alerts:	openSUSE openSUSE-SU-2016:1108-1 apparmor 2016-04-20

Comments (none posted)

chromium: multiple vulnerabilities

Package(s):	chromium-browser	CVE #(s):	CVE-2016-1651 CVE-2016-1652 CVE-2016-1653 CVE-2016-1654 CVE-2016-1655 CVE-2016-1657 CVE-2016-1658 CVE-2016-1659 CVE-2016-1656
Created:	April 15, 2016	Updated:	April 25, 2016
Description:	From the Debian advisory: CVE-2016-1651: An out-of-bounds read issue was discovered in the pdfium library. CVE-2016-1652: A cross-site scripting issue was discovered in extension bindings. CVE-2016-1653: Choongwoo Han discovered an out-of-bounds write issue in the v8 javascript library. CVE-2016-1654: Atte Kettunen discovered an uninitialized memory read condition. CVE-2016-1655: Rob Wu discovered a use-after-free issue related to extensions. CVE-2016-1657: Luan Herrera discovered a way to spoof URLs. CVE-2016-1658: Antonio Sanso discovered an information leak related to extensions. CVE-2016-1659: The chrome development team found and fixed various issues during internal auditing. Added CVE-2016-1656 from Red Hat advisory: android downloaded file path restriction bypass
Alerts:	Gentoo 201605-02 chromium 2016-05-17 Ubuntu USN-2955-1 oxide-qt 2016-04-27 openSUSE openSUSE-SU-2016:1136-1 Chromium 2016-04-24 openSUSE openSUSE-SU-2016:1135-1 Chromium 2016-04-24 Mageia MGASA-2016-0143 chromium-browser-stable 2016-04-21 SUSE SUSE-SU-2016:1060-1 Chromium 2016-04-17 openSUSE openSUSE-SU-2016:1061-1 Chromium 2016-04-17 Arch Linux ASA-201604-10 chromium 2016-04-17 Red Hat RHSA-2016:0638-01 chromium-browser 2016-04-18 Debian DSA-3549-1 chromium-browser 2016-04-15

Comments (none posted)

cryptopp: information disclosure

Package(s):	cryptopp	CVE #(s):	CVE-2016-3995
Created:	April 15, 2016	Updated:	April 25, 2016
Description:	From the cryptopp bug report: For both Rijndael::Enc::ProcessAndXorBlock and Rijndael::Dec::ProcessAndXorBlock there is some code to avoid timing attacks [...] This counter measure seems to be removed by the compiler. Hence, the binary may be vulnerable to timing attacks.
Alerts:	Mageia MGASA-2016-0147 libcryptopp 2016-04-25 Fedora FEDORA-2016-96b60bdb96 cryptopp 2016-04-15

Comments (none posted)

imlib2: denial of service

Package(s):	imlib2	CVE #(s):	CVE-2016-3993
Created:	April 14, 2016	Updated:	April 20, 2016
Description:	From the Mageia advisory: An out-of-bounds read caused by an off-by-one error in __imlib_MergeUpdate() in src/lib/updates.c in imlib2 1.4.8 and earlier (CVE-2016-3993).
Alerts:	Ubuntu USN-3075-1 imlib2 2016-09-08 Fedora FEDORA-2016-b4212484d5 imlib2 2016-05-24 Fedora FEDORA-2016-ff070e8faa imlib2 2016-05-24 openSUSE openSUSE-SU-2016:1330-1 imlib2 2016-05-18 Arch Linux ASA-201605-1 imlib2 2016-05-04 Debian DSA-3555-1 imlib2 2016-04-23 Mageia MGASA-2016-0140 imlib2 2016-04-13

Comments (none posted)

kernel: three vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2016-3951 CVE-2015-8839 CVE-2016-3672
Created:	April 20, 2016	Updated:	April 20, 2016
Description:	From the Red Hat bugzilla: A vulnerability was found in the usbnet Linux kernel driver. The bug allows physically proximate attackers to cause a denial of service (NULL pointer dereference and system crash) or possibly have other impact by inserting a USB device with an invalid USB descriptor. (CVE-2016-3951) A flaw was found in the Linux kernel when attempting to "punch a hole" in files existing on an ext4 filesystem. When punching holes into a file races with the page fault of the same area, it is possible that freed blocks remain referenced from page cache pages mapped to process' address space. Thus modification of these blocks can corrupt data someone else is now storing in those blocks when at some point those pages are written to disk. (CVE-2015-8839) A weakness was found in the Linux ASLR implementation. Any user able to running 32-bit applications in a x86 machine can disable the ASLR by setting the RLIMIT_STACK resource to unlimited. (CVE-2016-3672)
Alerts:	openSUSE openSUSE-SU-2016:2184-1 kernel 2016-08-29 openSUSE openSUSE-SU-2016:2144-1 kernel 2016-08-24 SUSE SUSE-SU-2016:2105-1 the Linux Kernel 2016-08-19 SUSE SUSE-SU-2016:1937-1 kernel 2016-08-02 SUSE SUSE-SU-2016:1764-1 kernel 2016-07-08 Ubuntu USN-3021-2 linux-ti-omap4 2016-06-27 Ubuntu USN-3021-1 kernel 2016-06-27 SUSE SUSE-SU-2016:1690-1 kernel 2016-06-27 SUSE SUSE-SU-2016:1696-1 kernel 2016-06-28 Debian DSA-3607-1 kernel 2016-06-28 Mageia MGASA-2016-0233 kernel-tmb 2016-06-22 openSUSE openSUSE-SU-2016:1641-1 kernel 2016-06-21 Debian-LTS DLA-516-1 kernel 2016-06-17 Mageia MGASA-2016-0225 kernel 2016-06-13 Ubuntu USN-2997-1 linux-ti-omap4 2016-06-09 Ubuntu USN-3004-1 linux-raspi2 2016-06-09 Ubuntu USN-3007-1 linux-raspi2 2016-06-10 Ubuntu USN-3005-1 linux-lts-xenial 2016-06-10 Ubuntu USN-3002-1 linux-lts-wily 2016-06-09 Ubuntu USN-3001-1 linux-lts-vivid 2016-06-09 Ubuntu USN-3000-1 linux-lts-utopic 2016-06-09 Ubuntu USN-2998-1 linux-lts-trusty 2016-06-09 Ubuntu USN-2996-1 kernel 2016-06-09 Ubuntu USN-3003-1 kernel 2016-06-09 Ubuntu USN-3006-1 kernel 2016-06-10 Ubuntu USN-2989-1 kernel 2016-05-31 openSUSE openSUSE-SU-2016:1382-1 kernel 2016-05-23 Ubuntu USN-2965-4 linux-snapdragon 2016-05-06 Ubuntu USN-2965-3 linux-raspi2 2016-05-06 Ubuntu USN-2965-2 linux-lts-xenial 2016-05-06 Ubuntu USN-2965-1 kernel 2016-05-06 Fedora FEDORA-2016-373c063e79 kernel 2016-05-06 Fedora FEDORA-2016-8e858f96b8 kernel 2016-04-19

Comments (none posted)

libtasn1: denial of service

Package(s):	libtasn1	CVE #(s):	CVE-2016-4008
Created:	April 15, 2016	Updated:	May 31, 2016
Description:	From the Red Hat bugzilla entry: The libtasn1 library, in its 4.7 version, can loop for a long time or indefinitely when it is used to parse DER representations of X509 certificates, leading to a denial of service. Some of these loops may in addition increase heap or stack usage, leading to more issues.
Alerts:	openSUSE openSUSE-SU-2016:1674-1 libtasn1 2016-06-24 openSUSE openSUSE-SU-2016:1567-1 libtasn1 2016-06-14 Debian-LTS DLA-495-1 libtasn1-3 2016-05-30 Mageia MGASA-2016-0170 libtasn1 2016-05-11 Debian DSA-3568-1 libtasn1-6 2016-05-05 Ubuntu USN-2957-2 libtasn1-6 2016-05-02 Ubuntu USN-2957-1 libtasn1-3, libtasn1-6 2016-05-02 Fedora FEDORA-2016-96bfd9e873 libtasn1 2016-04-27 Fedora FEDORA-2016-383b8250e6 libtasn1 2016-04-22 Arch Linux ASA-201604-9 libtasn1 2016-04-17 Fedora FEDORA-2016-048ffb6235 libtasn1 2016-04-15

Comments (none posted)

openssh: privilege escalation

Package(s):	openssh	CVE #(s):	CVE-2015-8325
Created:	April 18, 2016	Updated:	December 15, 2016
Description:	From the Debian advisory: Shayan Sadigh discovered a vulnerability in OpenSSH: If PAM support is enabled and the sshd PAM configuration is configured to read user- specified environment variables and the "UseLogin" option is enabled, a local user may escalate her privileges to root.
Alerts:	Red Hat RHSA-2016:2588-02 openssh 2016-11-03 Mageia MGASA-2016-0280 openssh 2016-08-31 Slackware SSA:2016-219-03 openssh 2016-08-06 openSUSE openSUSE-SU-2016:1455-1 openssh 2016-05-31 Fedora FEDORA-2016-d31c00ca51 gsi-openssh 2016-05-24 Ubuntu USN-2966-1 openssh 2016-05-09 Fedora FEDORA-2016-7f5004093e openssh 2016-04-24 Debian DSA-3550-1 openssh 2016-04-15 Scientific Linux SLSA-2016:2588-2 openssh 2016-12-14 Gentoo 201612-18 openssh 2016-12-07

Comments (none posted)

optipng: denial of service

Package(s):	optipng	CVE #(s):	CVE-2015-7802
Created:	April 14, 2016	Updated:	April 20, 2016
Description:	From the Mageia advisory: CVE-2015-7802 - Buffer over-read issue
Alerts:	Ubuntu USN-2951-1 optipng 2016-04-18 Mageia MGASA-2016-0135 optipng 2016-04-13

Comments (none posted)

optipng: code execution

Package(s):	optipng	CVE #(s):	CVE-2016-3981 CVE-2016-3982
Created:	April 18, 2016	Updated:	April 20, 2016
Description:	From the CVE entries: Heap-based buffer overflow in the bmp_read_rows function in pngxrbmp.c in OptiPNG before 0.7.6 allows remote attackers to cause a denial of service (out-of-bounds read or write access and crash) or possibly execute arbitrary code via a crafted image file. (CVE-2016-3981) Off-by-one error in the bmp_rle4_fread function in pngxrbmp.c in OptiPNG before 0.7.6 allows remote attackers to cause a denial of service (out-of-bounds read or write access and crash) or possibly execute arbitrary code via a crafted image file, which triggers a heap-based buffer overflow. (CVE-2016-3982)
Alerts:	Gentoo 201608-01 optipng 2016-08-10 Ubuntu USN-2951-1 optipng 2016-04-18 openSUSE openSUSE-SU-2016:1082-1 optipng 2016-04-17 openSUSE openSUSE-SU-2016:1078-1 optipng 2016-04-17

Comments (none posted)

poppler: code execution

Package(s):	poppler	CVE #(s):	CVE-2015-8868
Created:	April 15, 2016	Updated:	December 15, 2016
Description:	From the Red Hat bugzilla entry: A heap buffer overflow vulnerability was found in the poppler library. A maliciously crafted file could cause the application to crash.
Alerts:	Oracle ELSA-2016-2580 poppler 2016-11-10 Red Hat RHSA-2016:2580-02 poppler 2016-11-03 openSUSE openSUSE-SU-2016:1630-1 poppler 2016-06-19 openSUSE openSUSE-SU-2016:1322-1 poppler 2016-05-17 Ubuntu USN-2958-1 poppler 2016-05-02 Debian-LTS DLA-446-1 poppler 2016-04-29 Debian DSA-3563-1 poppler 2016-05-01 Fedora FEDORA-2016-3c4e438fc8 mingw-poppler 2016-04-25 Fedora FEDORA-2016-c848d48286 mingw-poppler 2016-04-25 Mageia MGASA-2016-0145 poppler 2016-04-21 Fedora FEDORA-2016-a97dfe609c poppler 2016-04-15 Scientific Linux SLSA-2016:2580-2 poppler 2016-12-14 Gentoo 201611-15 poppler 2016-11-22

Comments (none posted)

postgresql: two vulnerabilities

Package(s):	postgresql	CVE #(s):	CVE-2016-2193 CVE-2016-3065
Created:	April 14, 2016	Updated:	April 20, 2016
Description:	From the Mageia advisory: A vulnerability in PostgreSQL 9.3.x before 9.3.12 and 9.4.x before 9.4.7 leads to potentially incorrect policies being applied in cases where role-specific policies are used and a given query is planned under one role and then executed under other roles, which could happen under security definer functions or when a common user and query is planned initially and then re-used across multiple SET ROLEs. Applying an incorrect policy may permit a user to complete otherwise-forbidden reads and modifications. This affects only databases that have used CREATE POLICY to define a row security policy (CVE-2016-2193). A vulnerability was found in a way PostgreSQL 9.3.x before 9.3.12 and 9.4.x before 9.4.7 uses pageinspect functions. Certain function arguments crashed the server or disclosed a few bytes of server memory. The viability of attacks that arrange for presence of confidential information in the disclosed bytes was not ruled out. This affects only databases that have used "CREATE EXTENSION pageinspect" (CVE-2016-3065).
Alerts:	Mageia MGASA-2016-0136 postgresql 2016-04-13

Comments (none posted)

qpid-proton: TLS to plaintext downgrade

Package(s):	qpid-proton	CVE #(s):	CVE-2016-2166
Created:	April 15, 2016	Updated:	April 20, 2016
Description:	From the Red Hat bugzilla entry: Messaging applications using the Proton Python API to provision an SSL/TLS encrypted TCP connection may actually instantiate a non-encrypted connection without notice if SSL support is unavailable. This will result in all messages being sent in the clear without the knowledge of the user.
Alerts:	Fedora FEDORA-2016-e6e8436b98 qpid-proton 2016-04-15

Comments (none posted)

quagga: password disclosure

Package(s):	quagga	CVE #(s):
Created:	April 14, 2016	Updated:	April 20, 2016
Description:	From the openSUSE advisory: /etc/quagga and its contents were world-readable despite containing passwords.
Alerts:	openSUSE openSUSE-SU-2016:1030-1 quagga 2016-04-14

Comments (none posted)

systemd: two vulnerabilities

Package(s):	systemd	CVE #(s):	CVE-2014-9770 CVE-2015-8842
Created:	April 19, 2016	Updated:	April 20, 2016
Description:	From the openSUSE bug report: On Leap archived journal files in /var/log/journal/$uuid are all world-readable - both system and user. The current journal (i.e. system.journal for system instance or user-$UID.journal for user) are correctly restricted to root/systemd-journal or owning user). This allows any user to read anyone's archived journals by using explicit journalctl --file=/path/to/archived/file This problem does not exist on Tumbleweed where access to archived files are restricted in the same way as to current journal. When user attempts to read journal without specifying filename, journalctl tries to open current journal first and errors out, which hides this issue. On 13.2 it is worse - even current log in /run/log/journal is world-readable. (CVE-2014-9770) Unspecified (CVE-2015-8842)
Alerts:	openSUSE openSUSE-SU-2016:1414-1 systemd 2016-05-27 openSUSE openSUSE-SU-2016:1101-1 systemd 2016-04-19

Comments (none posted)

tiff: denial of service

Package(s):	tiff	CVE #(s):	CVE-2016-3186
Created:	April 18, 2016	Updated:	April 20, 2016
Description:	From the openSUSE bug report: A buffer overflow vulnerability was reported in libtiff library, in gif2tiff component. A maliciously crafted file could cause the application to crash.
Alerts:	Debian-LTS DLA-693-1 tiff 2016-11-02 Mageia MGASA-2016-0349 libtiff 2016-10-21 openSUSE openSUSE-SU-2016:2375-1 tiff 2016-09-25 openSUSE openSUSE-SU-2016:2321-1 tiff 2016-09-16 Debian-LTS DLA-610-1 tiff3 2016-09-05 Gentoo 201701-16 tiff 2017-01-09 openSUSE openSUSE-SU-2016:1103-1 tiff 2016-04-19 openSUSE openSUSE-SU-2016:1081-1 tiff 2016-04-17 Arch Linux ASA-201611-26 libtiff 2016-11-25 Arch Linux ASA-201611-27 lib32-libtiff 2016-11-25

Comments (none posted)

Kernel release status

The current development kernel is 4.6-rc4, released on April 17. "So there really isn't anything particularly interesting here. Just like I like it in the rc series. Let's hope it stays that way."

Stable updates: none have been released in the last week. The 4.5.2, 4.4.8, and 3.14.67 updates are in the review process as of this writing; they can be expected imminently.

Sasha Levin has released a set of stable-security kernel updates as well.

Comments (none posted)

The 2016 Linux Storage, Filesystem, and Memory-Management Summit

The 2016 Linux Storage, Filesystem, and Memory-Management Summit was held April 18 and 19 in Raleigh, North Carolina, USA. On the order of 100 developers representing those subsystems discussed a wide range of highly technical topics. LWN was there, resulting in the following reports.

The summit ran from one to three tracks, depending on the subjects to be discussed.

The plenary track included developers from all three subsystems and covered issues relevant to the kernel as a whole. The sessions from this track were:

• Standards update: what the T10 (SCSI), T13 (ATA), and NVM Express (NVMe) standards groups have been working on.

• Persistent-memory error handling: what should the system do when persistent memory turns out to be less persistent than it should be?

• Bulk memory-allocation APIs as a way of addressing networking performance bottlenecks.

• reflink() and related topics: further development of the reflink() system call, an online scrubber for XFS, and more in a "plenary" session that the memory-management developers were too busy to attend.

• Filesystems and containers / Self-encrypting drives: two lightning talks to finish out the first day.

• Multi-order radix trees: an enhancement to radix-tree functionality that might be useful beyond the memory-management subsystem.

• Performance-differentiated memory: how to cope with systems featuring memory with varying performance characteristics.

• DAX on BTT. How to make the DAX direct-access layer play well with BTT, which, inherently, is a layer of indirection.

The memory management track discussed the following topics:

• Two transparent huge page cache implementations. Transparent huge pages don't currently work with file-backed pages, but that's not through a lack of trying: there are currently two working implementations to choose between. In this session, the memory-management developers spent two hours trying to make that decision.

• Ideas for rationalizing GFP flags: how to better define and improve the semantics around the ubiquitous GFP_ memory-allocation flags.

• CMA and compaction: problems with and solutions for the kernel's mechanisms for supporting large contiguous allocations.

• Virtual machines as containers, and, in particular, the memory-management challenges that come with packing a lot of virtual machines into a host.

• Partial address-space mirroring is a hardware feature intended to improve reliability. As the discussion showed, though, the memory-management developers were not convinced it will work as well as advertised.

• Heterogeneous memory management: giving GPUs and other peripherals access to process address spaces.

• Memory-management testing: how can we find more problems before they bite users?

• Memory control group fairness: how to get control groups to make the right decisions when faced with problematic workloads.

• TLB flush optimization: reducing the performance cost that comes with flushing the translation lookaside buffer too often.

• Improving the OOM killer: this year's episode in the perennial discussion on how the kernel should handle running out of memory.

• Memory-management subsystem workflow: how is development working in this subsystem, and how can it be made to run more smoothly?

See also: Rik van Riel's notes for a terse summary of the memory-management sessions.

The filesystem-only track had a relatively small number of discussions, which were:

• Parallel lookups: how to support multiple simultaneous directory lookups.

• Network filesystems, and supporting network filesystems with case-insensitive semantics in particular.

• The xstat() system call: how can this long-desired functionality finally make its way into the kernel?

• Exposing extent information to user space: the best way to let applications learn about the layout of a file on disk.

The storage-only track also had a small number of sessions.

• James Bottomley has posted his notes from those discussions.

The combined filesystem and storage track had the following discussions:

• Persistent storage and remote data access protocols: what changes are needed to support accessing fast persistent devices over (relatively) slow remote access protocols.

• Block and filesystem interfaces: better ways for the block and filesystem layers to work together.

• DAX, mmap(), and a "go faster" flag: how to accommodate applications that are aware of persistent-memory behavior and want the best performance possible.

• Partial drive depopulation: what to do when a storage device goes partially bad, but you want to keep using the part that still works?

• fallocate() and the block layer: challenges in supporting the "bulk zero" functionality for block I/O

• Using the multiqueue block subsystem by default: we've had multiqueue block support for a few years now, maybe it's time to start phasing out the single-queue interface?

• Stream IDs and I/O hints: ways of telling block devices which data belongs together.

• Background writeback: how to make it great again, even if it has never been great before.

• Multipage bio_vecs: increasing the maximum size of an I/O operation in the block layer.

Note that these sessions are still being written up; they will be added to this page once they become available.

Group photo

This photo of the LSFMM 2016 group was provided by the Linux Foundation; more photos can be found on Flickr.

Acknowledgments

Thanks are due to LWN subscribers and the Linux Foundation for supporting our travel to this event.

Comments (none posted)

A storage standards update

The opening plenary at the 2016 Linux Storage, Filesystem, and Memory Management Summit (LSFMM) was an update from Fred Knight and Peter Onufryk on some of the relevant storage standards. Knight covered changes that are coming from the T10 (SCSI) and T13 (ATA) technical committees, while Onufryk talked about changes in the NVM Express (NVMe) interface specification for SSDs.

T10 and T13

Knight began with "conglomerates", which is a feature that has been in the T10 standard for a while. It is a way to group logical units. The 64-bit logical unit number (LUN) is split into two pieces: a major number, which identifies the group (conglomerate), and a minor number that identifies the unit within the group. There are new SCSI commands (BIND, UNBIND, ...) to create and manage these conglomerates.

The WRITE ATOMIC and WRITE SCATTER commands were up next. The former will either write all of the data or none of it. Some think that is how SCSI currently works, but if a write returns an error, there is no guarantee of how much data has been written.

WRITE SCATTER allows specifying logical block address (LBA) and length pairs, followed by the data for each segment, and will write the segments. There are no guarantees of atomicity, however, for the entire scatter write. Developers have asked for scatter and atomic to be combined somehow, but that made the storage vendors pull their hair out, Knight said, so the T10 committee said "no way". One thing that could be added that "may be useless" is to guarantee that each individual segment will either be written or not in its entirety. But the order of the writes is not known, so an error will indicate that some or all of the writes were not done.

There were complaints from the audience that some vendors could do atomic scatter writes, but that the large vendors don't want to do so, which is unfortunate. Knight said creating the standard for it is not the hard part—it is getting the vendors to agree to implement it.

Write streams are a way to associate writes so that the data ends up physically "close" on the device. Its origin is in current flash hardware that can be more efficient if writes to the same file end up in the same erase blocks, rather than scattered in many all over the drive. The number of streams is vendor dependent; it is currently up to 255, but there are vendors that want to have much lower limits. There is a reserved byte in the command to potentially expand the stream ID to 16 bits. It is a feature that is focused on supporting current hardware, so it is unclear how long it will stay relevant as flash hardware evolves in coming years, he said.

SCSI universally unique IDs (UUIDs) are assigned to vendors by the IEEE but, with the advent of software-defined storage (SDS), there needs to be a way to generate these IDs. The standard has been updated so that SDS devices can create an ID for their use. This was a feature that came directly from the Linux community, Knight said.

"Hints" for I/O (which are called "logical block markup" in T13) give information to the device about the expected access pattern for the data. The device is not obligated to remember or act on the hints; there are a bunch that are listed in the standards, but there are only two or three that make a difference. One of those is a way to "tag" I/O operations, which can, for example, separate operations for keys, data, and metadata that will help make databases run faster.

There was a question about why both streams and hints were supported, as it seems that they do much the same thing. But Knight said that streams are meant to place the data in the stream physically close on the device, while hints are meant to indicate the access patterns for the data. They can be used together, but hints can be ignored by the device.

There are some new log pages for additional statistics and counters, including counts of misaligned I/Os (e.g. 512-byte writes to a device with a 4KB block size) and compression and deduplication statistics. There is a new application tag mode page for the protection information that is used for data integrity, as well. The application tag can be placed in that page and does not need to be sent with the protection information on each block.

Next up was "depop" (or depopulation), which is a way to remove a portion of a device based on the failure of a head or other element. There are proposals for depop being worked on in T10, T13, and NVMe, Knight said. For today's 10TB or 20TB drives, if some component in the device goes bad, people want to keep using it. Manufacturers are already doing this; if they make a 10TB drive and find that 4TB are bad, they sell it as a 6TB drive to consumers.

Offline depop is effectively just a reformat of the drive for its new capacity. There is work on online depop going on, but there are "some interesting problems to be solved". The organization of logical blocks on a device is such that loss of a head (or a die on an SSD) does not equate to a simple range of LBAs, so the drive will need to report the list of bad LBAs to the host. Knight said that the committees are interested in any input that Linux developers can provide.

Knight closed his portion of the talk by noting that he has some amount of his time available to work with the Linux community to take its ideas to the standards bodies. He said that protection information without an application tag that he mentioned earlier was a suggestion from outside, as was the idea of hybrid drives that are partially regular disks and partially shingled magnetic recording (SMR) drives. He encouraged those interested to give him ideas to take back to the committees.

NVMe

Onufryk introduced the NVMe organization, which consists of several workgroups under a board. The management interface workgroup is concerned with controlling and monitoring NVMe devices. For example, the interface provides information on what devices are available, the temperature of the disks, and so on. Vendors want to be able to use it to update the firmware on the devices as well. There will also be work coming up on enclosure management and LED control, among other things.

James Bottomley noted that the management interface allows changes that happen without any notification to the operating system. He asked if there was a way for Linux to find out what these changes are. That is a "real mess right now", Onufryk said. The first version of the specification just focused on how to control and monitor the drive, but there will be work on how the management interface should integrate with the host. It is a "complex problem", though.

The technical workgroup is working on two big things: NVMe over fabrics and enhancements to NVMe for streams, directives, and virtualization. Fabric support is aimed at scaling NVMe to thousands of drives, which is beyond what is practical with PCIe. The idea is to have a thin encapsulation of the NVMe protocol that will work on fabrics. The first target is remote DMA (RDMA), though work is also being done on Fibre Channel.

There has been a lot of "blood, sweat, and tears" to keep the PCIe and fabric versions aligned. That means device makers should be able to switch out PCIe for RDMA or some other fabric and not change things much. We will "see if it works".

While there is roughly 90% commonality between the PCIe and fabric implementations, there are some differences. The major ones are the identifier used to target a particular device and the way that devices are discovered. The queueing and data transfer are quite similar between the two.

The specification for NVMe over fabrics will likely be accepted by the end of May, at which point it will become public. Host and target drivers for Linux are under development. They will be released as open source when the specification is released.

The first of the NVMe enhancements Onufryk presented was on host-device information exchange. To best use NVM, the host and device need to exchange various kinds of information beyond just the I/O data. That exchange can take place at various points (before, during, or after a data access) and can target various elements (LBA, the physical media or partition, or the access relationship).

One piece of information that can be given to the device from the host was mentioned by Knight earlier: stream IDs. For NVMe, those IDs are 16-bit values in the write commands to identify writes that are associated with each other because they are expected to have the same lifetime. There is little available space in the NVMe command packet so a "directive" type and ID was added that can be used for streams or other hints to be sent with the commands, but only per command.

There are also some virtualization enhancements to NVMe. Today there are emulated hardware devices or paravirtualized drivers, which just puts software in the way, Onufryk said. There is a need to provide direct access to NVMe devices from virtual machines.

The NVMe architecture supports single-root I/O virtualization (SR-IOV), but more is needed. In particular, the new virtualization enhancements will define a standard mechanism to allocate interrupt and queue resources to a virtual NVMe device. There are also additions to control the performance of the virtual devices so that the performance of the entire device can be split up between virtual machines—with each getting a specific portion of that performance.

Those enhancements have all been approved for the next revision of the specification. Someone asked about the status of copy offload for NVMe. Onufryk said that it is not yet being worked on and that thin provisioning is in the same bucket.

Comments (none posted)

Persistent-memory error handling

One of the key advantages of persistent memory is that it is, for lack of a better word, persistent; data stored there will be available for recall in the future, regardless of whether the system has remained up in the meantime. But, like memory in general, persistent memory can fail for a number of reasons and, given the quantities in which it is expected to be deployed, failures are a certainty. How should the operating system and applications deal with errors in persistent memory? One of the first plenary sessions at the 2016 Linux Storage, Filesystem, and Memory-Management Summit, led by Jeff Moyer, took on this question.

Error handling with traditional block storage is relatively easy: an I/O request will fail with an EIO error, and the application, assuming it is prepared, can handle the error in whatever way seems best. But persistent memory looks like memory to the system, and memory errors are handled differently; in particular, they can trigger a low-level machine-check error. Some systems can recover from that machine check, others will be forced to reboot. Either way, the system has to be able to handle the problem.

Time for a bit of terminology that caused some confusion in the session. Jeff was talking in particular about errors in "load" operations — reading from persistent memory using normal CPU instructions. Those were differentiated from "reads," which are file operations performed with a system call like read(). Similarly, "stores" (using memory operations) and "writes" (file operations) are seen differently. Errors with reads and writes can be returned via the normal system call status; errors with loads and stores are a bit more complicated.

In cases where a machine check from a load operation is recoverable, the kernel can simply deliver the error to the application via a SIGBUS signal. But, even there, it became clear that the situation is not entirely simple: Keith Packard noted that the discussion was about load errors, and asked what happens when a store goes wrong. The problem there is that store operations are not usually synchronous, so there will be no immediate indication of an error. Paranoid software can do a flush and a load after every store to ensure that the data has been stored properly; there does not seem to be any better way.

On "less expensive" systems where the machine check is not recoverable, it's entirely possible that the system will end up in a reboot loop where, after each boot, it tries again to access the failing persistent-memory range. This behavior is generally seen as undesirable. As it turns out, even fancier hardware is sometimes subject to non-recoverable machine checks, so something has to be done to ensure reliable operation on all systems.

The ACPI specification includes a mechanism for scrubbing an address range for errors; the UEFI firmware can run it as part of the boot process. Address ranges with errors can be flagged, and the operating system can, once it boots, query that list of ranges with errors and create a bad-block list that it knows must be avoided. When an application tries to access a range with an error via mmap(), the bad pages can be left unmapped and, should the application try to access them, the SIGBUS can be delivered. The scrubbing operation is not necessarily fast, so it would be unsurprising if it didn't run on every boot, but it can be run when errors begin happening.

The solution as described so far, though, only works at the level of pages. The error granularity reported by the hardware can be as fine as a single 64-byte cache line; marking an entire (4KB) page as being bad when only 64 bytes have been truly lost is less than ideal. One way of narrowing things down would be for the application to open the file with the reported data loss and issue a series of 512-byte reads, narrowing the problem down to a single 512-byte "sector." But, Jeff said, that "still seems a little perverse." It would be nice to be able to directly inform an application about exactly what has been lost.

A number of possibilities for providing that information were discussed. Christoph Hellwig suggested that the information provided with the SIGBUS signal could be expanded to include the exact range that was lost. Dan Williams said that the application could read the bad-block list from sysfs, then use the FIEMAP ioctl() operation to figure out which block in the file was bad. That works today, he said, except that the bad-block list is not updated while the system is live. Ted Ts'o said it would be useful to have a new ioctl() command to query the failing byte range directly.

James Bottomley said that the most friendly approach would be to remap the bad block and hide it entirely, only informing applications if data has actually been lost. It was agreed that remapping would work if an error is detected on a write operation, but the real problem is with reads (or, properly, loads) where data is known to have been lost. In that case, applications should not be forced to dig through the bad-block list; there should be a more direct interface. There also needs to be some sort of interface to clear the error (typically remapping the block) so that the given address range becomes usable again.

As the session wound down, a few residual questions came up, but no real decisions were reached. Ted asked whether the problem of non-recoverable machine checks would go away as the hardware improves; Jeff answered that it might, but that doesn't change the real issue of how to convey problems to user space. Ted also asked whether this information should be provided to applications at all — isn't that assuming a fundamental change in application behavior? Ric Wheeler answered that applications that care about data integrity already keep multiple copies of the data; they just need to know where things go wrong.

As has been seen for a while, persistent memory raises a number of questions with regard to how it should be presented to user space. While many of the problems are being solved, it seems likely that persistent memory will be a discussion topic at events like this for some time yet.

Comments (24 posted)

Two transparent huge page cache implementations

The transparent huge pages (THP) mechanism, in the kernel since 2.6.38, allows the system to use huge (typically 2MB) pages without application knowledge or involvement. Huge pages can be a significant performance improvement for a number of workloads. The feature only works with anonymous (application data) pages, though; support for transparent huge pages for the page cache (file data) has never made it into the mainline, despite the fact that the page cache often is the biggest user of memory on the system. There are, however, two working implementations out there; the first task of the memory-management track at the 2016 Linux Storage, Filesystem, and Memory-Management Summit was to try to choose between them.

One of the contenders is Kirill Shutemov's THP-enabled tmpfs patch set. Kirill's work is based on compound pages, a fairly elaborate mechanism for binding individual memory pages into a single, larger page. The other solution is the team-pages patch set from Hugh Dickins. A "team page" can be thought of as a new, arguably lighter-weight way of grouping pages together. Hugh's work has the advantage of having been deployed in production at Google for over a year; that gives developers a relatively high level of confidence that it lacks obscure bugs — always a nice feature in memory-management patches. Even so, it became clear, the choice between the two is far from straightforward.

Differing objectives

Kirill started off by saying that one of his primary goals was the ability for applications to access individual 4KB subpages of a huge page without the need to split the huge page itself. The current anonymous THP implementation does not have that property, but the ability to access smaller subpages is more important in the page-cache setting. Hugh, instead, was more focused on getting things working quickly, meaning that a less intrusive (but not necessarily smaller) implementation was required. Compound pages, he said, are hard to manipulate; they also force a deeper level of integration with various subsystems, including memory control groups, that he would rather avoid. There was some back-and-forth on whether compound pages are truly more complex than team pages, but without anything conclusions being reached.

Both implementations currently work with the tmpfs filesystem, which is a start, but a proper transparent huge page cache implementation needs to work with "real" filesystems as well. Kirill stated that he is working on that objective which, he asserted, will be more easily achieved with compound pages than with the team-pages approach, which is known to be incompatible with ext4 in its current form.

There was some discussion of how smaller files are handled. Team pages are assembled from individual pages, and, thus, should deal with small files reasonably well. A huge page will be allocated (memory availability allowing) at the beginning, but isn't fully charged against the allocating process. If need be, the huge page can be split to return memory to the system. Compound pages seemingly have worse small-file performance at the moment, allocating and charging a huge page from the outset. That has led to some testers reporting out-of-space errors with this patch set.

Andrea Arcangeli, the original author of the THP feature, thought that people were worrying too much about the small-file issue. He said that THP should be seen purely as a performance optimization. Anybody who is concerned about whether huge pages are using too much memory should simply not enable the feature.

Sticking points

Hugh made the assertion that the choice was between one implementation that is working, running on thousands of machines, and popular with its users and another that, he said, is "getting there." The team-pages code, he said, is ready to go in, with the possible exception of review of the ABI aspects — mount options, sysfs features, etc. It quickly became clear that there was no consensus for that in the room, though.

One of the key sticking points first came up at this time. The current THP code uses compound pages; Kirill's work is an extension of that approach. The team-pages mechanism is, instead, entirely new. If it is merged, the kernel will be using different techniques for anonymous and page-cache huge pages, essentially doubling (or worse) the amount of code that must be maintained going forward. Vlastimil Babka asked whether Hugh's patches could be converted to use compound pages; Hugh said that might be possible, but he would rather merge what he has now, then let Kirill do the conversion later if he is interested in doing it.

Another issue is "recovery" — substituting a huge page for a set of small pages at some future time if, for whatever reason, a huge page is not allocated at the outset. Team pages seem to have fewer problems with recovery, especially in the case of small files that grow, since the underlying huge page is allocated at the outset if possible. Failing that, either approach needs to run work in the background to coalesce ("collapse") sets of small pages into huge pages. The team-pages patches currently lack such a mechanism; compound pages appear to be in better shape in that regard.

There was some discussion of details — whether the work should be done in the khugepaged thread or in work queues in the context of the processes owning the pages — but that was peripheral to the main issue. As Mel Gorman put it, this question doesn't affect the decision that was being discussed.

Mel went on to complain that the purpose of the session was to choose between the two implementations, and that the group seemed no closer to that objective. This question has implications beyond just THP; he noted that the team-pages patches are currently in Andrew Morton's -mm tree. They are creating conflicts with other patches, notably his own node-accounting patches. If we are not merging team pages for 4.7, he said, those patches should not be in -mm (and, thus, in linux-next) at this time. Hugh said he really wanted the patches to get some exposure and testing, but that they could be backed out for now if need be.

Kirill said that there is no reason to rush the team-pages patches into the kernel now, just to add compound pages later. That, he said, would just add a bunch of churn. But, Hugh said, it would also give Linux users the opportunity to benefit from this work now, to which Kirill responded that Hugh had held on to the patch set for a year, so there should be no urgency now. The team-pages patches are ready, and have been for six months or so, Hugh said; he went on to say that, while Kirill has done well with the compound-page work, he (Hugh) doesn't know how long it will be until he is confident in that work.

Setting requirements

At about this point, Mel went to the front of the room in an attempt to focus the group and get a decision made. Using the flip chart, he started a list of requirements each approach would have to meet before the room — including the competing developer — would accept it into the mainline. It took a while, but some clear requirements did result.

On the compound-pages side, small files must not waste memory. Mel noted that, when the PowerPC architecture went to a 64KB native page size, the amount of memory required to run a basic system quadrupled. Andrea said that the anonymous THP implementation never allocates huge pages for small virtual memory areas; THP for the page cache should make similar decisions for small files. Kirill said that something like this is supported now via a mount option, but some work needs to be done. Among other things, the mount option should go away; things need to just work without administrative tuning. This requirement was extended to include fast recovery when small files grow into large files — the system should be able to swap in huge pages in short order.

Hugh also suggested that compound pages need to demonstrate a high level of robustness before it could be considered. This requirement was seen as being somewhat unfair, though: it is easy to show a lack of robustness, but difficult to show its presence. In the end, it will be incumbent on Hugh to show which robustness problems, if any, exist in the compound-pages implementation.

One of the requirements for team pages is similar: it has to have a recovery mechanism for files that didn't get huge pages assigned initially. In particular, khugepaged or something like it must be able to collapse pages when appropriate.

The harder requirement, though, is to move away from having independent mechanisms for anonymous and page-cache pages. If the team pages approach is to be adopted, there must be a plausible commitment to implement team pages for anonymous pages as well. If, once the implementation is in place, it is shown (in the form of performance problems, for example) that the problems are sufficiently different that two approaches are necessary, the two can remain separate. But until such a thing has been conclusively demonstrated, the goal needs to be a single approach for both cases. There is a lot of concern about excessive complexity in the memory-management code; few people want to add to it.

Finally, the current incompatibility between team pages and non-tmpfs filesystems (the ext4 filesystem in particular) needs to be resolved. In practice that means that team pages must stop using the PagePrivate flag, since ext4 (along with other filesystems) is already using it.

The session concluded with both Kirill and Hugh agreeing that, if the other developer's system met the requirements, they would not block its merging. Hugh also agreed that team pages would come out of the -mm tree for now, since it is not destined for merging in 4.7. What will be merged in subsequent cycles remains to be seen; it would not be entirely surprising if it were a topic of discussion again at LSFMM 2017.

Comments (none posted)

Ideas for rationalizing GFP flags

The kernel's memory-allocation functions normally take as an argument a set of flags describing how the allocation is to be performed. These "GFP flags" (for "get free page") control both the placement of the allocated memory and the techniques the kernel can use to make memory available if need be. For some time, developers have been saying that these flags need to be rethought; in two separate sessions at the 2016 Linux Storage, Filesystem, and Memory-Management Summit, Michal Hocko explored ways of doing that.

GFP_REPEAT

The first session, in the memory-management track, started with a discussion of the GFP_REPEAT flag which, as its name would suggest, is meant to tell the allocator to retry an attempt should it fail the first time. This flag, Michal said, has never been useful. It is generally used for order-0 (single-page) allocations, but those allocations are not allowed to fail and, thus, will retry indefinitely anyway. For larger requests, he said, it "pretends to try harder," but does not actually do anything beneficial. Michal would like to clean this flag up and create a better-defined set of semantics for it.

The kernel does have the opposite flag in the form of GFP_NORETRY, but that one, he said, is not useful for anything outside of order-0 allocations. What he would like to see instead is something he called GFP_BESTEFFORT; it would try hard to satisfy the request, but would not try indefinitely. So it could retry a failed request, and even invoke the out-of-memory killer but, should that prove fruitless, it would give up. This flag would be meant to work for all sizes of requests.

He is trying to move things in that direction, starting with the removal of GFP_REPEAT from order-0 allocation requests around the kernel. The next step would be to start placing the new flag in the places where it makes sense. As an example, he mentioned transparent huge pages and the hugetlbfs filesystem. Both need to allocate huge pages but, while an allocation failure for a transparent huge page is just a missed optimization opportunity, a failure in hugetlbfs is a hard failure that will be passed back to user space. It clearly makes sense to try harder for hugetlbfs allocations.

Johannes Weiner asked whether it would be a good idea to provide best-effort semantics by default while explicitly annotating the exceptions where it is not wanted. The existing GFP_NORETRY flag could be used for that purpose. Michal said that doing so would cause performance regressions, leading Andrew Morton to question whether "taking longer but succeeding" constitutes a regression. The point is that some callers do have reasonable fallback paths for failed allocations and would rather see the failures happen quickly if they are going to. Andrew asked how often that sort of failure happens, but nobody appeared to have any sort of answer to that question. It will be, in any case, highly workload-dependent.

Johannes persisted, saying that it can be difficult to know where the memory allocator should be told to try harder, but it is usually easy to see the places where failure can be handled easily. There was also a suggestion to make the flags more fine-grained; rather than use a vague "best effort" flag, have flags to specify that retries should not be done, or that the out-of-memory killer should not be invoked. Mel Gorman noted that he has already done some work in that direction, adding flags to control how reclaim should be performed.

That led to a wandering discussion on whether the flags should be positive ("perform direct reclaim") or negative ("no direct reclaim"). Positive flags are more descriptive, but they are a bit more awkward to use since call sites will have to mask them out of combined mask sets like GFP_KERNEL. There are also concerns that there aren't many flag bits available for fine-grained control.

The session ended with Michal asking if the group could at least come to a consensus that his work cleaning up GFP_REPEAT made sense. There seemed to be no objection there, so that work can be expected to continue.

GFP_NOFS

Later that day, the entire LSFMM group was present while Michal talked about a different GFP flag: GFP_NOFS. This flag instructs the memory allocator to avoid actions that involve calling into filesystem code — writing out dirty pages to files, for example. It exists for use by filesystem code for a number of reasons, the most straightforward of which is the avoidance of deadlocks. If a filesystem acquires locks then discovers that it must allocate memory, it doesn't want the allocator coming back and trying to obtain the same locks. But there is more to it than that; GFP_NOFS reflects a number of "indirect dependencies" within the filesystems. Also, XFS uses it for all page-cache allocations, regardless of deadlock concerns, to avoid calling so deeply into filesystem code that the kernel stack overflows.

There are, Michal said, too many uses of GFP_NOFS in the kernel tree; they needlessly constrain the memory allocator's behavior, making memory harder to obtain than it should be. So he would like to clean them up, but, he acknowledged, that will not be easy. The reason for any given use of GFP_NOFS is often far from clear — if there is one at all.

His suggestion is to get rid of direct use of that flag entirely; instead, setting a new task flag would indicate that that current task could not call back into filesystem code. XFS has a similar mechanism internally now; it could be pulled up and used in the memory-management layer. A call to a function like nofs_store() would set the flag; all subsequent memory allocations would implicitly have GFP_NOFS set until the flag was cleared.

There are a number of reasons for preferring this mechanism. Each call to nofs_store() would be expected to include documentation describing why it's needed. It allows the "no filesystem calls" state to follow the task's execution into places — security modules, for example — that have no knowledge of that state. Chris Mason noted that it would save filesystem developers from sysfs, which brings surprises of its own. Ted Ts'o added that there are a number of places where code called from ext4 should be using GFP_NOFS for its allocations, but that doesn't happen because it would simply be too much work to push the GFP flags through the intervening layers. Thus far, he has been crossing his fingers and hoping that nothing goes wrong; this mechanism would be more robust.

Michal asked the filesystem developers in the room how much work it would be to get rid of the GFP_NOFS call sites. Chris said that the default in Btrfs has been to use it everywhere; a bunch of those sites have since been fixed, but quite a few remain. He would be happy to switch to the new API, he said. Ted agreed, as long as the transition would be gradual and GFP_NOFS would not disappear in a flag day, as it were. The end result, he said, would be nice.

There was some talk of refining the mechanism to specify the specific filesystem that should be avoided, allowing the memory allocator to call into other filesystems. The consensus seemed to be that this idea would be tricky to implement; the possibility of stack overruns was also raised. Michal will go ahead and put together an API proposal for review. He hopes it will succeed: the fewer GFP_NOFS sites there are, the better the memory allocator's behavior will be.

Comments (13 posted)

Persistent memory as remote storage

In a combined storage and filesystem session at the 2016 Linux Storage, Filesystem, and Memory Management Summit (LSFMM), Chuck Lever talked about using remote DMA (RDMA) for access to persistent memory. But, more generally, he was "soliciting feedback and rotten tomatoes" about what changes might be needed to make the various protocols and persistent storage classes/types work well together.

While he mostly discussed RDMA, Lever said that many of the same issues apply to other protocols, such as iSER and SRP at the block layer and SMB Direct and NFS/RDMA at the file layer. The performance equation for those protocols and fast persistent devices is such that the cost of making data durable may be less than that of the I/O to get the data to the device.

So, Lever asked, why marry slow technology to this new fast technology? Data replication for disaster recovery is one particularly good use case. It can be set up so that there are geographically diverse failure domains so that the data will be available for recovery. There are other use cases as well.

Today, Linux uses a "pull mode" to do I/O to remote targets, where the initiator exposes a region of its memory to the RDMA controller and sends a request to the target, which then uses that memory to complete the request. Once the initiator receives a reply, it invalidates the memory it exposed so it can no longer be accessed. For a read, the target simply places the data into the initiator's memory using an RDMA write. But for a write request, the target must do an RDMA read to the initiator to get the data to be written and await the response before it can write it. That means there is an additional round-trip for writes.

There are some advantages to pull mode, Lever said, including good memory security, since the initiator only exposes small amounts of memory and only for the duration of the request. In addition, the work to do the transfer is moved to the target side, leaving the CPU on the initiator available for application work. There are several downsides too, however. There is more than one interrupt for each request and the extra round-trip for write requests. In addition, the target CPU has to be involved in all requests.

The NFS server on Linux does not have zero-copy write—except for small I/O operations, as Christoph Hellwig pointed out. Lever said that RDMA could perhaps do zero-copy writes to get better performance. He asked: should splice() be used to do so? Hellwig replied that "splice() is really nicely over-hyped" and doesn't really help this kind of problem. He suggested that any I/O for a fast device should be using direct I/O to avoid the page cache.

For the future, Lever wondered about switching to a "push mode" instead. The initiator would register its interest in regions of a file and the target would expose memory for the initiator to use for read and write operations on those regions. It would return handles to the regions for the initiator to use; multiple RDMA read and write operations could be performed by the initiator before it informed the target that it was done. At that point, the handles would be invalidated (and the memory no longer exposed).

Ted Ts'o asked what the "security story" was for push mode. Lever replied that it uses "reliable connections" where there are only two peers. That connection is set up so that one side can view the other's memory based on the handles. Those handles are only valid for a single connection and the hardware guarantees that other endpoints can't interfere with the connection.

One problem is that there is no generic way to ensure that writes have reached durable storage for the remote storage protocols. Each operating system, network/fabric, and device has different durability guarantees and its own way to ensure that a remote write is stored safely. Sagi Grimberg suggested that code to ensure durability could be written once for all the different options and made available as a library, something like what DAX has. There was general agreement that there should be an API made available that hides the differences.

Comments (none posted)

Maru: a pocket desktop

It appears to be widely accepted that the Linux desktop has achieved limited success at best, while the Linux palmtop — in the form of Android — has been wildly successful. The two classes of systems are generally thought of as being quite different, but it is worth remembering that the handsets we carry now have more computing power than the desktop systems we were using in the recent past. Given the right peripherals, an Android handset should be more than capable of providing a reasonable desktop experience. The Maru distribution is an experiment intended to prove that point by turning a smartphone device into a portable Debian desktop.

Maru, in its current state, is clearly not for everybody; it is currently running in a "private beta" and only works on the Nexus 5 handset. As luck would have it, your editor is in possession of both the hardware and the patience to wait until his number came up for a beta invitation. Playing with this system shows that there are some issues to deal with yet, but the idea has a lot of potential.

Installing Maru will be a familiar process to anybody who has flashed an Android device. The bootloader must be unlocked, and the owner needs to be cognizant of the fact that all data on the device will be overwritten; then it is just a matter of a couple of fastboot flash commands. Or it should be: that resulted in a phone that got stuck looping through the interminable "optimizing apps" process — amusing at first, but even your editor, who is easily amused, found it tiresome by about the fourth time through. Explicitly wiping the data area (using the TWRP recovery image) and starting over got past that issue.

The installation results in a standard — if severely minimal — Android "Lollipop" system; Maru has not made the move to the Marshmallow release. As is always the case with a third-party build, the Google apps are not there; they can be added from the Open GApps repository if desired — though it is somewhat assumed that the user knows how to do this. There are reports that only the "pico" version of Open GApps will actually fit on the device after the Maru installation; the 32GB device used here still had 26GB of free space, though, and was not so constrained.

Experiencing Maru as anything but an Android device requires some additional hardware in the form of a Bluetooth keyboard, a Bluetooth mouse, and a USB-to-HDMI dongle. The latter device plugs into the phone's USB port and allows the handset to drive an ordinary monitor (or television); the documentation advises buying one that allows for a power connection, since running in the desktop mode can drain the battery quickly.

Once the Bluetooth devices have been paired and the monitor connected, the user is presented with an attractive XFCE desktop (for those who, inexplicably, think that translucent terminal emulators are attractive, anyway). At least, that happens briefly. One of the first discoveries is that the desktop is only active when the handset screen is on, and desktop activity does not keep the handset from timing out and shutting down. So working on the desktop happens in 30-second bursts, separated by (hopefully shorter) bursts of profanity. Making the system usable requires setting the timeout value to a high number, which may not be what one wants when the desktop is not in use. This, perhaps, is part of why running the desktop tends to drain the battery quickly.

With the external keyboard and mouse, the desktop works as one would expect. A lot of the basic utilities are present, and some not-so-basic ones (e.g. LibreOffice) as well. Anything else can be had in the usual manner from the Debian repository. The desktop is surprisingly responsive; even LibreOffice is able to launch within a few seconds. Building kernels is probably a poor idea, but the device clearly has the power to handle most day-to-day tasks.

There is no access to Android apps in the Debian desktop, of course, and the desktop never appears on the handset screen. The two are separate worlds living on the same device. The one exception is that the Android data area has been mapped into the Debian world, so that, for example, photos taken with the camera can be accessed on the Debian side.

If one pokes around with a shell, what one finds is an apparently complete Debian distribution running; it even has systemd running the show. The system has been thoroughly isolated from the Android system (some reports on the net say that LXC containers are being used). Except for the access to Android's data area, the Android side of things might as well not be there from Debian's point of view. The Debian system does run on the Android kernel, though (it's not in a virtual machine), so an enterprising attacker could probably find a way to break through that barrier.

Some parts of the system are not entirely at ease with this setup. Systemd, for example, duly logs that the time has been changed — every second. But as a whole it seems to work well.

Anybody looking for the source to Maru will, for now, be disappointed; it does not appear to be available. That might strike some people as surprising, given that it is, in the end, a distribution of Debian, much of which is covered by copyleft licenses. Back in February, Maru creator Preetam D’Souza announced that the distribution would be open-sourced, almost as if it were a new idea. Since then, though, there has been no visible progress on that front. Maru is, thus, quite clearly violating the GPL at the moment. One gets the impression of a small (i.e. one-person) project struggling to cope with a larger-than-expected level of interest. One of the best ways to do that, though, would be to get the source out there and let the community help. Hopefully this issue will be straightened out in the near future.

One might be tempted to ask where a distribution like this might be useful. The advantage of a smartphone is its portability; lugging around a keyboard, mouse, HDMI adapter, and, possibly, monitor tends to detract a bit from that. It is probably easier to just pack a laptop and be done with it. But if the peripherals are available in multiple locations, and all that needs to be carried is the handset, the appeal becomes a bit more clear.

What would be nice, of course, would be a higher degree of integration between Android and Debian, making the device a bit less schizophrenic than it is now. It would be useful, for example, if applications from either side could appear on either the device screen or the external desktop. That is asking a lot, though, considering just how foreign much of the Android system is relative to a typical Linux desktop distribution. But, perhaps, if the two cohabitate for long enough, they will eventually learn to trust each other a bit more. In the meantime, Maru is an interesting experiment in running a Linux desktop in an Android setting.

Comments (19 posted)

Distribution quotes of the week

-- Jesse Smith

Matthew Miller (NetworkWorld interview)

-- Ian Jackson

Comments (46 posted)

Introducing open source DC/OS

Mesosphere has announced the release of DC/OS under the Apache License 2.0. "DC/OS derives from Mesosphere’s Datacenter Operating System, a commercial product built around Apache Mesos. Open sourcing DCOS has always been part of our strategic roadmap and we’re proud to have collaborated with our launch partners for today’s unveiling. DC/OS is a software platform that’s 100 percent open source, comprised of more than 30 component technologies, including Apache Mesos and Marathon. Some of the technologies were always open source, including Mesos, while others were previously proprietary code developed by Mesosphere, such as the GUI and our Minuteman load balancer." Over 60 partner companies participated in the open source release.

Comments (33 posted)

Final bits from the (outgoing) DPL

Neil McGovern has a few parting words and delegations at the end of his term as Debian Project Leader. He looks at the treasurer team re-delegation, front desk de-delegation, DSA spending limits update, a new technical committee member, and more.

Full Story (comments: 1)

Debian Project Leader Election 2016 Results

Unsurprisingly, Mehdi Dogguy is the new Debian Project Leader. His term started April 17.

Full Story (comments: none)

Debian Bug Squashing Party in Salzburg/Austria

There will be a Debian Bug Squashing Party September 23-25 in Salzburg, Austria. "Team meetings/sprints during the BSP are welcome, just let me know in advance so we can organize appropriate rooms."

Full Story (comments: none)

Red Hat Enterprise Linux 5 One-Year Retirement Notice

Red Hat has announced that RHEL 5 will be retired in one year. "In accordance with the Red Hat Enterprise Linux Errata Support Policy, support for Red Hat Enterprise Linux 5 will be retired on March 31, 2017, at the end of Production Phase 3. Until that date, customers will continue to receive Critical impact security patches and selected urgent priority bug fixes for RHEL 5.11 (the final RHEL 5 release)."

Full Story (comments: none)

Distribution newsletters

• DistroWatch Weekly, Issue 657 (April 18)
• 5 things in Fedora this week (April 19)
• Lunar Linux weekly news (April 15)
• openSUSE Tumbleweed – Review of the Weeks (April 15)
• Ubuntu Kernel Team - Weekly Newsletter (April 12)
• Ubuntu Weekly Newsletter, Issue 462 (April 17)

Comments (none posted)

Schaller: Fedora Workstation Phase 1 – Homestretch

Christian Schaller celebrates the completion of the (informal) first phase of the Fedora Workstation project. "Another major piece of engineering that is coming to a close is moving major applications such as Firefox, LibreOffice and Eclipse to GTK3. This was needed both to get these applications able to run natively on Wayland, but it also enabled us to make them work nicely for HiDPI. This has also played out into how GTK3 have positioned itself which to be a toolkit dedicated to pushing the Linux desktop forward and helping that quickly adapt and adopt to changes in the technology landscape."

Comments (47 posted)

LXLE Gives Aging Hardware a New Lease on Life (LinuxInsider)

LinuxInsider reviews LXLE. "However, LXLE -- short for Lubuntu Extra Life Extension -- taps fewer system resources than Xfce, making it just as fast and a bit more compatible with aging hardware. It features security fixes, updated SeaMonkey (the default Web browser) packages and various small visual improvements. It is an easy-to-use alternative to other Linux derivatives. LXLE comes with an impressive selection of default applications. It adds useful modifications and tweaks to improve performance and functions, and has several design changes and tweaks that make it even more suitable."

Comments (none posted)

OCaml 4.03

The next version of the OCaml programming language, 4.03, has had a few beta versions already, and should be released in the following weeks. A major theme of this new release is to help users improve the efficiency of their programs by making changes to the language, compiler implementation, and tooling. It also contains a large variety of other changes that demonstrate the accelerating development of the language implementation over the past few years, which is caused in part by the steady increase in the number of external contributors. Some promising experimental projects that are still in development suggest that future releases may retain this accelerated pace.

A quick tour of OCaml

OCaml is a typed functional language of the ML family, close to Standard ML or F# and, to a lesser extent, Haskell. Functional programming emphasizes first-class functions, immutable data, and making it easier for users to reason about their programs. OCaml is a mature language with a robust implementation; it is garbage-collected and has a native compiler and efficient runtime resulting in reasonably fast programs, as well as separate compilation and fast compile times. Idiomatic OCaml programs tend to be shorter but slower than optimized C or C++ code, but in the same ballpark as the faster languages implemented on the Java Virtual Machine (JVM).

Here is a taste of OCaml code:

    (* Binary tree with leaves carrying an integer. *)
    type tree =
      | Leaf of int
      | Node of tree * tree

    let rec exists_leaf test tree =
      match tree with
      | Leaf v -> test v
      | Node (left, right) ->
          exists_leaf test left || exists_leaf test right

    let has_even_leaf tree =
      exists_leaf (fun n -> n mod 2 = 0) tree

The values of type tree are either the Leaf constructor followed by an integer or a Node constructor followed by two sub-trees. This is an instance of an algebraic data type, which is a form of discriminated union common in functional programming.

The function exists_leaf takes two arguments, a test function and a tree, and returns true if one of the leaves of the tree satisfies the test function. Pattern matching (match tree with ...) allows inspecting the value of an algebraic data type; it performs a conditional test on the constructor (Leaf or Node), and gives names to its parameters (v, left, right).

The general function exists_leaf is then used to define the more specific has_even_leaf function; in this definition, an anonymous function (fun n ...) is passed as a test function.

Algebraic data types and pattern-matching make it easy to write programs that manipulate symbolic data: interpreters, compilers, static analyzers, theorem provers, model checkers, etc. Another strength of ML languages is their handling of generic, polymorphic code. Instead of a type tree whose leaves contain integers, one can define a parameterized type 'a tree (in Java notation, that would be Tree<A>) whose leaves contain values of type 'a:

    type 'a tree =
      | Leaf of 'a
      | Node of 'a tree * 'a tree

The rest of the code does not have to be changed at all; the function exists_leaf becomes a generic function that works on trees of any type, without requiring extra annotations..

OCaml has never seen large adoption in the world of enterprise software, and as such remains lesser-known than other mainstream programming languages. It has, however, kept and grown an active community of users since the 1990s, with lots of code released as free software, visible industrial users, and widespread adoption in some domains of computer science academia. LWN readers may be familiar with the OCaml Weekly News, the weekly summary of activity in the OCaml community that was launched by David Mentré in March 2001 (as Caml Weekly News) and has been edited by Alan Schmitt since late 2001.

OCaml is the implementation language of Coccinelle, a "semantic patching" tool for C that is regularly used in the Linux kernel community and frequently mentioned on LWN (for example in 2009, 2010, and 2014). Some other notable open-source projects are the Coq proof assistant, the MLdonkey peer-to-peer client, and the MirageOS library operating system for constructing unikernels — its supporting company, Unikernel Systems, was acquired by Docker in January 2016.

One strong niche of typed functional languages is compiler implementations; OCaml is used for the OCaml compiler itself, but also for the Haxe compiler, the old bootstrap compiler for Rust, the Hack language implementation at Facebook, and the reference interpreter of the WebAssembly project. OCaml is also used in several open-source static analysis and program verification projects, such as the Frama-C static analyzer for C programs, the Infer memory-safety analyzer for Java and Objective-C programs, the Why3 program verification platform, and the Flow type analyzer for JavaScript. Some of these projects rely on Js_of_ocaml, which is an excellent compiler from OCaml to JavaScript, to easily deploy OCaml code on the web.

OCaml project development

Like Java or Python, but unlike C or Ada, the OCaml programming language has a dominant implementation, and the user-exposed language evolves along with this implementation. This implementation, born in 1995, was developed inside a programming language research group at INRIA, a French public research institute. Several of these initial developers, in particular Xavier Leroy and Damien Doligez, are still actively maintaining the language. They have since been joined by other maintainers from academia or industry. There are currently 13 contributors with commit access — but none of them work full-time on the OCaml implementation. As a frequent contributor, I have commit access, but am not an expert or maintainer of any specific part of the compiler. I mostly help fixing bugs and by reviewing and integrating patches proposed by external contributors.

In the last few years, several changes were made to encourage external contributions to the compiler distribution. Both bug fixes and patch proposals were previously handled on an austere Mantis issue tracker. In January 2014, the project started accepting pull requests against the non-official GitHub mirror of the reference Subversion repository and, in October 2015, this GitHub repository became the official source repository of the project. Compiler hacking sessions were also organized, notably by Jeremy Yallop and Leo White in Cambridge UK, to encourage people already familiar with the language to jump in and make their first contribution to the compiler implementation.

These various efforts proved successful, as the number of external contributions to the compiler distribution code base has noticeably increased. Mantis never received more than a few dozen patches a year, but 130 GitHub pull requests were submitted in 2014, 254 in 2015, and there have already been 149 pull requests in 2016.

Having many changes proposed by external contributors can be a curse as much as a blessing, if the bandwidth of available maintainers is too restricted to effectively evaluate, provide feedback, and make decisions on the proposals. Of the 535 pull requests so far, only 82 are still open, which shows that the project is reasonably effective at managing those contributions.

While the evolution of the compiler implementation and tooling is relatively fast, the language designers have remained fairly conservative in accepting language changes. Strict compatibility requirements are expected; breaking existing user programs is almost never accepted. Tastefully evolving a programming language is a difficult craft, and some users legitimately complain that the changes are too slow — or too fast. OCaml users appreciate the language compromise between expressivity and simplicity; each new advanced feature risks moving the language outside this comfort zone.

Compiler optimizations

The OCaml compiler distribution has historically remained a relatively simple project. In particular, the compiler did not perform aggressive optimizations. Good performance for the generated program came instead from a few high-impact optimizations and a choice of data representation and garbage-collection strategies tailored to typical OCaml programs. Those programs generally have a high allocation rate of short-lived data structures, more immutable than mutable data, and more symbolic data traversal and transformation than numeric computations.

The relative simplicity of the compiler has been changing under the pressure of users moving into new problem domains, in particular numeric computations.

To achieve separate compilation of highly polymorphic functions, OCaml uses a uniform data representation where each value fits in one word of memory. Immediate values such as integers, booleans, or constructors of an algebraic data type that have no parameters, are represented as "tagged" words, with their lowest bit set — so that the garbage collector knows they are not pointers. All other values, such as records/structures, double-precision floating-point numbers, or constructors with parameters, are "boxed": allocated on the heap and represented by a pointer.

Tagging and boxing impose a runtime overhead on numeric operations; inside a function declaration, the compiler will remove redundant boxing and unboxing operations, so that intermediate computations do not perform (un)boxing or (un)tagging. The choice of which (un)boxing operations to insert and remove is called an "unboxing strategy". Inlining can significantly increase performance, as it widens the scope of these local optimizations.

The main change for the 4.03 release is the addition of a new intermediate representation to the compiler, called Flambda. It is designed to support more aggressive inlining and unboxing strategies, and more optimizations than were previously feasible.

This new Flambda middle-end is the result of a multi-year effort by Pierre Chambart; reviewing it and integrating it into the compiler code base was itself massive and has been led by Mark Shinwell and Leo White. The results are promising but there is still a margin for improvement. In particular, compilation times are noticeably degraded (generally twice slower; the OCaml compiler is fast, which makes it a difficult target to catch). It is also currently impossible to link code that uses Flambda with code that doesn't, as the format of compiled object files differs. For this reason, this new compilation pass is not enabled by default in 4.03. Hopefully, the forthcoming iterations of this work will reduce compilation time and implement new optimizations, making Flambda the better choice for everyone.

The tragedy of optimization writers is that performance-sensitive code that already exists in the wild is often written in a style that specifically caters to the limitations of the existing optimizer, so it will not see much improvement from new optimization passes. The hope is that new optimizations will make it easier to write high-performance code by enabling useful idioms that were previously judged too costly.

Stronger inlining capabilities are particularly useful in the context of a general push in the functional programming community toward deeper layering of abstractions and, in particular, a programming style. Monads let users implement libraries for many programming notions (mutable state, backtracking, cooperative concurrency, exceptions, continuations,...) that are traditionally either directly part of the language implementation, or not supported at all. This can be very flexible and expressive, but it also requires aggressive optimizations to be competitive performance-wise with direct language support for the same features. In particular, monads make heavy use of series of short-lived anonymous functions or data structures. GHC, the dominant implementation of the Haskell programming language, has long needed these optimizations to be competitive on performance because of a different idiomatic style relying more heavily on monads and a lazy evaluation strategy that tends to incur more bookkeeping.

In parallel to the Flambda work, Alain Frisch has long been working on improving unboxing strategies, in particular for floating-point code. Again, the idea is that more flexible unboxing strategies make it easier for programmers to write well-performing floating-point code without having to know all the low-level tricks to write code that is unboxed well. Several interesting changes from Frisch and Vladimir Brankov have been merged during 4.03; more unboxing work has already been merged in the development branch that will become the future 4.04 release. For more details, Frisch has a blog post that explains the general issues of OCaml value representation and the details of the past, present, and future of unboxing strategies. More unboxing ideas can be found in a blog post by Andy Wingo about Guile, a GNU implementation of the dynamically typed functional language Scheme.

Tools for performance reasoning

Another important theme in the 4.x series is the addition of first-class support of attributes and extensions that is being led by Frisch. Attributes are part of the standard syntax that do not directly affect the program semantics, but are handled by external tools and preprocessors to analyze and transform the code, not unlike Java annotations. Extensions are quoted program fragments that cannot be interpreted directly, but must be translated by compile-time preprocessors into valid code. Some attributes (and extensions) are handled by the compiler itself, so they serve as compiler pragmas; 4.03 supports new attributes intended to let users better control performance of their applications.

During one of the compiler hacking sessions, Simon Cruanes contributed a new [@tailcall] attribute that expresses the intent that a given function call should benefit from tail-call optimization; it results in a warning if the call is in fact not optimized. Because programmers reason about tail-calls to analyze the stack-consumption behavior of their programs, it is important to let them check that their assumptions are correct.

Jérémie Dimino implemented [@unboxed] and [@untagged] attributes to be used in the Foreign Function Interface, which lets the compiler unbox or untag OCaml data representations before passing them to C functions, instead of doing the transformation manually on the C side. This can result in further optimizations, as other boxing or tagging operations may cancel out.

Finally, the Flambda pipeline comes with an [@inline] attribute for a function declaration to require that calls to the function be inlined and an [@inlined] attribute to check that a particular call site is inlined. This should also help programmers reason about compiler optimizations, which is a necessary evil when the optimizer becomes more aggressive and correspondingly less predictable.

Garbage-collection improvements

A final important area of change is the garbage collector (GC) implementation. This is unusual in the OCaml world as the GC code has been performing well and has been fairly stable since its implementation by Doligez in the early 1990s. It is a precise, generational, incremental garbage collector. For more details on the OCaml garbage collector, see this online book chapter.

Doligez worked on improving the worst-case latency for the workloads of Jane Street, which is a finance company relying heavily on OCaml. The timing of this work happens to coincide with work in the Haskell world to decrease garbage-collector pauses for Facebook workloads. Some younger languages like Go tend to have invasive changes to their GC implementation on each new release, but OCaml and Haskell implementations have traditionally remained fairly stable. But performance-sensitive industrial applications can still suggest further changes and industrial users have the funding and workforce to get these delicate changes implemented and accepted upstream.

François Bobot contributed support for ephemerons, which are a generalization of the weak pointers already supported by the OCaml GC. Weak pointers are references that are not counted by the GC as implying that the memory pointed to is still alive. They are useful when caching results of computation: one may not want the presence of an object in the cache to prevent it from being collected if nobody else is using it. However, weak pointers do not suffice to implement associative maps where both keys and their associated values should be held weakly; see this short PDF abstract by Bobot for more details and the Wikipedia ephemeron article for information about support in other languages.

Other changes

The detailed 4.03 release Changelog contains slightly more than 250 change items, which are mostly smaller, unrelated fixes, improvements, or features. We cannot list all or even most of them, but will mention three more notable changes brought by this release.

The first change is inline records, implemented by Frisch with help from Jacques Garrigue, which provide a way to give field names to the parameters of discriminated union constructors. This feature brings usability benefits — names are good for readability — but also had performance justification, as it sometimes allows for more compact data representation than was previously possible. It is also the result of a discussion spanning two and a half years and hundreds of messages: language design through open discussion can be taxing.

The second change is the work of David Allsopp to provide a wrapping of Windows native support for symbolic links, as part of a general push by Allsopp to make some currently Unix-specific OCaml applications to work better on Windows. It is possible to write portable applications in OCaml, but the Unix-using majority of developers have a hard time resisting the convenience of symbolic links or fork() calls.

The third change is a series of improvements by Florian Angeletti to the official reference manual that clarify the behavior of some more advanced features. Most languages' reference manuals or specifications tend to be exhaustive but rather terse, and user communities often fall back to tutorials, books, or blog posts to exchange more beginner-friendly information. Contributors motivated to improve the centralized reference documentation and make it more accessible are an invaluable resource.

Future releases

Major OCaml releases have traditionally occurred approximately every year and half, with some number of minor bug-fix releases in between. With the increased pace of change, this rhythm is becoming more stressful for Doligez, who is the release manager, as each release has more changes and increased risk of regressions. It is also frustrating for the external contributors who want others to be able to use their changes without waiting too long. Finally, it creates a perverse incentive to rush adding invasive changes shortly before the feature freeze, a few months before each release, to avoid waiting for another full release cycle.

The current plan for the coming releases is to move to a shorter release cycle — closer to a bi-annual major release. This requires a significant change in development practices, so it may take a few more releases to come into effect. But one could theoretically hope for a 4.04 release in late 2016.

Comments (4 posted)

Quotes of the week

— Fridrich Strba of the Document Liberation Project at Libre Graphics Meeting, explaining the project's philosophy.

— Peter Hutterer (Thanks to Bastien Nocera.)

Comments (3 posted)

MicroPython 1.7 is available

Version 1.7 of MicroPython, the Python implementation for microcontrollers, is available. New features include a cross-compiler that can generate bytecode executable on any MicroPython runtime, a "vastly improved" ESP8266 port with a new event loop, many additional extensions, and greatly improved networking support.

Full Story (comments: none)

Libgcrypt 1.7.0 released

Version 1.7.0 of the libgcrypt library has been released. The new release adds several new algorithms and modes, including the SHA3-224, SHA3-256, SHA3-384, SHA3-512, and MD2 hash algorithms, the ChaCha20 stream cipher, and the Poly1305 message authentication algorithm. Several new curves are now supported for elliptic curve cryptography (ECC), including Curve25519, sec256k1, GOST R 34.10-2001, and GOST R 34.10-2012. Quite a few performance improvements have been made as well, and enhancements have been made to several other cryptographic primitives.

Full Story (comments: none)

Clementine 1.3 released

Version 1.3 of the Clementine audio player has been released. Major new features include compatibility with the Ampache streaming-audio server, support for DRM-free .m4b (audiobook) files, and support for file storage using the Seafile cloud service. Many more features are listed in the Changelog, including user-interface improvements and speed increases for many operations.

Comments (none posted)

Firebird 3.0 is available

Version 3.0 of the Firebird SQL database server has been released. The release notes highlight several changes, including SMP support, new object-oriented APIs for C++, per-database configuration options, and support for IPv6. Also added are new tools for user management, including support for multiple access-control databases.

Comments (none posted)

Development newsletters from the past week

• What's cooking in git.git (April 18)
• Git Rev News (April 20)
• OCaml Weekly News (April 19)
• Perl Weekly (April 18)
• PostgreSQL Weekly News (April 17)
• Python Weekly (April 14)
• Ruby Weekly (April 14)
• Tahoe-LAFS Weekly News (April 19)
• Wikimedia Tech News (April 18)

Comments (none posted)

Costa: Designing a Userspace Disk I/O Scheduler for Modern Datastores: the Scylla example (Part 1)

Over at the Scylla blog, Glauber Costa looks at why a high-performance datastore application might want to do its own I/O scheduling. "If one is using a threaded approach for managing I/O, a thread can be assigned to a different priority group by tools such as ionice. However, ionice only allows us to choose between general concepts like real-time, best-effort and idle. And while Linux will try to preserve fairness among the different actors, that doesn’t allow any fine tuning to take place. Dividing bandwidth among users is a common task in network processing, but it is usually not possible with disk I/O without resorting to infrastructure like cgroups. More importantly, modern designs like the Seastar framework used by Scylla to build its infrastructure may stay away from threads in favor of a thread-per-core design in the search for better scalability. In the light of these considerations, can a userspace application like Scylla somehow guarantee that all actors are served according to the priorities we would want them to obey?"

Comments (11 posted)

Brauch: Processing scientific data in Python and numpy, but doing it fast

On his blog, Sven Brauch has some suggestions on how to use NumPy to process scientific data and how to avoid some pitfalls that will ruin its performance. "In general, copying data is cheap. But if your program simulates 25 million particles, each having a float64 location in 3d, you already have 8*3*25e6 = 600 MB of data. Thus, if you write r = r + v*dt, you will copy 1.2 GB of data around in memory: once 600 MB to calculate v*dt, and again to calculate r+(v*dt), and only then the result is written back to r. This can really become a major bottleneck if you aren’t careful. Fortunately, it is usually easy to circumvent; instead of writing r = r+dv, write r += dv. Instead of a = 3*a + b, write a *= 3; a+= b. This avoids the copying completely. For calculating v*dt and adding it to r, the situation is a bit more tricky; one good idea is to just have the unit of v be such that you don’t need to multiply by dt. If that is not possible, it might even be worth it to keep a copy of v which is multiplied by dt already, and update that whenever you update v. This is advantageous if only few v values change per step of your simulation. I would not recommend writing it like this everywhere though, it’s often not worth the loss in readability; just for really large arrays and when the code is executed frequently."

Comments (78 posted)

Garrett: Remembering David MacKay

Matthew Garrett remembers David MacKay, shortly after his passing. "I was already aware of the importance of free software in terms of developers, but working with David made it clear to me how important it was to users as well. A community formed around Dasher, helping us improve it and allowing us to develop support for new use cases that made the difference between someone being able to type at two words per minute and being able to manage twenty. David saw that this collaborative development would be vital to creating something bigger than his original ideas, and it succeeded in ways he couldn't have hoped for." (Thanks to Paul Wise)

Comments (1 posted)

PostgresOpen 2016 - Call For Papers

PostgresOpen will be held September 13-16 in Dallas, Texas. The call for papers closes May 30. "Presentations can be on any topic related to PostgreSQL, including, but not limited to, case studies, experiences, tools and utilities, migration stories, existing features, new feature development, benchmarks, performance tuning, and more!"

Full Story (comments: none)

CFP Deadlines: April 21, 2016 to June 20, 2016

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
April 24	August 20 August 21	Conference for Open Source Coders, Users and Promoters	Taipei, Taiwan
April 26	August 22 August 24	LinuxCon NA	Toronto, Canada
April 27	August 12 August 14	GNOME Users and Developers European Conference	Karlsruhe, Germany
April 30	June 11	TÜBIX 2016	Tübingen, Germany
April 30	October 27 October 28	Rust Belt Rust	Pittsburgh, PA, USA
April 30	August 25 August 26	The Prometheus conference	Berlin, Germany
May 1	August 24 August 26	KVM Forum 2016	Toronto, Canada
May 2	June 24 June 25	devopsdays Silicon Valley 2016	Mountain View, CA, USA
May 6	October 26 October 27	All Things Open	Raleigh, NC, USA
May 6	July 13 July 15	LinuxCon Japan	Tokyo, Japan
May 8	August 12 August 16	PyCon Australia 2016	Melbourne, Australia
May 15	July 2 July 9	DebConf16	Cape Town, South Africa
May 15	September 1 September 8	QtCon 2016	Berlin, Germany
May 15	June 11 June 12	Linuxwochen Linz	Linz, Austria
May 16	October 31 November 2	O’Reilly Security Conference	New York, NY, USA
May 23	October 17 October 19	O'Reilly Open Source Convention	London, UK
May 23	August 20 August 21	FrOSCon - Free and Open Source Software Conference	Sankt-Augustin, Germany
May 24	August 18 August 21	Camp++ 0x7e0	Komárom, Hungary
May 24	November 9 November 11	O’Reilly Security Conference EU	Amsterdam, Netherlands
May 25	October 5 October 7	International Workshop on OpenMP	Nara, Japan
May 29	September 20 September 23	PyCon JP 2016	Tokyo, Japan
May 30	September 13 September 16	PostgresOpen 2016	Dallas, TX, USA
June 3	June 24 June 25	French Perl Workshop 2016	Paris, France
June 4	July 30 July 31	PyOhio	Columbus, OH, USA
June 5	September 26 September 27	Open Source Backup Conference	Cologne, Germany
June 5	September 9 September 10	RustConf 2016	Portland, OR, USA
June 10	August 25 August 26	Linux Security Summit 2016	Toronto, Canada
June 11	October 3 October 5	OpenMP Conference	Nara, Japan
June 15	September 8 September 9	First OpenPGP conference	Cologne, Germany
June 15	November 16 November 17	Paris Open Source Summit	Paris, France

If the CFP deadline for your event does not appear here, please tell us about it.

EuroPython 2016: Schedule online

The schedule for EuroPython has been announced. The conference runs July 17-24 in Bilbao, Spain. "With over 180 sessions, over 150 speakers, one day for workshops, 5 days of talks, training, keynotes, lightning talks and open spaces, followed by 2 days of sprints, EuroPython will be one of the most exciting and vibrant Python events this year."

Full Story (comments: none)

PCI Microconference Accepted into 2016 Linux Plumbers Conference

LPC will be held November 2-4 in Santa Fe, NM. The PCI Microconference will look at PCI support in the kernel. "One issue with current PCI support is that resource allocation is handled on a per-architecture basis, leading to duplicate code, and, worse yet, duplicate bugs. This microconference will therefore look into possible consolidation of this code." There will also be discussions of firmware issues, hardware quirks, power management, and more.

Full Story (comments: none)

Events: April 21, 2016 to June 20, 2016

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
April 20 April 21	Vault 2016	Raleigh, NC, USA
April 21 April 24	GNOME.Asia Summit	Delhi, India
April 23	DevCrowd 2016	Szczecin, Poland
April 23 April 24	LinuxFest Northwest	Bellingham, WA, USA
April 25 April 27	Cuba International Free Software Conference	Havana, Cuba
April 25 April 29	OpenStack Summit	Austin, TX, USA
April 26 April 28	Open Source Data Center Conference	Berlin, Germany
April 26	Open Source Day 2016	Warsaw, Poland
April 28 May 1	Mini-DebCamp & DebConf	Vienna, Austria
April 28 April 30	Linuxwochen Wien 2016	Vienna, Austria
April 30	Linux Presentation Day 2016.1	many cities, Germany
May 1 June 29	Open Source Innovation Spring	Paris, France
May 2 May 5	FOSS4G North America	Raleigh, NC, USA
May 2 May 3	PyCon Israel 2016	Tel Aviv, Israel
May 9 May 13	ApacheCon North America	Vancouver, Canada
May 10 May 12	Samba eXPerience 2016	Berlin, Germany
May 14 May 15	Community Leadership Summit 2016	Austin, TX, USA
May 14 May 15	Open Source Conference Albania	Tirana, Albania
May 16 May 19	OSCON 2016	Austin, TX, USA
May 17 May 21	PGCon - PostgreSQL Conference for Users and Developers	Ottawa, Canada
May 24 May 25	Cloud Foundry Summit	Santa Clara, CA, USA
May 26	NLUUG - Spring conference 2016	Bunnik, The Netherlands
May 28 June 5	PyCon 2016	Portland, OR, USA
June 1 June 2	Apache MesosCon	Denver, CO, USA
June 4 June 5	Coliberator 2016	Bucharest, Romania
June 11 June 12	Linuxwochen Linz	Linz, Austria
June 11	TÜBIX 2016	Tübingen, Germany
June 14 June 15	PyData Paris 2016	Paris, France
June 19 June 21	DockerCon	Seattle, WA, USA

If your event does not appear here, please tell us about it.