Transparency, donations, and elementary OS

A very public spat between the founder of the elementary OS Linux distribution and one of the board members for Software in the Public Interest (SPI) in early February got kind of ugly. As with any dispute of this nature, there were various claims and counter-claims that make it difficult to determine all of what transpired. But, based on what can be discerned, there are some wider issues to consider.

The public part of the disagreement began when elementary OS founder Daniel Foré posted on Google+ about a telephone conversation he had with Joshua Drake, who is a member of SPI's board. Foré felt that he got a "really aggressive sales call" from Drake about having elementary OS join SPI. Foré suggested that Drake wanted the project to join so that SPI could get a cut of the money that the distribution brings in. As noted in an LWN review of the first elementary OS "Freya" beta back in August, the distribution strongly encourages $10 contributions for downloading. SPI does take a 5% cut of donations for member projects for its services.

Foré obviously did not like the perceived pressure to join SPI. He pointed out that the distribution already has a legal entity behind it, but that Drake was adamant that elementary OS needed to be associated with a non-profit. Foré went on to note that because he was not interested in joining, Drake planned a "blog about how poor your product is". He concluded by saying that what Drake was doing amounted to extortion.

For his part, Drake did post on his blog about elementary OS, though that post has since been taken down. It can still be found, however, but it doesn't come off quite the way Foré predicted. Drake does criticize the project for its web presentation, in particular he was concerned that site visitors get no indication of the for-profit structure of the limited liability corporation (LLC) behind elementary OS. While Drake was certainly critical of the project/company, it's hard to say to say that he went completely overboard.

But elementary OS press lead Cassidy James Blaede was not at all happy with Drake's post. He rebutted each of the main points Drake made, though, of course, it largely boils down to a difference of opinion. Elementary OS is not shy about asking downloaders for money, but there is not much information on the web site about the corporate structure or its for-profit nature. Essentially, that is most of what Drake was complaining about in his original post. In a replacement post that was clearly intended to tamp down the flames, Drake reiterated that more concisely.

As is often the case in these kinds of conflicts, though, the toothpaste is pretty hard to get back into the tube once it gets out. Drake's original blog post, coupled with Foré's and Blaede's Google+ postings (and the inevitable comment stream accompanying them), quickly spread to reddit and Phoronix. A post from Alberto Ruiz calling SPI "despicable" probably helped fan the flames.

But, at its heart, the dispute is about transparency. Drake may have made a ham-handed approach to Foré—at best he was misunderstood, at worst he was well out-of-bounds—and his withdrawn blog post was fairly confrontational, but what he was after was not completely unreasonable. The elementary OS web site is still not clear about where donations to the project go, or that it is a for-profit organization.

There is also the question of where the money that is donated actually goes. Blaede mentioned "servers, merch to resell, and occasionally conference/travel costs and development hardware" as expenses for the project. Elsewhere, the bounties offered at Bountysource have been noted as another project expense. All of that seems perfectly reasonable, but the project/company could be more open about how it spends its money, especially when it is more aggressive than most open-source projects in soliciting donations.

Just prior to Drake and Foré's telephone conversation—presumably part of what sparked it—elementary OS had gotten a lot of flak for its posting (since revised, original here) that downloaders who choose to not pay are "pretty much cheating the system". Those words were poorly chosen, though the project clearly does feel that users should pay for its work. The revised text now reads: "We want users to understand that paying for software is important and not paying for it is an active choice." That is, at least, a less antagonistic way to put it.

There are other revisions to that post, mostly toning down some of the rhetoric, but elementary OS does have a bit of a point. While the distribution is based on Ubuntu (which is based on Debian, of course) and builds atop all of the work those projects do, elementary OS does add value that some find useful and/or compelling from a design standpoint. Traditionally, open-source projects have been fairly passive about donations, but elementary OS appears to be bucking that tradition. So far, that has led to a lot of noise in various locations, but is it really so wrong for a project to try to get enough revenue to fund its activities?

Many of the criticisms about the distribution's funding stance are that it hasn't done "enough" to deserve the level of donations it is seeking. That may well be true, but there is nothing inherently wrong with trying. There are lots of projects that are struggling due to lack of funding, along with various efforts to find funding for them (including organizations like Snowdrift.coop that we covered back in December). One could argue that elementary OS is just trying a different approach. It remains to be seen whether it will be successful or not.

On the transparency side, elementary OS could certainly do more to make it clear that it is a for-profit entity. It is not obliged to share its finances—it is far from the only company in our community with closely held financial information—but doing so might help with donations. In the final analysis, it is a company trying to find its way in the (crowded) Linux distribution space; if it isn't doing "enough" to attract funding to keep it going, perhaps it will transform into something more like what its critics would like to see. Or it may die on the vine. But its code is free, which is pretty hard to criticize.

Comments (11 posted)

A look at LibreOffice 4.4

Version 4.4 of the LibreOffice office suite was released in January. The 4.4 development cycle placed a significant emphasis on improving the user experience (UX) and polishing the user interface (UI), both of which are aspects of LibreOffice that have been the target of criticism in the past. Indeed, the new release does present a cleaner and more organized UI in many places—improvements that can make or break an office suite. But there are other noteworthy changes to be found in version 4.4, from improved interoperability with Microsoft Office to the results of a massive code-clean-up effort.

The latest release can be downloaded from the LibreOffice site. Linux builds are provided for 32-bit and 64-bit Intel systems, in RPM and Debian packages as well as in tar archives. There is a lengthy release notes page on the project's wiki that outlines the changes to each component (Writer, Calc, Impress, and so on).

Some of the most prominent new features, though, extend across all of LibreOffice's constituent parts—such as the UI enhancements. During Google Summer of Code (GSoC) 2014, two students worked on converting all of LibreOffice's dialog boxes to the latest GtkBuilder format, which is both smaller and easier to work with than the older Glade format. Similarly, the drop-down toolbars and sidebars have been reworked, and many (if not all) right-click context menus have been reorganized.

The tricky aspect of this sort of development work is that it can easily go unnoticed, because it is largely about removing awkwardness: fixed-sized dialog boxes, drop-down menus that require a lot of scrolling, context menus that contain unnecessary (and rarely used) items. But if one knows where to look for them, the improvements are there to see. Take the "Bullets" and "Numbers" toolbar menus in Writer, for example. Clicking on the new menu items displays a rather large preview of each available style, which makes the alternatives instantly easy to distinguish from one another—something that was not so clear in older releases. But quantifying whether this change makes writing a little easier or much easier is hard to do.

Similarly, it is easy to point out that Cut, Copy, and Paste are now in the same position in every right-click context menu, but the practical impact it has is difficult to measure. Suffice it to say, then, that LibreOffice 4.4 incorporates a wealth of such changes that, when taken together, make the office suite far less frustrating to work in.

Of particular note on the UX front, the sidebar in Writer now provides access to the Styles and Gallery functions, which were previously hidden in separate dialog boxes. Styles allows the user to apply a pre-defined formatting style to selected text (e.g., footnotes, quotations, or various headers); Gallery offers drag-and-drop access to images that the user can place into a document. Both of these are functions that a typical user may need to drop into repeatedly during an editing session, so not having to open up a new window that steals focus from the main document is a big improvement.

The content of the toolbars has been significantly rearranged as well, with an eye toward making the most-common entries the easiest to reach. For those who read left-to-right languages, that means that the common entries are the furthest to the left, and the least likely to be hidden off screen. In retrospect, this seems like a rather obvious change to make, but in earlier versions of LibreOffice, the emphasis was evidently on grouping related toolbar items together. It is hard to be objective about which approach is universally better but, at the very least, not having important tools lost off-screen is certainly an improvement.

Users of a typographic bent will notice several improvements to the way fonts and special characters are treated. The character-selection widget (which is used to insert characters not found on the current locale's keyboard) has been simplified. There are also several interesting substitutions that are now automatically inserted by the spellchecker: typing (c) will insert the © sign; (tm) will insert ™, and so on. Version 4.4 also adds a new set of open-licensed fonts (Caladea and Carlito) designed to serve as drop-in replacements for Microsoft's Cambria and Calibri—which are, essentially, the modern equivalent of the so-called "core fonts" Times New Roman and Arial.

Among the many other application-specific improvements found in the 4.4 release, Writer now supports placing text into geometric shapes (and having the text confined to the outline of the shape) and the status bar at the bottom of the window now more clearly presents information about the current page. Calc's "Statistics Wizard," which automatically analyzes a set of cells, gained several new functions (like Quartiles and the Chi-square test). Draw finally gained a zoom tool, while Impress gained improved support for using OpenGL slide transitions. All applications can now export digitally signed PDFs; users must manually load their signing certificate into LibreOffice's certificate storage in order to take advantage of the feature.

There are quite a few additional file types now readable by LibreOffice 4.4. Much of the work involved in this area comes from the Document Liberation Project, a spin-off of LibreOffice that focuses on building file-format conversion libraries. Among the newly supported formats are Adobe PageMaker, MacDraw, RealAudio, Sony's LRF e-book format, Ogg Opus multimedia, and MPEG Audio.

Perhaps of more importance at a practical level, though, is that LibreOffice 4.4 updates its support of Microsoft's OOXML format to the latest revision. A sad fact of life is that the majority of LibreOffice users still need to exchange files with Microsoft Office users on a regular basis. LibreOffice may always lag somewhat behind Microsoft when it comes to OOXML file support (indeed, it is in Microsoft's interest to make competing office suites play catch-up), but there is no denying that it is a necessary feature.

The LibreOffice project reports that it undertook a major effort during the 4.4 development cycle to perform code reviews with the Coverity scanning service. The Coverity scans reportedly turned up more than 6,000 defects—all of which were fixed. That number included more than 1700 null pointer dereferences and 1100 uninitialized members. For a project of LibreOffice's size, it is impressive to see that much ground covered in such a relatively short time.

In some ways, LibreOffice 4.4 is not an exciting release: there is no banner-worthy new feature, but a great many small improvements instead. Then again, office suites might not ever need to trade on a reputation for excitement; steady progress and stability are more important. The UI and UX improvements offer a nicer visual design and an improved workflow, respectively; that might be enough to attract some more users to the project, but if their main impact is simply to make the editing experience smoother or more pleasing for existing users, that is a worthwhile end of its own.

Comments (7 posted)

Scalar typing in the PHP world

Some history

PHP has traditionally taken a loose approach to types, trying to make things "just work" in almost all circumstances. A simple session in the interpreter's interactive mode shows how accommodating it can be:

    $ php -a
    php > echo 1 + "1";
    2
    php > echo 1 + true;
    2
    php > echo 1 + "1with_extra_junk";
    2

This flexibility undoubtedly makes PHP fun for some programmers to use; there is little of the messing around required by some other languages to keep the interpreter happy. Among other things, it allows the creation of library functions that can take a wide variety of types as parameters and do something (maybe even the right thing) with them.

But others would like to see things tightened down a bit. It would be nice, they argue, if a PHP function could count on the types of the arguments passed to it. That would enable certain kinds of errors to be caught by the interpreter; it would also make it easier to write static analyzers for the language that could be used for both correctness checking and performance improvements. These developers have made their case well enough that there is little controversy about adding some type checking; the problem arises when one tries to determine how much checking makes sense.

The specific issue at hand has to do with the passing of simple, scalar parameter types to functions — types like integers, floating-point numbers, strings, and booleans. The first attempt to add type hints for these parameter types was posted by Anthony Ferrara in 2012. It changed the function-definition syntax to look like this:

    function f ((int) arg) {
	// "arg" is guaranteed to be int here
    }

This version implemented what has been termed "weak" typing, in that a non-integer argument passed to f() would be converted to the int type if possible. So floating-point numbers would be truncated, strings would be converted, etc. This proposal generated a lot of discussion that eventually bogged down; a year later, Anthony loudly bailed out of PHP development, strongly criticizing the environment found on the php-internals mailing list.

Weak or strong?

As the various versions of the proposal went by, the real point of contention became clear. Nobody seems to object to the idea of adding scalar type hints to function definitions; the real problem had to do with how strictly they would be enforced. The "strict typing" camp wants the interpreter to disallow all implicit type conversions; if f() is defined to require an integer argument, calls like f(1.0) or f("1") should generate an error. In this view of the world, if an API developer puts type declarations onto a function, that function must be called with parameters of exactly the specified type.

The arguments for strict checking are fairly straightforward. It can catch errors that weak (or no) type checking will miss — the sort of nasty error that never happens on a developer's test system but which certain types of users can provoke easily. Strict checking lets API developers design their code with the knowledge that it will be called with the correct types. It also expands the range of static analysis options available, again turning up errors but also, it is said, opening the door to new optimization techniques.

On the other side, opponents of strong typing feel that it would drastically change the nature of the language. PHP has always allowed a high level of freedom in the mixing of types. Changing that for PHP 7 would create a language that, in their view, is not PHP. Among these folks, many of whom are core developers of the language, adding strong typing would seemingly be like trying to get Python developers to add braces or Perl developers to restrict themselves to "one way to do it." Opponents also worry that strict typing may just lead developers to write lots of casts, effectively taking away the benefits that strict typing is supposed to provide.

An attempt at compromise

Andrea appears to have despaired of ever bridging the gap between these two factions; instead, she put together a new proposal that tried to please everybody, then called for a vote. In the call, she optimistically said "I don’t think there needs to be, or will be, much further discussion"; that is not how things turned out.

The new scalar typing mechanism is still weakly typed, but with a twist. A PHP developer can add a line like this to their code:

    declare(strict_types=1);

The effect of this line will be to turn on strict scalar type checking for the rest of the source file in which it is found. Library authors can add types to their function definitions; they can even turn on strict checking within the library with the above option. But it will be the author of the calling code who decides whether or not to add that line to their own code and turn on strict checking for their program.

In theory, this mechanism gives everybody what they want; developers who want to work in a strict-checking environment can have it, while those who want weak typing get it by default. In practice, this approach appears to have pleased almost nobody. Proponents of strong typing would rather see it enabled as a mandatory thing; they want callers of their functions to be required to pass the correct types. Opponents, instead, see it as a way of sneaking an alien concept into the language. PHP creator Rasmus Lerdorf also worries that it could end up breaking a lot of code and generally leading to surprising results.

To top it off, almost nobody seems to like the use of declare() which, among other things, can make it hard to tell which rules apply to a given segment of code. As Zeev Suraski put it:

Even the declare() option has proponents, though; Anthony returned to the fray to argue in its favor:

The current vote is scheduled to end on February 19. As of this writing, the results suggest that proposal might just pass with something very close to the required supermajority. In theory, that should put this long-lived debate to rest; long experience suggests, though, that an issue this contentious will not be resolved quite so easily.

Comments (148 posted)

On moderation

Those exceptions lead some people to ask for more "moderation" of the comment stream.

Those of us who work on LWN have put a great deal of ourselves into it for many years. LWN is, in many ways, a labor of love. When somebody posts a comment that is offensive, abusive, spammy, or just a troll, it feels like they have painted obscene graffiti onto our houses. But we remain highly reluctant to "moderate" (or "delete," or "censor") those comments for a number of reasons. At the top of the list is the simple fact that we believe that "freedom" means more than the ability to share software with each other. To silence a voice with editorial force feels like an attack on freedom; it is not something to be done lightly.

Let there be no mistake: we do not support those who post abusive, bigoted, or demeaning comments, and we feel that there is no place for them on LWN. Our desire and goal is to run a site where that kind of comment is simply not present. But chances are that we will continue to use the heavy hand of "moderation" toward that goal rather less frequently than some would like.

Trying to control the direction of a comment stream is a slippery slope in a number of ways. In a heavily controlled environment, it is hard to tell whether the opinions on offer reflect the community, or whether they simply reflect what the moderator thinks is OK to be said. We want LWN to reflect our community, not our view of what that community should be.

What are the objective criteria by which moderation would be performed? If we try to rid the stream of anything that might be offensive to anybody, we will have a bland discussion indeed. And why stop at limiting offense? Consider that the recent call for more moderation quickly led to a request to cut off a discussion that, while admittedly long and repetitive, was focused on an area of great concern to many developers. The desire to see moderation used in such a situation shows how slippery this slope can be.

Also relevant are the facts that moderation is simply expensive and a source of legal liability. Putting together an edition every week is a serious grind; we have nobody with spare time on their hands to patrol the comments. The only way to make that kind of time available would be to reduce the number of articles we write for the site. That would be a heavy cost, and it still doesn't address the fact that unpleasant personalities feel free to post outside of LWN's business hours.

It is not surprising that a number of publications have recently taken the step of eliminating comments altogether. That is not a step we would like to take, but we will if that appears to be necessary.

But such an extreme step should not be necessary. We as a community are better than that.

Experience shows that simple requests to tone down or close off a discussion are often successful. We may well be making more such requests in the future. But the success of LWN's comment areas depends mostly on the commenters; we do not have the time, energy, or desire to play "comment nanny" all the time. If we could all make an effort to recognize when a discussion has run its course, avoid repeating ourselves, and refrain from feeding trolls, things would work better for all of us.

At the same time, we can respond to overtly offensive comments by stating, politely, that they are not welcome in our community. That, in your editor's opinion, is more likely to lead to a more respectful community in the long run than trying to moderate our way into a world where such people just do not seem to exist.

As a last resort, we may implement the ability to close down specific threads if the need arises — but we will endeavor to almost never use it. We already have the ability to block comments from specific accounts; we almost never use that either. The use of such tools is indicative of a failure at a higher level; it is a sad day when they must be put into action.

This is not the position that some people would like us to take, but it is what we feel that we are able, morally and economically, to do. And please bear in mind that comments here have gotten somewhat better over the years, even if they are still not where we would like them to be. Those with strong stomachs can look at some previous discussions for evidence of that. "Things used to be worse" is no justification for an unacceptable state of affairs now, but it does show that progress can be made. We are a community of adults, and we are certainly capable of behaving that way.

Comments (66 posted)

Mandatory Firefox extension signing

Mozilla recently announced a plan to roll out a cryptographic signature framework for Firefox extensions in the coming months. The system is designed to clamp down on so-called "gray area" extensions that, while far from falling under the traditional definition of malware, make dubious and shady changes to the way a user's browser functions. The change will mark a significant shift for Mozilla, which, in the past, has taken a wide-open approach to extension development and installation. But the new system will be mandatory: after a transition period, Firefox will not run extensions that have not been signed by Mozilla, and all extension developers will need to submit their extensions for a test that checks for bad behavior. Naturally, not all extension developers are happy about how that will affect their workflow, but Mozilla insists it is the option that best balances protecting users and encouraging extension development.

The announcement was posted by Mozilla's Add-ons Developer Relations Lead, Jorge Villalobos. He listed several types of malicious behavior commonly found in the gray-area extensions: altering the user's homepage, changing the browser search behavior, injecting ads into pages, and injecting scripts into social media sites. The extensions in question typically make these changes in secret, often under the guise of doing something else. Many present themselves as toolbars of one form or another; others get side-loaded, surreptitiously, when the user runs an installer or updater for some unrelated piece of desktop software.

In the past, Mozilla attempted to curtail such extensions by publishing a set of public guidelines for acceptable add-on behavior, and blocking extensions that violate the rules. That strategy may have scaled as far as it can, though—the blocklist is quite long, and developers have taken to new strategies to evade it, such as randomly generating a different extension ID for every copy.

The plan

Preventing bad behavior in extensions is more difficult in Firefox than in other browsers like Chrome or Apple's Safari, Villalobos said, for two reasons: first, Mozilla intentionally allows its add-ons to alter more browser functionality than the competing browsers do, and second, because Mozilla does not act as the central distribution point for add-ons. Anyone can publish and distribute a Firefox extension from any site.

Finding a way to curb gray-area extensions without acting as the final gatekeeper for extension publication was the issue. As Villalobos explained, the plan that emerged after many months of discussion was to run all extensions through an automated test that checked for the blocked, gray-area behaviors, then to sign those extensions that pass using a key that only Mozilla holds. Firefox has long supported signed extensions, but the feature has rarely been used by developers. Mozilla will add a compile-time switch that makes Firefox to check for the Mozilla signature on every extension. Extensions that do not pass the signature check will be deactivated. Future Firefox releases will then be built with the signature-checking switch enabled.

That part of the plan is relatively straightforward; where things get more complicated is how the extension-signing process will be implemented. First, there will be new steps required for developers whose extensions are not hosted at Mozilla's official add-on site, addons.mozilla.org (AMO). Since AMO-hosted extensions are already required to pass a review step, the good-behavior test and signature will simply be rolled into that same process. But developers who provide extensions on their own site will have to create an AMO account and submit each build for the test-and-signature process.

The extension IDs used by each account will be logged, so that Mozilla can note suspicious behavior (like submitting a different user's extension ID, or submitting thousands of IDs). These third-party developers' AMO accounts and extensions will never be visible to the public, though, and they can continue to distribute their code at their own sites. Villalobos also mentioned that there will be yet a third process for use by developers who create extensions designed only to be used on an internal (e.g., enterprise) network—but the nature of that internal-extension-development process has yet to be announced.

The second issue is that not all well-behaved extensions will pass an automated test. But the precise nature of the automated test has not yet been made public. Villalobos said that any extension that fails can be submitted for a manual review, which Mozilla hopes will take at most a couple of days. Furthermore, he expressed Mozilla's willingness to work with known extension developers who have special needs on a case-by-case basis. The vast majority of extensions are expected to pass the automated test, since very few of them have any reason to use the functionality exploited by the problematic gray-area extensions.

Finally, although the plan hinges on not providing a run-time switch to disable signature checking (because any run-time preference or switch could be exploited by a malicious extension or installer), Mozilla will release the Developer Edition and Nightly builds of Firefox with signature-checking turned off, so that developers can test their code without waiting on signatures. Mozilla will also provide special "unbranded" Firefox builds with signature-checking disabled, via a separate download site. The unbranded builds are something of a last resort: extension developers can point users toward those builds if they cannot or will not get their extensions signed.

The signature-checking system will be rolled out in a transitional state for two full Firefox release cycles (i.e., 12 weeks), perhaps beginning as soon as Firefox 39 (currently scheduled for late June). During the transition phase, extensions that fail the signature check will prompt a warning, but will continue to run. As of now, no other add-on types (such as themes, dictionaries, plug-ins, or search engines) will be signature-checked, and there are no plans to roll out signature checking for Thunderbird or for Seamonkey (although either of those projects could independently decide to deploy the same scheme).

Feedback

As might be expected, the announcement drew plenty of vitriolic criticism; quite a few commenters on the blog post accused Mozilla of "Apple-like" or "Google-like" behavior by exerting central control over extension distribution, for example. Others likened the plan to censorship, arguing (for instance) that once the process was in place, the US government or the entertainment industry would begin pressuring Mozilla not to sign extensions they disliked. The debate was also carried over to the mozilla.addons.user-experience discussion group, where the deliberations certainly had their own share of name-calling—although participants in that forum focused primarily on how the new system would impact the process of developing extensions.

In both discussions, however, some common questions recurred. Several asked why the signature-checking scheme relied only on a signing key from Mozilla—as opposed to, say, per-developer keys, or reusing some existing trust mechanism. Villalobos replied that per-developer keys are too easily compromised by malicious developers. Similarly, Adam Novak argued that trying to implement similar safeguards through the SSL Certificate Authority (CA) system would also be easily undermined by malware developers.

Since Mozilla is intent on being the only entity that can sign an extension as passing the good-behavior test, others complained about the fact that there will be no run-time mechanism to disable signature checking. At his blog, Jeff Lyon called the plan a violation of the core principles in the Mozilla manifesto and of the spirit of free software. Subsequently, he posted his own alternative proposal to the discussion forum, in which users would be able to switch off signature-checking by visiting a Mozilla-hosted site that would generate a cryptographic token (signed by Mozilla) for each user. The token, he said, would ensure that the user had expressed their consent to switch off signature-checking, and it could be hidden behind a CAPTCHA-like "human test" to prevent abuse by malware authors.

But Lyon's suggestion, like the other run-time switch proposals, was rejected. Essentially, the Mozilla team argued, there is no run-time mechanism that cannot be exploited: CAPTCHAs are easily broken, web page interaction can be scripted and run without the user's knowledge, and settings can be altered. Mike Connor replied that there is no way to allow unsigned non-malware extensions to run that would not be exploited. "It's unlocking a door to let in a friend, then leaving it unlocked forever." In addition, he said, the goal is to dramatically reduce the amount of gray-area malware in the wild, as a "herd immunity" goal.

Forward progress

Ultimately, most of the extension developers' concerns boil down to the fact that they like to push out test builds for their extension's users—sometimes even making a special build to be sent directly to a single bug reporter. Villalobos and the other Mozilla employees continued to insist that the automated extension test would not introduce a significant slow-down to that process. But it seems like many extension developers find that hard to believe.

Perhaps it would be easier to make that case if there were more details available. After all, it is easy for skeptics to imagine that the extension-checking process will be slow and work poorly when there is no evidence to the contrary. That puts Mozilla in an unenviable position, to be sure.

Similarly, there is a lot of confusion about some parts of the plan that have yet to be rolled out—such as the "unbranded" Firefox builds that will have signature-checking disabled. As Villalobos described it, users will have to seek out the unbranded Firefox builds and install them intentionally (perhaps through the Mozilla Developer Network, rather than the usual download site). Once installed, the lack of "Firefox" branding should make it obvious that a different browser is in use, so it will be hard for a malware author to trick a user into installing the vulnerable build. But that, too, is a mechanism that it is all too easy to imagine not working as expected. Mozilla will still have to show it functioning in the wild to win over some developers.

It is also unclear how the signature scheme will affect Linux distributions. Some might be interested in making Firefox builds available that have the signature-checking code compiled out. Whether or not that change would be enough to trigger a trademark dispute of the sort that led Debian to rebrand its Firefox builds as "Iceweasel" has not been explored.

In the meantime, Connor reminded those in the thread that the real goal was to improve users' security:

Villalobos, too, pointed out that the problem Mozilla is attempting to solve is that the status quo offers essentially no mechanism to block bad actors:

Connor also took issue with the suggestion that the plan violates free-software principles, noting that anyone who objects to it can still install one of the Nightly or Developer Edition builds—or even recompile Firefox themselves.

Tyler Downer estimated that the number of gray-area Firefox extensions in the wild was in the millions. That is surely a problem that needs to be addressed. How onerous extension developers and users ultimately find the new signature-checking scheme remains to be seen. It may make a significant dent in this long-standing problem—although it will surely not be the end of the Firefox malware story.

Comments (23 posted)

Security quotes of the week

Comments (36 posted)

deb.haskell.org compromised

Comments (27 posted)

FreeBSD random number generator broken for last 4 months

Comments (28 posted)

chromium: multiple vulnerabilities

Comments (none posted)

clamav: multiple vulnerabilities

Comments (none posted)

cpio: directory traversal

Comments (none posted)

cups: buffer overflow

Comments (none posted)

dbus: denial of service

Comments (none posted)

file: memory leak

Comments (none posted)

glibc: two vulnerabilities

Comments (none posted)

jython: code execution

# (umask 000; jython -c 'import xmllib')
# ls -l '/usr/share/jython/Lib/xmllib$py.class'
-rw-rw-rw-. 1 root root 52874 Apr  3 17:24 /usr/share/jython/Lib/xmllib$py.class

Comments (none posted)

kernel: multiple vulnerabilities

Comments (none posted)

mdadm: command injection

Comments (none posted)

moodle: cross-site scripting

Comments (none posted)

oracle-jre-bin: multiple unspecified vulnerabilities

Comments (none posted)

perl-Gtk2: code execution

Comments (none posted)

php5: multiple vulnerabilities

Comments (none posted)

pigz: directory traversal

Comments (none posted)

puppetlabs-stdlib: privilege escalation

Comments (none posted)

roundcubemail: cross-site scripting

Comments (none posted)

sudo: information disclosure

Comments (none posted)

unzip: code execution

Comments (none posted)

virt-who: information leak

Comments (none posted)

xorg-server: information leak/denial of service

Comments (none posted)

Kernel release status

Stable updates: none have been released in the last week.

Comments (none posted)

Quotes of the week

Comments (1 posted)

3.20 merge window part 2

• The parallel NFS (pNFS) subsystem has gained support for the under-development FlexFile layout. This layout allows file metadata to be stored in a different location from the file contents.

• The persistent storage subsystem can optionally provide a new special file (/dev/pmsg0) allowing user-space programs to log data into the store.

• The Smack security module can now interface with the netfilter system, using security labels to filter packets.

• The ubifs filesystem now has multiqueue block layer support (increasing its performance) and support for security.* extended attributes (making security module support possible).

• The Android binder code has been equipped with security hooks allowing it to be brought under SELinux (or any other security module) policies.

• The I2O bus subsystem has been moved into the staging directory with the idea of removing it from the kernel entirely in the near future. As far as the developers can tell, nobody is using this code; if that impression is mistaken, now would be a good time to say something.

• See this posting for a description of the changes to the Intel graphics driver in this development cycle.

• The nonvolatile memory support patches have been merged, making it possible to host filesystems in persistent memory with good performance.

• The PA-RISC architecture is no longer able to run 32-bit HP-UX binaries. The number of users upset by this change is expected to be small.

• The lazytime filesystem option (with support in ext4, initially) has been merged. Lazytime allows accurate tracking of file access times without creating lots of write I/O to the filesystem.

• New hardware support includes:
  • Systems and processors: IBM s/390 z13 processors, Artesyn MVME2500 single board computers, Conexant Digicolor SoCs, NVIDIA Tegra132 SoCs, Freescale LS2085A SoCs, and Mediatek MT65xx & MT81xx ARMv8 SoCs,

  • Graphics: ATMEL HLCDC display controllers and Samsung Exynos7 SoC display controllers. Also, the "fbtfb" subsystem has been added to the staging tree; it provides support for a wide range of small TFT LCD display modules.

  • Industrial I/O: Solteam Opto JSA1212 proximity and ambient light sensors, Kionix KMX61 6-axis accelerometer/magnetometers, Freescale MMA9551L intelligent motion sensors, Freescale MMA9553L intelligent pedometers, Semtech SX9500 proximity sensors, Capella Microsystems cm3232 ambient light sensors, Qualcomm SPMI PMIC voltage analog-to-digital converters (ADCs), Cosmic Circuits 10001 ADCs, and Samsung Sensorhub microcontroller units.

  • Miscellaneous: APM X-Gene GPIO standby controllers, Fujitsu MB86S7x GPIO controllers, Altera mailbox units, Version 2.0 Trusted Platform Modules (TPMs), Abracon AB-RTCMC-32.768kHz-B5ZE-S3 realtime clocks, Armada 38x Marvell SoC realtime clocks, ETRAX FS serial ports, Alphascale ASM9260 timers, Rockchip rk3288 timers, Conexant Digicolor timers, and Dallas/Maxim DS1685 realtime clocks.

  • Pin control: Allwinner A31s SoC pin controllers, Amlogic Meson SoC pin controllers, and Xilinx Zynq pin controllers, Qualcomm 8916 pin controllers.

  • USB: Rockchip USB2 PHYs and NXP ISP1761 USB device controllers.

Changes visible to kernel developers include:

• The ARM I/O memory-management unit (IOMMU) layer has a new, generic page-table management API. See drivers/iomu/io-pgtable.c and io-pgtable.h for an overview of this API.

• The LED subsystem now has a new device class for LEDs operating in the "flash" (as in camera flash) mode.

• The kernel address sanitizer (KASan) has been merged. KASan monitors kernel memory references in an attempt to catch code reaching into memory it has no business touching. For now, KASan only works on the x86_64 architecture, and memory hotplug must be disabled.

• Developers working with the GDB debugger may want to look at the new set of helper scripts added under scripts/gdb.

• The printk() family of functions has a new format type (%pb) for the printing of bitmaps. The number of bits in the bitmap must be specified as the field width in the format string.

So far, this development cycle seems to be a relatively slow one, as was suggested before the merge window opened. Still, the emphasis should be on "relatively"; nearly 8,000 patches is not a small number.

The version number for this kernel is yet to be determined. Linus ran a poll on Google+ that came out in favor of calling it 4.0, but he has not said what he will actually do. Tune in next week for the final changes that come in for this cycle and, presumably, an answer to the naming question.

Comments (1 posted)

Epoll evolving

An epoll overview

Epoll is designed to function somewhat like select() or poll(), but with more options and with higher performance when large numbers of file descriptors are in use. Each call to select() or poll() can involve an entirely new set of file descriptors, so the kernel must go through the process of validating each one, checking for I/O readiness, and adding the polling process to the appropriate wait queue. But the actual list of file descriptors tends not to change much between calls, so much of that work is unnecessary duplicated effort. The epoll calls get around this problem by separating that setup work from the act of waiting for a file descriptor to become ready.

A process using this API must begin by creating a special file descriptor to use with polling; that is done with a call to one of:

    #include <sys/epoll.h>

    int epoll_create(int size);
    int epoll_create1(int flags);

Either call will return a file descriptor to be used with the remaining epoll functions. The size parameter to epoll_create() is no longer used; the flags argument added for epoll_create1() can be used to set the close-on-exec flag for the resulting file descriptor.

The next step is to add all of the file descriptors that are to be monitored, using:

    int epoll_ctl(int efd, int op, int fd, struct epoll_event *event);

If op is EPOLL_CTL_ADD, the given fd will be added to the set. The event parameter is used to describe which events will be polled for; see the man page for details. epoll_ctl() can also be used to remove file descriptors or to modify how the polling is done.

Finally, actually waiting for one of the file descriptors in the set to become ready is done with:

    int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);
    int epoll_pwait(int epfd, struct epoll_event *events, int maxevents, int timeout,
                    const sigset_t *sigmask);

The events that cause epoll_wait() to return (up to maxevents) will be stored in the events array; the timeout parameter is specified in milliseconds. The epoll_pwait() version also allows the specification of a set of signals to be blocked (or unblocked) during the call. Again, see the man page for details.

epoll_ctl_batch() and epoll_pwait1()

This patch set from Fam Zheng introduces two new system calls to the epoll family. The first of these addresses a performance problem that comes up in situations where it is often necessary to make changes to the file descriptors in the epoll set. A call to epoll_ctl() can only add, modify, or delete a single file descriptor; if a lot of descriptors need to be changed, there will need to be a lot of epoll_ctl() calls to get that work done. The proposed epoll_ctl_batch() system call gets around that problem by processing multiple file descriptors in a single call:

    int epoll_ctl_batch(int epfd, int flags, int ncmds, struct epoll_ctl_cmd *cmds);

The cmds structure essentially duplicates all of the arguments that would have been passed to an epoll_ctl() call. By passing an array of these structures, a program can perform operations on multiple file descriptors with a single system call.

Fam's other change is to add a new system call to perform the actual polling:

    struct epoll_wait_params {
	int clockid;
	struct timespec timeout;
	sigset_t *sigmask;
	size_t sigsetsize;
    };

    int epoll_pwait1(int epfd, int flags,
                     struct epoll_event *events, int maxevents,
                     struct epoll_wait_params *params);

This version of epoll_wait() adds a new flags parameter, but does not define any flag values, so flags must be zero. The parameters shuffled off into the params structure are mainly intended to give the application more control over timeout processing. The millisecond-resolution timeout understood by epoll_wait() has proved to be too coarse for a number of use cases. The new system call defines a timeout with nanosecond resolution, getting around that problem.

Better multi-threaded behavior

Jason Baron (of Akamai) has run into a different problem that only comes up with a relatively uncommon usage pattern. Normally, there will only be one process using a polling function to monitor a given file descriptor. But, in Jason's use case, there can be multiple threads, each of which is using epoll to track the same set of file descriptors. In this setting, an event on any given file descriptor will cause all of the waiting processes to wake up, even though only one of them will be able to actually handle the event. This thundering-herd problem is something that Jason would like to avoid.

His response is this patch adding a couple of new flags to be attached to file descriptors via epoll_ctl(). The first of these, EPOLLEXCLUSIVE, requests that only one process be woken up to handle an event on the associated file descriptor. Internally, the change is a simple matter of using add_wait_queue_exclusive() instead of add_wait_queue() when setting up the polling. Obviously, all processes polling the same file descriptor would have to use the exclusive mode to get down to a single wakeup per event.

That change did not entirely solve Jason's problem, though, in that it ended up with the same process waking up in response to each event. Since one of the reasons for epoll's existence is to allow processes to be left on all of the file-descriptor-specific wait queues between calls to epoll_wait(), the process that is at the head of any given queue will remain there. It will thus be the one to receive all of the exclusive wakeups. But the whole point of having multiple processes polling the same file descriptor is to spread the work around; waking the same process every time thwarts that objective. To deal with that, Jason added another flag, called EPOLLROUNDROBIN, that causes the kernel to work through each of the polling processes in turn.

Support for this mode was added to the scheduler in the form of a new wait queue function:

    void add_wait_queue_rr(wait_queue_head_t *q, wait_queue_t *wait);

When waiting is done using add_wait_queue_rr(), only a single process will wake, just like with add_wait_queue_exclusive(). But, in addition, the process at the head of the wait queue will be moved to the tail, so it will not see another wakeup until all of the other processes have had their turn.

Jason's patch posting includes results from a little benchmark program showing a nearly 50% reduction in execution time when the exclusive mode is used. When there are a lot of wakeups happening (as can be the case for many network-oriented workloads), the extra overhead caused by thundering herds can be crippling.

Both patches have received a fair number of review comments, and Fam's patches in particular have evolved quite a bit since the first posting in January. Your editor's unscientific impression is that API-related patches are getting more attention than they once did. That, of course, would be a good thing; APIs live forever, so it's better to get them right before shipping them to users and being committed to supporting them forever. These patches seem close to being ready, though; they might make an appearance in the next merge window.

Comments (9 posted)

Creating scalable APIs

Non-scaling APIs—those that perform poorly on systems with large numbers of CPUs—are a painful fact of life, an unavoidable consequence of many of these APIs having been created on uniprocessors by people with little or no experience with parallelism. However, designing scalable APIs can be a challenge even for those of us with significant parallel-programming experience.

Therefore, it should come as no surprise that Austin T. Clements's Ph.D. dissertation [PDF] entitled “The Scalable Commutativity Rule: Designing Scalable Software for Multicore Processors” received a well-deserved SIGOPS Dennis M. Ritchie award in 2014. The most important points are covered in the corresponding 2013 Symposium on Operating System Principles (SOSP) paper [PDF] and ACM Transactions on Computer Systems (TOCS) article (both co-authored by M. Frans Kaashoek, Nickolai Zeldovich, Robert T. Morris, and Eddie Kohler), which are further summarized here. This work resembles the Laws of Order paper covered by an LWN article back in 2011. However, as you might guess from the name, the scalable commutativity rule is concerned with scalability, in contrast with the memory-barrier focus of Laws of Order.

Overview of the scalable commutativity rule

We all learned about the commutative law of addition, which states that X + Y gives the same value as Y + X, for all possible values of X and Y. The scalable commutativity rule uses a more general definition of commutativity that encompasses the order of execution of concurrent operations as well as the order of arguments to a single operator or function. For example, suppose that one CPU atomically adds the value X to variable A at about the same time that another CPU atomically adds the value Y to this same variable. Because addition is commutative in this more concurrent sense, we know that regardless of the order of execution, the overall effect will be to add X + Y to A.

The key insight behind the scalable commutativity rule is that if the ordering of a pair of operations is irrelevant according to the API, then it should be possible to implement those operations so as to avoid scalability bottlenecks. In contrast, if the ordering of the two operations is critically important, then it is most likely impossible to avoid the bottlenecks between these operations.

For example, suppose that the two operations are inserting two objects, each with its own key, into a search data structure not already containing elements with those keys. Suppose further that this structure does not allow duplicate keys. No matter which of the two operations executes first, the result is the same: both objects are successfully inserted. Therefore, the scalable commutativity rule holds that a scalable implementation is possible, and there is, in fact, a wealth of scalable implementations of search structures, ranging from hash tables to radix trees to dense arrays.

In contrast, suppose that the two operations are inserting two objects with the same key into a search data structure that does not already contain an object with that key. Now the order is critically important, because the first operation will succeed and the second will fail. That operation is unlikely to be scalable. For example, in a hash table with per-bucket locking, both operations would attempt to acquire the same lock, thus preventing them from executing concurrently.

But pure mathematical commutativity requires that a given pair of concurrent operations always give the same result, regardless of not only execution order, but also initial state and argument values. This means that pure mathematical commutativity is too restrictive to be useful for predicting scalability in practice. For example, conditional lock acquisition is non-commutative from a pure mathematical viewpoint because if two CPUs attempt to conditionally acquire a given lock at the same time, one of the CPUs will acquire the lock and the other will get a failure indication. Which CPU acquires and which gets a failure indication depends on the order of execution, hence the non-commutativity from a pure mathematical viewpoint.

Of course, in practice, we usually work hard to make sure that different CPUs are attempting to acquire different locks, and in this case, conditional lock acquisition is independent of execution order. In other words, from a pure mathematical viewpoint, conditional lock acquisition is not commutative, but for common use cases in practice, it commutes—and scales—just fine. The authors therefore define a alternative notion of commutativity that is state-dependent, interface-dependent, and monotonic, which they abbreviate as “SIM commutativity”. Because this type of commutativity depends on the values of the arguments of the function in question (unlike mathematical commutativity), conditional locking is SIM commutative because if the two CPUs are acquiring different locks, the results will be independent of the order of acquisition. This definition is, therefore, compatible with the long-standing parallel-programming advice to “lock data, not code.”

API analysis and the Linux kernel

As with the Laws of Order, a key strength of the scalable commutativity rule is that it can be applied to an API, even before the implementation is written. In fact, the authors have constructed a tool named Commuter that can analyze whether or not a given API is SIM commutative. They applied this tool to a subset of 18 POSIX system calls, which produced no fewer than 13,664 test cases exercising commutative pairs of operations.

Of course, the fact that an API is commutative does not necessarily mean that a given implementation scales. For example, a kernel that relied solely on a Big Kernel Lock for synchronization would not scale well regardless of the API. Therefore, the authors also created a tool named mtrace that, in conjunction with other tools described in Section 5 of the SOSP paper, analyzes the Linux kernel to find bottlenecks that the scalable commutativity rule indicates could be avoided. This set of tools uses a modified version of QEMU to carry out this analysis, and take the test cases mentioned earlier as input.

The Linux kernel did surprisingly well, avoiding violations of the scalable commutativity rule for 9,389 of the 13,664 test cases, or almost 70% of them. But the authors figured that it was possible to do better. Unfortunately, their time budget did not permit them to rewrite the Linux kernel, so they instead created a research OS kernel named sv6. This kernel avoided violations of the scalable commutativity rule for 13,528 of the 13,664 test cases, or more than 99% of them.

API analysis and POSIX

The authors' tool also identifies POSIX operations that do not commute, which thus might be harder to implement scalably. One example is fork(), which copies an entire process, thus failing to commute with pretty much anything that process might do. This is, of course, especially painful when invoking fork() from a multithreaded process. This can be particularly annoying if the child immediately does an exec(), thus rendering almost all fork()-based non-commutativity irrelevant. They suggest using the obscure posix_spawn() in place of fork()/exec() because posix_spawn() is equivalent to those two calls but is much more commutable.

The authors also call out the non-commutativity inherent in the POSIX rule requiring that open() return the lowest-numbered available file descriptor. They point out that this non-commutativity could be eliminated via an O_ANYFD flag to open(). These and other suggestions are described in the next section.

Design and implementation guidance

Sections 4 and 6.3 of the SOSP paper provide some design and implementation guidance.

Decompose compound operations. The paper gives two examples of commutativity-challenged compound operations: fork() and stat(). As noted above, the fork() system call copies a process, and most of the time most of this work is discarded by a subsequent exec(). However, all this copying causes fork() to be non-commutative with everything from memory writes to open() and close() system calls. Decomposing the fork() system call to eliminate most of the doomed copy operations, and then combining it with exec() results in something called posix_spawn(), which avoids much of the fork() system call's non-commutativity. In theory, then, posix_spawn() should be much more scalable than is fork().

Similarly, the stat() system call returns so much information on the specified file that it becomes non-commutative with almost any access to that same file—and most of the information is ignored by any particular caller. Providing a way to return only the information of interest would greatly increase commutativity, and should, in theory, increase scalability.

In practice, it is not clear that the non-commutativity of fork() and stat() are the most urgent bottlenecks in the Linux kernel. On the other hand, Figure 7 of the SOSP paper does show some benchmark results that indicate that the Linux kernel does have some potential room for improvement.

Embrace specification non-determinism. Interestingly enough, this was one of the loopholes to the Laws of Order, which reappears here as a way to improve the commutativity of APIs in order to permit more highly scalable implementations. Here, the authors take aim at the POSIX standard's insistence that open() and friends return the lowest-numbered available file descriptor, which was mentioned above. Although this restriction was helpful back in the days of 16-bit Unix systems that lacked the dup2() system call, for almost all of us, those days are thankfully long gone. This requirement makes a pair of concurrent open() calls non-commutative, thus imposing a limit on scalability. In contrast, if open() and friends were permitted to return an arbitrarily selected file descriptor, then file descriptor allocation would not necessarily impose a scalability bottleneck.

Permit weak ordering. This was another of the loopholes to the Laws of Order, which reappears here as another way to improve the commutativity of APIs in order to permit more highly scalable implementations. The paper gives the example of strict ordering of messages sent on Unix-domain sockets, especially for the SOCK_DGRAM sockets where the user might have neither need nor desire for any sort of ordering. Presumably, weak ordering would also help in the case of the stat() system call.

Release resources asynchronously. Asynchronous freeing of memory is, of course, used heavily by read-copy update (RCU), but the authors note that it also applies in many other cases. For example, an munmap() system call executed in a multithreaded process requires synchronous TLB shootdowns in order to prevent other threads from doing “doomed” writes to memory that is in the process of being unmapped. This, in turn, means that the munmap() system call cannot commute with a memory write to the unmapped region, again imposing a limit on scalability. The authors suggest that an madvise() flag be provided to allow asynchronous munmap(), increasing commutativity and, in turn, potentially increasing performance and scalability.

Layer scalability. The authors note that their research operating system's filesystem uses scalable data structures with naturally commutative operations. Doing this at each layer of the software stack would clearly make it easier to get good results further up the stack.

Defer work. To the best of my knowledge, this good advice was first stated in Alexia Massalin's dissertation [.ps.gz]. However, the advice bears repeating, and the authors describe its application to generating unique identifiers (specifically inode numbers). Their approach uses a per-CPU counter concatenated with the ID of the CPU generating the identifier. This approach ensures that these identifiers never repeat, and thus that each CPU can safely generate inode numbers without any sort of synchronization with the other CPUs. Deferring the synchronization ensures that identifier generation need never be a scalability bottleneck.

Compatibility considerations might make this approach to identifier generation difficult to apply in some cases, but the general approach of deferring work can be quite useful. For but one example, work deferral is used heavily by RCU.

Precede pessimism with optimism. The idea here is to have a highly scalable optimistic stage followed by a less scalable, more general, and hopefully rarely used pessimistic stage. The paper gives double-checked locking as one example, and handling of special cases of the rename() system call as another. However, many other examples of this approach come to mind, for example, the per-CPU-cached memory allocators within the Linux kernel. This design approach has also been called “parallel fastpath”, for example here and in Section 6.4 of my book, Is Parallel Programming Hard, And, If So, What Can You Do About It?.

Don't read unless necessary. Most kernel hackers avoid unnecessary writes in order to avoid the resulting cache misses, but the authors point out that unnecessary reads can also limit scalability, especially if safely carrying out the read requires acquiring locks. They give pathname lookup as an example, pointing out that a typical implementation of the old-school Unix namei() function (lookup_fast() and lookup_slow() in the Linux kernel, though still living in namei.c) returns the inode in addition to checking whether the pathname exists. The fact that it returns the inode makes it non-commutative with a number of other operations, including rename(). The sv6 research OS's ScaleFS therefore provides a separate internal interface that checks for the pathname without accessing the inode in order to avoid this potential scalability bottleneck.

This is a good list of scalability techniques, and most of them focus on promoting commutativity, at least in the common case. However, embrace specification non-determinism, permit weak ordering, and defer work represent a different approach that is discussed in the next section.

Scaling despite non-commutativity

The scalable commutativity rule states that whenever interface operations commute, they can be implemented in a way that scales. It is tempting to assume that the inverse rule also applies, namely, that whenever interface operations fail to commute, there can be no scalable implementation. However, the inverse does not always apply, as evidenced by the following exceptions:

1. If the hardware provides a register containing a fine-grained, strictly increasing time, objects could be scalably “updated” by using a timestamped per-thread log.
2. If the hardware allows certain communications patterns to scale, then non-commutative operations using those patterns might scale. In our search-structure example, a hardware content-addressable memory might allow a search structure to be constructed that scaled well even with concurrent operations on the same key.
3. If the algorithm can tolerate weak ordering, then permit weak ordering in order to decouple mechanisms that would otherwise impose a non-commutativity scalability bottleneck on the system.
4. If the algorithm can tolerate non-determinism, then embrace specification non-determinism in order to decouple mechanisms that would otherwise impose a non-commutativity scalability bottleneck on the system.
5. If the algorithm can tolerate latency, then defer work in order to avoid scalability bottlenecks by delaying potentially conflicting operations until there is no longer the potential for contention.

There have been any number of efforts to provide hardware acceleration in order to take advantage of the first and second exceptions. RCU and some forms of scalable counters take full advantage of the third, fourth, and fifth exceptions. The first and second exceptions to the inverse are also exceptions to the scalable commutativity rule itself because they invalidate hardware assumptions underpinning the formal proof of that rule.

Given that both RCU read-side critical sections and counter addition are commutative, it is quite reasonable to ask why they would need to take advantage of any of these exceptions.

For RCU, the problem is that although RCU readers commute with each other, they do not commute with grace periods: If a given rcu_read_lock() executes just before a given synchronize_rcu() begins, then that synchronize_rcu() must wait on the RCU read-side critical section started by that rcu_read_lock(). If the order of these two operations is reversed, then synchronize_rcu() need not wait. RCU can nevertheless avoid scalability bottlenecks by delaying both the beginnings and the ends of grace periods. This permits batching, which allows the overhead of a given grace-period operation to be amortized over an arbitrarily large number of updates, thus providing excellent scalability despite the non-commutativity.

Similarly, although a pair of additions to a given counter are commutative, additions to a counter do not commute with reads from that counter. Also, if you run a “scalable” counter on a large system with (say) 50% updates and 50% reads, the results probably will not be pretty and definitely will not be scalable. However, this scalability bottleneck can be avoided by allowing counter reads to return outdated values for the counter, in effect adding a delay between the time an update happens and the time it is visible to readers. This added delay allows readers to fetch the value of a single counter that is periodically updated at a rate low enough to avoid scalability problems, as described in Section 5.2.4 of Is Parallel Programming Hard, And, If So, What Can You Do About It?.

Conclusions

This does raise a question as to how severe non-commutativity problems really are and, of course, one ready answer is: “Clearly not severe enough to have been addressed yet”. That said, one great benefit of this work is that it allows developers to more easily determine whether a bottleneck is inherent in the API, or whether it is the fault of the implementation. Another welcome aspect of this paper is the discussion of ways to avoid non-commutativity bottlenecks, as is the demonstration of the benefits of avoiding these bottlenecks in the guise of the sv6 research OS. Finally, this paper has the virtue of making it quite clear exactly what is and is not commutative.

If I was to identify a shortcoming of this paper, it would be that the existence of non-commutativity (or perhaps non-scalable non-commutativity?) does not necessarily translate into a serious bottleneck. For example, a given instance of non-commutativity might only become a bottleneck for 1,000,000-CPU shared-memory systems, which currently face other types of bottlenecks so severe that to the best of my knowledge, no such system exists. It might therefore make no sense to eliminate this instance of non-commutativity. However, this is by no means a fatal flaw. It instead indicates that it is necessary to consider commutativity to be but one of many inputs to the task of creating a good API design, rather than blindly doing whatever the Commuter tool says to do. And perhaps this will be addressed in future work.

I nevertheless expect this important work to inform future API design, allowing future implementations to avoid a great many scalability bottlenecks. Or, at least, to avoid those bottlenecks that are not mandated by standards.

Acknowledgments

I owe thanks to Robert Morris and Austin Clements for their careful review and thoughtful comments. I am grateful to Jim Wasko for his support of this effort.

Answers to Quick Quizzes

Quick Quiz 1: Why bother with commutativity? Why not just partition the problem and get on with life?

Answer: It turns out that partitioning is in fact one way to promote commutativity. After all, when you partition the problem, operations acting on different partitions can happen in any order without changing the result. So partitioning is one way to achieve commutativity, but there are also other valuable techniques.

Back to Quick Quiz 1.

Quick Quiz 2: That is quite an improvement, when will these changes be applied to the Linux kernel?

Answer: Hopefully sooner rather than later.

That said, the authors did use some implementation techniques that might not be unanimously welcomed by Linux users. For one example, they completely dispensed with mass storage in favor of RAM-based filesystems (which might well be addressed in future work). In addition, some of the bottlenecks identified in Linux will be more important than others.

Back to Quick Quiz 2.

Quick Quiz 3: Isn't posix_spawn() perfectly commutable with everything?

Answer: Close, but not quite. The list of things that can fail to commute with posix_spawn() includes: exit(), fatal signals, writes to the memory containing the pathname of the executable, open(), and close(), among many others. Nevertheless, posix_spawn() is considerably more commutable than is fork().

Too bad about those software-engineering code-compatibility concerns.

Back to Quick Quiz 3.

Quick Quiz 4: But if the point of avoiding the read is to avoid acquiring a lock, aren't we really avoiding writes, not reads?

Answer: You can certainly look at it that way. Without the locks, namei() would make use of the permit weak ordering exception called out above, at least assuming that it avoided crashing in the process. However, the fact remains that names like lookup_fast() do not connote “write” to most people.

Interestingly enough, Linux's lookup_fast() can avoid most writing if it is able to stay on the fastpath. However, pathname coherence is guaranteed by a read-seqlock operation, which could, in theory, inflict a scalability bottleneck on lookup_fast() in the face of frequent path rearrangements. Such rearrangements could (again, in theory) cause the read-seqlock to always fail, limiting lookup_fast() even without considering any lookup_fast() writes.

So reads can result in scalability problems. Admittedly indirectly, and caused by someone else's writes, but the point still stands: reads really can harm your scalability and performance.

Back to Quick Quiz 4.

Quick Quiz 5: But the first two exceptions depend on hardware features that are not universally available, so why would anyone want their software to depend on such features?

Answer: Theory, meet portable practice.

That said, it all depends on the situation. The more you need the additional scalability, the more willing you will be to depend on specific hardware. And perhaps now that multicore hardware has well and truly arrived, additional hardware scalability optimizations will become available. Stranger things have happened.

Back to Quick Quiz 5.

Comments (29 posted)

Reproducible Android app builds

The subject of reproducible builds is a popular topic of late in the free-software world, with major projects—such as Debian and Tor—making it a high priority. Although compiling verifiable, reproducible binaries on desktop and server systems is a significant engineering problem, there is even more complexity to the problem on mobile platforms, where a single vendor can act as gatekeeper and can impose rules on the app-distribution model that get in the way. Nevertheless, F-Droid and the Guardian Project have undertaken an effort to make reproducible builds feasible for Android apps, so that users can independently establish that the packages they install correspond to the source code that is published.

The Guardian Project develops security- and privacy-enhanced mobile apps: encrypted text chat, encrypted voice calling, Tor-based web browsing, and so forth. Although there are a few Guardian Project applications that run on other platforms, most of its apps target Android. All of the source code is available, but—naturally enough—the users for whom security and privacy are most critical may also have cause to be suspicious that an app installed via the Google or Amazon app store has been compromised.

This is where F-Droid, an entirely free-software Android app repository, comes into the picture. Every app package published in the F-Droid repository can be built from source, because only apps under free-software licenses are allowed. Thus, choosing apps from F-Droid does make it theoretically possible for users to verify not just that the package has not been tampered with since it was built, but that the app's binary corresponds to a specific source code release. Of course, a similar guarantee would accompany having users compile apps locally themselves, but that defeats the purpose of having an app repository altogether.

Furthermore, Android apps are supposed to carry a signature from the developer, not from the repository. Historically, F-Droid has used per-app signing keys to sign the .apk packages that it distributes—but those signatures are, in a sense, weaker than a signature from the upstream app developer, since some sort of alteration could have been made to the code by the repository. If it could be shown that the upstream source release compiles into a binary that is bit-for-bit identical to the one served up by the repository, though, that would also allow the user to rest assured that no malicious code has been inserted and nothing important has been removed or altered.

This is essentially the approach under development. The F-Droid publication framework downloads a signed binary app from the upstream developer, then builds a new .apk itself from the upstream source bundle and build recipe. The publication system then compares its (unsigned) .apk to the payload portion of the .apk bearing the upstream signature. If the two match, then users can see that the code was not tampered with during the F-Droid's build process. There are several tools available for checking the signature of an Android app, though the feature is not exposed during normal installation.

For this process to work, of course, both the upstream app developer and the F-Droid build system must use the same reproducible build process to create the .apk file in question. The Guardian Project has been working on this task since early 2014. In June of that year, the first reproducible build was released: Lil'Debi version 0.4.7.

Lil'Debi is a developers' tool, not an end-user application; it bootstraps a chroot-ed Debian environment on an Android phone. But the exercise proved that the deterministic build process was possible, with a few caveats—namely, that compatible versions of the JDK and other build tools be used on each system. The build process requires a few additional precautions to produce bit-for-bit identical builds. Specifically, the contents of the .apk file must be sorted into the same order for each build, and faketime must be used to ensure that the timestamps match. After Lil'Debi's 0.4.7 release, the Guardian Project released a shell script for users to compare two .apk files with.

On February 11, the F-Droid published its first verifiable, reproducible Guardian Project app: Checkey, a tool for checking Android app signatures. At present, the Checkey .apk provided through the F-Droid repository is the binary built and signed by the Guardian Project; the package has just been verified to generate the same hash as the binary built by F-Droid. The long-term plan, however, is to deliver the F-Droid binary with the upstream project's signature extracted and then re-attached, then signed a second time with the F-Droid signing key. GnuPG supports checking multiple signatures in a single file, so long as the same cipher preferences are used for both signatures.

Moving forward, the Guardian Project's LocationPrivacy app and the LEAP Encryption Access Project's BitMask app are the next two apps to be built using the reproducible build process. Ultimately, the F-Droid project hopes to make its reproducible build system available to users at large, so that anyone can independently verify that an .apk package has not been altered between its upstream release and when it was downloaded by the user.

Android users who are concerned about app integrity may not amount to a majority in the mobile application space, but the special cases of today often have a way of becoming the widespread concerns of tomorrow. Even if F-Droid remains the only mobile-app distributor to offer verifiable app packages, users stand to benefit from having somewhere to turn for increased security.

Comments (none posted)

Distribution quotes of the week

Comments (none posted)

End of the m0n0wall project

Full Story (comments: none)

DebConf15: Call for Sprints

Full Story (comments: none)

Debian Trademark Team

Comments (none posted)

Distribution newsletters

• Debian Project News (February 16)
• DistroWatch Weekly, Issue 597 (February 16)
• 5 things in Fedora this week (February 13)
• 5 things in Fedora this week (February 18)
• Ubuntu Weekly Newsletter, Issue 404 (February 15)

Comments (none posted)

Linux for Astronomers (Linux Journal)

Comments (9 posted)

Scanning Python programs with Bandit

Using a program written in a particular language to search for problems in another program written in that same language has a certain amount of charm. High-level languages, such as Python, can actually give programmers easy access to the data structures built by the parser, so that these analysis programs can see the program exactly as the interpreter does. An OpenStack sub-project called Bandit does exactly that. It uses the Python abstract syntax tree (ast) module to seek out potential security problems in Python code.

OpenStack is a free-software cloud platform that is written in Python—over a million lines of it—so it makes sense for the project to have tools to look for problems in its code. Bandit comes from the OpenStack Security Group and is specifically targeted at finding security flaws in the OpenStack—or any other Python—code base. It is structured as a framework that reads in and parses the source code, then runs through a battery of tests, implemented as plugins, to seek out the use of dubious Python modules and functions, potentially dangerous code constructs, and more.

The README file outlines how to clone and install Bandit into a Python virtual environment, though it can also be installed without the virtual environment in the usual fashion:

    $ python setup.py install

    $ bandit examples/*.py

Running Bandit on a moderately sized body of Python code—one that runs a nearby web site—produced an 800-line report that mostly contained false positives. That is not really a surprise for a couple of reasons. For one, the tool and tests are geared toward OpenStack and how it uses Python and its modules. For example, any strings that looked like SQL were flagged as SQL injection warnings—unless the SQLAlchemy module is imported (in which case the severity is reduced to informational). That may make sense for OpenStack, but other projects may have their own mechanisms to avoid SQL injection.

In addition, one totally unrelated string that happened to contain both "select" and "from" triggered the warning. The tests would need to be much more sophisticated to avoid that kind of false positive.

Beyond that, though, a tool of this nature needs to be pretty conservative about skipping over potential problems. That means it will alert on lots of things that aren't actually vulnerabilities, but require a human to determine that. Thus, false positives.

But the list of warnings definitely had some things to look into. For example, there are complaints about network services that may be binding to all interfaces on the system, which may be unexpected. Use of the os.popen() function is generally frowned upon and its use requires some investigation. Use of the random module "is not suitable for security/cryptographic purposes", which is probably not how it is being used, but it should be looked into. And so on.

Beyond the SQL injection complaints, the most common warning was for the use of the pickle module, which can have security (and maintenance) implications. Its use in this code base should be safe (though it is something we plan to replace for other reasons at some point). Disabling those top two warning types resulted in a much shorter report, with only 14 warnings to investigate.

While disabling the pickle test was easy to do, by commenting out some lines in the bandit.yaml configuration file, turning off the SQL injection tests was not quite as straightforward. Since many of the tests are plugins, moving the injection_sql.py file aside in the install plugins directory worked well enough to disable it, but was rather inelegant. A more proper solution was to set up a profile in the configuration file that excluded the hardcoded_sql_expressions test.

Another option is to go through the source and annotate any line that causes a false positive with a "# nosec" comment. One could also filter the results based on the level of the complaint (i.e. information, warning, or error) though, in this case, that would likely mask some problems. Those with an aversion to colored output will want to use the -o file option to send the output to a file (thankfully without the color escape sequences).

Taking a spin through the examples directory will give a good overview of the kinds of things Bandit is looking for. As noted earlier, much of it is advisory in nature, like complaints about calling eval() (which is a warning, though calling exec() is considered an error), use of os.exec() and os.spawn(), use of the MD5 hash function, or any strings from a list of often-used default passwords. Beyond that, there are warnings about urllib.urlopen() ("Audit url open for permitted schemes. Allowing use of file:/ or custom schemes is often unexpected."), yaml.load() (recommending yaml.safe_load() in its place), mark_safe() (a potential cross-site scripting vector), and so on. Any of those could be security problems or completely innocuous, but someone needs to look to be sure.

OpenStack plans to add Bandit into its continuous integration system, and eventually may have it actually reject patches that don't pass. That is still probably a ways off based on the kinds of warnings it is producing now. Those classified as errors might be suitable, however. As with most projects, though, there is a pretty long to-do list on the web page.

For other projects, Bandit will likely take some tweaking to reduce the false positive rate based on the project's coding style and typical Python usage. That should be pretty easy to do, as the framework provides much of what is needed to write these types of tests. Adding new tests (or modifying existing ones) is straightforward, with plenty of code to look at for ideas. Bandit is well worth a look for Python-based projects.

Comments (none posted)

Quotes of the week

Comments (2 posted)

Wayland 1.7.0

Comments (34 posted)

PulseAudio 6.0 available

Version 6.0 of the PulseAudio framework has been released. New is support for several Bluetooth audio profiles under BlueZ 5, socket activation using systemd, and improved support for multi-channel audio profiles.

Full Story (comments: none)

cmocka 1.0 available

Version 1.0 of the cmocka unit-testing framework has been released. Notably, the 1.0 release features a new test runner that allows tests to be grouped, although there are many other updates and improvements to be found.

Full Story (comments: none)

SystemTap 2.7 available

Version 2.7 of the SystemTap instrumentation system has been released. New in this release are several changes to the tracepoint probe behavior, speed-ups when passing string data to and from functions, and a great many changed tapsets and scripts. The new release also happens to coincide with SystemTap's 10th anniversary as a project.

Full Story (comments: none)

Development newsletters from the past week

• What's cooking in git.git (February 11)
• What's cooking in git.git (February 17)
• GNU Toolchain Update (February)
• LLVM Weekly (February 16)
• OCaml Weekly News (February 17)
• OpenStack Community Weekly Newsletter (February 13)
• Perl Weekly (February 16)
• PostgreSQL Weekly News (February 15)
• Python Weekly (February 12)
• Qt Weekly (February 17)
• Ruby Weekly (February 12)
• This Week in Rust (February 17)
• Tor Weekly News (February 18)
• Wikimedia Tech News (February 16)

Comments (none posted)

Routing on OpenStreetMap.org

The OpenStreetMap blog announced the roll-out of point-to-point direction routing on the main OSM site. Historically, OSM has encouraged other developers to build this sort of functionality on top of the OSM data set, but has presented a limited interface on its own site. "But we have a saying: “what gets rendered, gets mapped” – meaning that often you don’t notice a bit of data that needs tweaking unless it actually shows up on the map image. Lots of things aren’t shown on our default rendering, so the feedback loop offers less incentive for people to get them correct. And that goes doubly for things that you never “see” on the map – subtle things like “no left turn” at a particular junction, or “busses only” access on a tiny bit of road, or tricky data issues like when a footpath doesn’t quite join a road that it should join on to. Now that people can see a recommended route directly on the OSM homepage, they have an incentive to quickly pop in and fix little issues like that. The end effect will be OSM’s data going up one more level in terms of its quality for routing."

Comments (none posted)

Where do we stand 30 years after the founding of the Free Software Foundation? (Opensource.com)

Comments (7 posted)

CFP Deadlines: February 19, 2015 to April 20, 2015

If the CFP deadline for your event does not appear here, please tell us about it.

Events: February 19, 2015 to April 20, 2015

If your event does not appear here, please tell us about it.