XBMC on Android != XBMC for Android

The XBMC media center front-end runs on a wide variety of platforms—most recently Android, as we saw in January. But as the project discovered recently, not everything advertised to the public as "XBMC" is its work. Those who follow the project closely are unlikely to be confused, but the wider marketplace is another matter. Moreover, XBMC developers have had to spend their time and energy on the clarifications, which is quite understandably a source of frustration.

At issue is a spin-off of XBMC which calls itself "XBMC for Android." Shortly after the first releases of the official port of XBMC to the Android platform, the spin-off project announced its own release, which it tagged "the first end user friendly release of XBMC for Android" in a blog post that makes no mention of the fact that it is not the work of the official XBMC project. That blog announcement was picked up by a number of online news sources, including the popular consumer electronics blog Engadget. XBMC contacted all of the sites running the story, and all of them corrected their copy to note that XBMC for Android is not affiliated with the official XBMC project.

Then in May, the cycle repeated itself. The XBMC project released its latest update, version 12.2, on May 3, and on May 4 the XBMC for Android project announced its release, which was again picked up by Engadget, who again had to alter the story after an email from the genuine XBMC project. That prompted XBMC's Nate Thomas to post a story on the official XBMC blog titled "This Site is the Only Official Source of XBMC Software." Initially Thomas was upset that Engadget had not corrected the second story, although it did so later (which Thomas noted in the post).

But the bigger issue remains that a fork of the official project exists which has chosen to brand itself with a name confusingly similar to that of the upstream project. A commercial entity with a registered trademark to protect might well pursue legal action in such a situation, but the best approach for a GPL-licensed software project is considerably less clear. XBMC does not have sales to protect, of course, and in fact the project is expressly open to working with commercial XBMC derivatives (at the moment, Plex and Voddler; formerly Boxee as well). But it does undertake the costs of supporting users, a task which is greatly complicated when the user had unknowingly installed a modified version of the software.

What's in a (my) name?

How a project's name can be used by other parties is a tricky affair, and it is not as well-understood in free software circles as is copyright. As Karen Sandler explained in her SCALE 8x talk about trademarks, a project does not have to register a trademark (at least in the United States) in order to protect it. Rather, a trademark carries weight based on how well it establishes a brand in the minds of actual users. XBMC does not advertise a trademark on the name XBMC, nor does it have an explicit trademark-usage policy, but if it did feel that its brand was being exploited, it would still be able to make that case before a court. And it might have the structure in place to do so—which not every project would. In January 2009, the project formed the XBMC Foundation to handle donations and sponsorships, and it is represented by the Software Freedom Law Center.

However, as Sandler also discussed in her talk, there is such a thing as the acceptable "nominative use" of someone else's trademark: it is a protected use of another party's trademark to refer factually to the trademarked product. That is why Ford cannot stop Chevrolet from mentioning it in its own TV commercials. So advertising "Foo for Android" would probably not be considered a violation of the Android trademark by a court, if it is clear that the "for Android" part was a factual reference to the platform and not part of the product's name.

That particular parsing exercise is not quite as clear for XBMC for Android, however. The spin-off project does include a disclaimer on its About page, reading "XBMCANDROID.COM is not connected to or in any other way affiliated with XBMC, Team XBMC, or the XBMC Foundation." That is a plus; seemingly an addition made to the page since January. But it does not go very far; the same page states:

That wording suggests some sort of collaborative relationship between the two projects. Perhaps more to the point, the announcements picked up by Engadget and other online news sources are a lot less clear about a distinction between XBMC for Android and the official XBMC release; for example they go back and forth between referring to the download as "XBMC for Android" and "XBMC." A comment on the official XBMC blog post claiming to be from an XBMC for Android team member says a disclaimer was also included in the press release sent to Engadget, although it does not appear in the blog announcement.

A little less ambiguation

Assuming there is no intent to confuse, however, the spin-off project could still stand to do a lot more clarification. The links and other site content are as inconsistent about the name of the software as were the announcements. The site sports the official XBMC logo at the top of the page (and the Android logo) as well. But ultimately the most confusing part of the spin-off project is that fact that it has not selected a unique name for itself. It clearly could have, just as its team members clearly could have chosen to work on their own branch within the official XBMC project.

Historically, name collisions are a pretty rare event in open source software; the major ones can probably be counted on one hand. But the reason is not that open source projects fiercely protect trademarks (much less unregistered ones). Instead, it is the simple practical matter of minimizing confusion. Name collisions make packaging and distribution harder, and when they occur the bigger and better-known project generally has such an advantage that the fork is eventually forced by circumstance to rename itself.

For the time being, XBMC seems content to deal with this particular situation as a public relations matter. As the larger, more well-established project, the odds are good that events will resolve in its favor. But the issue will not disappear entirely; even within the comments on the XBMC blog post, there is a message from yet another XBMC fork, which calls itself "XBMC Android TV." Sadly, it probably would not hurt to double check the next announcement you see about XBMC—if it does not originate directly from xbmc.org.

Comments (1 posted)

DRM in HTML5 published in draft form

Despite the protests of multiple critics, the World Wide Web Consortium's (W3C) HTML working group has published the first working draft of the Encrypted Media Extensions (EME) proposal. The EME proposal is commonly referred to as "DRM for HTML" or words to that effect, although that is not a strictly accurate description. Reading through the proposal, however, may only cloud the real issues at stake. EME merely defines an API for browsers to pass messages back and forth between a content server and DRM module. But those DRM modules are little different from the closed plugins used to deliver DRM today—except that the EME proposal would bear the W3C seal of approval. Of course, the proposal must first make it all the way through the W3C's labyrinthine standardization process, and whether W3C endorsement makes any real difference for users depends on who you ask.

The EME proposal was released on May 10. Its exact status it a bit complicated; it is labeled a "public working draft," which designates the lowest level of maturity in the W3C development process. Officially, a working draft is "a signal to the community to begin reviewing the document;" hypothetically, after several working drafts, a "last call announcement" is published, followed by "candidate recommendation," "proposed recommendation," and (finally) "recommendation"—with reviews, revisions, and red tape separating each stage. Still, the publication of this initial draft does signal that EME is moving forward, in spite of objections to the core concept from advocates of software freedom, privacy, and related matters.

The EME draft has three editors: Google's David Dorwin, Microsoft's Adrian Bateman, and Netflix's Mark Watson. Also noteworthy is that although the proposal is officially a product of the HTML working group, that means only that it impacts HTML elements; the EME authors operate as an effort distinct from the HTML working group's larger activities.

An uncomfortably close look

The EME proposal defines two abstract components: the Content Decryption Module (CDM) and the Key System. A CDM is a browser component (either built-in or installed as an extension) responsible for authenticating a user session with a remote content server and for decrypting the media served up by the server. A Key System is a complete implementation of an access-control framework in all of its constituent parts: the remote media streaming server, its standards for session and key management, and whatever else the provider chooses to implement.

Both are essentially black boxes as far as the browser is concerned; the CDM can decrypt encrypted media frames and hand them right back to the browser's playback stack, or it can render them directly to hardware—the EME proposal does not address this functionality. Similarly, a Key System is identified by a reverse domain name (e.g., com.example.payperviewservice), and EME outlines several interfaces for session management messages, but the key format and encryption method are left up to each implementer, as are session IDs and other initialization data.

In practice, a playback scenario would involve a JavaScript application that provides the local user interface and connects to the Key System to fetch content for an HTML mediaElement (that is, a <video> or <audio> element on the page; EME does not currently affect any other HTML elements), then passes that content to the CDM, which both authenticates the user and decrypts the content. It is the hooks for these JavaScript applications that make up the meat of the EME proposal. There is a new keys attribute for mediaElements, setMediaKeys and createSession methods which the application uses to get key and session information from the CDM, and a message event which the application uses to relay messages from the CDM to the Key System.

For instance, see the proposal's "other CDM" example, which depicts a mediaElement:

    <video src="foo.webm" autoplay onneedkey="handleKeyNeeded(event)">
    </video>

In the example, the video element triggers the EME script with the new onneedkey event. The script first initializes the video.keys attribute with the hard-coded "com.example.somesystem.1_0" Key System, then initializes the keySession, which tells the CDM to generate a key request message. The script then fires the request message off via XMLHttpRequest(); if the Key System responds with a key, the keySession is updated, and playback is started or resumed. The encrypted audio or video frames from the server are handed to the CDM, which decrypts them with the key and renders them as needed. There are several scenarios not addressed in this example, such as the availability of multiple CDMs in the browser (where the application might need to do some work to select the correct CDM).

From outside, this initialization and handshake process is pretty simple. One will note that the keySession is initialized with initData, the contents of which is not specified by the EME proposal—the CDM and the Key System determine what this data is (and how it authenticates the session). Likewise, the format of the key and the decryption of the media frames is pushed entirely out to the CDM and the remote Key System server.

What is left to the browser is a generic session initialization system and messaging framework, tied into <video> or <audio> elements with some new attributes and methods. As such, it does not directly mandate a DRM implementation. Actual DRM schemes would be implemented in CDMs, which are assumed to be resistant to key extraction, modification, and other methods that could be used to access mediaElement content outside of the parameters the CDM enforces. Similarly, the API between a CDM and the browser is not defined.

Feedback

So far, of course, there are no CDMs in the wild. On the other hand, there are "non-normative" (i.e., informational) sections in the proposal that discuss DRM details with more specificity. Section 7.1 defines an initialization data format and the encryption scheme for WebM content, and section 7.2 does the same for the ISO Base media File Format (ISOBMFF, the generic underpinnings of .MP4). Both use AES-128 in "counter" mode (CTR) for encryption; the WebM initialization data is a key ID, while the ISOBMFF initialization data is a more complicated structure that can hold separate keys for separate tracks. Section 6 also defines a trivial Key System named org.w3.clearkey (the only Key System defined in the document) which is mandatory for conforming browsers to implement. However, this clear key scheme has little practical value: it sends a decryption key in the clear as a JSON element.

But if the EME proposal does not actually enshrine a DRM scheme into the HTML specification, one might reasonably ask what all the fuss is about. After all, the Free Software Foundation denounced the publication, as did the Electronic Frontier Foundation.

The answer is two-fold. First, at a technical level, it is true that EME does not itself include a (non-trivial) DRM scheme. But EME's functionality hinges entirely on the presence of CDMs in the browser, which users by definition will have no control over. To software freedom advocates, that means handing control of a user's machine to proprietary code—which could very well contain anything from unintentional security flaws to secret user tracking frameworks. Whether built in to the browser or provided as separate downloads, only binary CDMs will be trusted: if users can compile their own CDM, they can alter it to include workarounds (e.g., to save content or encryption keys).

The same is true today, of course, which is why content services deploy protected content via proprietary plugins like Flash and Silverlight—in that sense, reliance on binary CDMs instead of binary plugins is at best a lateral move. Google and Microsoft very well could include binary CDMs in their browsers, but open source browsers like Firefox cannot, so they would be relying on third-party CDMs. And as Mozilla's Henri Sivonen observed (forwarded by Gervase Markham), support for binary plugins for Linux and other minority desktop operating systems is not good, and binary CDMs are certainly unlikely to be better. In fact, Sivonen doubts open source browsers will be well-served even on other OSes:

The other objection to the EME proposal is that it fundamentally goes against the stated mission of the W3C, both by advocating the distribution of proprietary CDMs in browsers and by endorsing restricted content itself. The EFF criticism of EME hits on this point, noting that the EME scheme is a mechanism for sites to require proprietary software or hardware in order to render web content:

The Free Culture Foundation argued that the proposal goes against the W3C mission statement, which is to make the benefits of the Web "available to all people, whatever their hardware, software, network infrastructure, native language, culture, geographical location, or physical or mental ability."

In addition, HTML working group member Manu Sporny argues that the proposal runs afoul of other W3C ideals, because it is advertised as a solution to prevent the piracy of copyrighted content, which he claims it simply does not do. Sporny also contends that in order for EME to be effective, sites will need to undertake expensive measures like re-encrypting each title for every user, which is a poor value proposition.

The EME proposal does have a few supporters outside of the drafters of the document itself, though. Ars Technica's Peter Bright declared it a "victory" for the open web, on the grounds that it keeps HTML relevant to the critical big-media producers, who would otherwise switch to non-Web delivery mechanisms—such as the DRM-capable media services already provided by iOS and Android. It is certainly true that content services looking to make money from audio and video—particularly through rental and subscription services—are not likely to embrace unencrypted mediaElements in the browser, but by shuttling it directly from the server to a binary-only CDM, there is not a whole lot of "open web" involved in the transaction; a fact Bright seems to gloss over.

Naturally, any company interested in deploying DRM-restricted multimedia as a product is going to do so whether it gets a W3C stamp of approval or not. If affecting that outcome is the goal of a protest against the EME proposal it will likely have no impact. But there is still an argument to be made that the W3C should not play a part in the process. It might seem like an uphill battle; W3C Director Tim Berners-Lee approved the EME proposal publication and has stated his position in favor of allowing DRM.

In addition, HTML working group co-chair Paul Cotton has been very vocal about shutting down criticism of EME on the mailing list, telling everyone with a complaint to take it to the Restricted Media Community Group (in which Netflix's Watson is the only EME editor who participates). But vocal opposition to W3C behavior has been effective in the past. Sivonen noted that the W3C patent policy was revised due to opposition from free software advocates. The draft policy endorsed "reasonable and non-discriminatory" (RAND) patent licenses, but feedback from Mozilla and others eventually shifted the policy to a fully royalty-free model. Whether history repeats itself, only time will tell.

Comments (15 posted)

A look at the PyPy 2.0 release

It's hard to say why, but May appears to be the month where we look in on PyPy. Three years ago, we had a May 2010 introduction to PyPy, followed by an experiment using it in May 2011. This year, the PyPy 2.0 release was made on May 9—that, coupled with our evident tradition, makes for a good reason to look in on this Python interpreter written in Python.

One might ask: "Why write a Python interpreter in Python?" The possibly surprising answer is: "performance". It's not quite as simple as that, of course, as there are some additional pieces to the puzzle. To start with, PyPy is written in a subset of Python, called RPython, which is oriented toward making a Python dialect that is less dynamic and acts a bit more like C. PyPy also includes a just-in-time (JIT) compiler that flat out beats "normal" Python (called CPython) on a variety of benchmarks.

PyPy has been making steady progress for over ten years now, and has reached a point where it can be used in place of CPython in lots of places. For a long time, compatibility with the standard library and other Python libraries and frameworks was lacking, but that situation has improved substantially over the years. Major frameworks like Django and Twisted already work with PyPy. 2.0 adds support for Stackless Python with greenlets, which provide micro-threading for Python. Those two pieces should allow asynchronous programs using gevent and eventlet to work as well (though gevent requires some PyPy-specific changes).

In order to support more Python modules that call out to C (typically for performance reasons), PyPy now includes CFFI 0.6, which is a foreign function interface for calling C code from Python. Unlike other methods for calling C functions, CFFI works well for both CPython and PyPy, while also providing a "reasonable path" for IronPython (Python on .NET) or Jython (Python on the Java virtual machine).

Trying it out

Getting PyPy 2.0 is a bit tricky, at least for now. Those who are on Ubuntu 10.04 or 12.04 can pick up binaries from the download page (as can Mac OS X and Windows users). While many distributions carry PyPy in their repositories, 2.0 has not arrived yet. There are "statically linked" PyPy binaries, but the 64-bit version (at least) doesn't quite live up to the name—it requires a dozen or so shared libraries, including older versions of libssl, libcrypto, and libbz2 than those available for Fedora 18.

Normally, given constraints like that, building from source is the right approach, but the project has some fairly scary warnings about doing so. According to the docs, building on a 64-bit machine with 4G or less of RAM will "just swap forever", which didn't sound all that enticing. But there is a workaround that doesn't use CPython and instead requires PyPy using some magic incantations (an environment variable and command-line flag) to limit the memory usage—but that means making the static PyPy binary work. A little symbolic linking (for libbz2) and some library building (openssl-1.0.0j) resulted in a functioning PyPy. There is no real reason not to use that, but I was a little leery of it and curious to continue with the build process.

Running the PyPy build is a rather eye-opening experience. Beyond the voluminous output—including colored-ASCII Mandelbrot set rendering, lots of status, and some warnings that are seemingly not serious—it took more than 2 hours (8384.6 seconds according to the detailed timing information spit out at end of the build process) on my not horribly underpowered desktop (2.33 GHz Core 2 Duo). The Linux kernel only takes six minutes or so on that system.

Calculating the Mandelbrot set while translating is not the only whimsical touch that comes with PyPy. Starting it up leads to a fortune-like quote, though one with a PyPythonic orientation:

    $ pypy
    Python 2.7.3 (b9c3566aa0170aaa736db0491d542c309ec7a5dc, May 11 2013, 17:54:41)
    [PyPy 2.0.0 with GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    And now for something completely different: ``PyPy is a tool to keep otherwise
    dangerous minds safely occupied.''
    >>>> 

As a quick test of PyPy, I ran an amusingly shaped Mandelbrot program in both CPython and PyPy. As expected, the PyPy version ran significantly faster (just over 3 minutes, compared to 8+ minutes for CPython 2.7.3). In addition, the bitmaps produced were identical.

PyPy comes with its own set of standard library modules, but additional modules can be picked up from an existing CPython site-packages directory (via the PYTHONPATH environment variable). Trying out a few of those (BeautifulSoup 4 for example) showed no obvious problems, though a PyPy bug report shows problems using the lxml parser, along with some other, more subtle problems. The compatibility page gives an overview of the compatibility picture, while the compatibility wiki has more information on a per-package basis. A scan of the wiki shows there's lots more work to do, but it also shows a pretty wide range of code compatible with PyPy.

ARM and other projects

One of the more interesting developments in the PyPy world is ARM support, for which a 2.0 alpha has been released. It supports both ARMv6 (e.g., Raspberry Pi) and v7 (e.g., Beagleboard, Chromebook) and the benchmark results look good, especially given that the ARM code "is not nearly as polished as our x86 assembler". The Raspberry Pi Foundation helped get PyPy onto ARM with "a small amount of funding".

The PyPy project is running several concurrent fundraising efforts, three for specific sub-projects, and one for overall project funding. The transactional memory/automatic mutual exclusion sub-project is an effort to use software transactional memory to allow Python programs to use multiple cores more effectively. It would remove the global interpreter lock (GIL) for PyPy for better concurrency. PyPy hackers Armin Rigo and Maciej Fijałkowski gave a presentation at PyCon 2013 on this effort.

Another ongoing sub-project is an effort to add Python 3 support to PyPy. That would allow input programs in Python 3, but would not change the PyPy implementation language (RPython based on Python 2.x). A status report back in March shows good progress. On x86-32 Linux, the CPython regression test suite "now passes 289 out of approximately 354 modules (with 39 skips)".

The third sub-project is to make NumPy work with PyPy. NumPy is an extension for doing math with matrices and multi-dimensional arrays. Much of that work is done in C code, so PyPy's JIT would need to use the vector instructions on modern CPUs. A brief status update from May 11 shows some progress, as does the 2.0 release announcement (though the temporary loss of lazy expression evaluation may not exactly be considered progress).

Overall, the PyPy project seems to be cruising along. While none of the fundraising efforts have hit their targets, some fairly significant money has been raised. Beyond that, some major technical progress has been made. The sub-projects, software transactional memory in particular, are also providing interesting new dimensions for the project. We are getting closer to the day when most Python code is runnable with PyPy, though we still aren't there yet.

Comments (10 posted)

Infected Linux web servers pushing malware

Windows malware is not something that Linux users and administrators typically need to be concerned about. Though our systems can't be infected by such code, it's unlikely we want them to be used as a vector for its transmission, but that's just what's happening with the Linux/Cdorked.A malware. Originally linked to Apache web servers, the malware has now been found in both lighttpd and Nginx web servers as well. While there is a fair amount known about how Cdorked works, it is still an open question as to how it is getting installed onto Linux systems.

The scope of the problem has widened a few times. At first it seemed that only cPanel-based servers were affected by the malware when it was first reported in late April. By early May, it was clear that "it cannot be attributed solely to installations of cPanel because only a fraction of the infected servers are using this management software", according to a report from ESET, an anti-virus company. That report was also the one that pointed out that it was more than just Apache web servers being affected.

Essentially, infected systems have a malicious version of the web server binary installed. That is the only difference that shows up on the disk of affected systems. All of the configuration information for the malware is pushed into a shared memory segment via unlogged HTTP requests sent by the attacker. Typically, the compromised web server is configured to serve some subset of incoming HTTP requests with a redirect to a site of the attacker's choice—likely one that serves up malware.

The subset of requests is fairly carefully chosen. Cdorked avoids redirecting any request that looks like it might be accessing an administrative page, presumably to avoid doing anything suspicious to anyone likely to be technically sophisticated. It also sets a cookie when it does the redirection, so that it doesn't do so again.

The redirections are mainly done to sites that host the Blackhole exploit kit, which tries to use a wide variety of browser holes and plugin exploits to infect a victim's system. Typical configurations noted by ESET redirect Windows Firefox and Internet Explorer users to the Blackhole sites, while sending iPhone and iPad users to a pornographic site, presumably trying to monetize requests that would not be affected by the Blackhole exploits.

There are some other interesting restrictions on who gets redirected. Roughly half of the IPv4 address space has been blacklisted (or whitelisted depending on how you look at it) and will not get redirected. There is no obvious pattern to the IP address ranges. Similarly, there are restrictions based on the language configured for the browser, so browsers set to Japanese, Finnish, Russian, Ukrainian, Belarusian, and Kazakh are immune as well.

ESET has discovered a sophisticated set of commands that can be issued to the trojanized web server binaries. That includes a command to create a connection to the specified address with a shell attached. That would allow the attacker full access as the UID that is running the web server, as described in an earlier ESET blog post. As might be expected, the other commands (all of which are sent via obfuscated URLs to the infected host) manipulate the IP blacklist, language list, URL to redirect to, and so on.

The fact that the attackers have created trojanized binaries for three separate web servers (at least so far), the extensive "command and control" mechanism, and the stealthy nature of the attack, all add up to a fairly sophisticated set of attackers. ESET has also found evidence of subverted nameservers at the shared hosting sites that are (presumably unknowingly) serving the Blackhole kits.

The most puzzling aspect, perhaps, is how the initial infection of the web servers is happening. "One thing is clear, this malware does not propagate by itself and it does not exploit a vulnerability in a specific software", according to ESET. It will be interesting to see what set of vulnerabilities was actually used here; hopefully that will be determined soon. In the meantime, for detecting compromised systems there is a tool to look for the shared memory segments, though checking the integrity of web server binaries is another possibility. On the worrisome side, though, is that whatever vulnerability was used, it can presumably be used again. This one seems worth keeping an eye on.

Comments (2 posted)

Security quotes of the week

Comments (none posted)

Hack.lu call for papers

Comments (none posted)

Local root vulnerability in the kernel

Comments (38 posted)

firefox: multiple vulnerabilities

Comments (none posted)

firefox: multiple vulnerabilities

Comments (none posted)

gpsd: code execution

Comments (none posted)

httpd: command execution

Comments (none posted)

java: multiple unspecified vulnerabilities

Comments (none posted)

kdelibs: username and password disclosure

Comments (none posted)

kernel: multiple vulnerabilities

Comments (none posted)

libtiff: two vulnerabilities

Comments (none posted)

openstack-keystone: password disclosure

Comments (none posted)

owncloud: multiple vulnerabilities

Comments (none posted)

php-geshi: multiple vulnerabilities

Comments (none posted)

php-sabredav-Sabre_DAV: local file exposure

Comments (none posted)

python-httplib2: SSL certificate verification failure

Comments (none posted)

xen: denial of service

Comments (none posted)

Kernel release status

Stable updates: 3.9.2, 3.8.13, 3.4.45, and 3.0.78 were released on May 11; 3.2.45 came out on May 14.

In the 3.8.13 announcement Greg Kroah-Hartman said: "NOTE, this is the LAST 3.8.y kernel release, please move to the 3.9.y kernel series at this time. It is end-of-life, dead, gone, buried, and put way behind us never to be spoken of again. Seriously, move on, it's just not worth it anymore." But the folks at Canonical, having shipped 3.8 in the Ubuntu 13.04 release, are not moving on; they have announced support for this kernel until August 2014.

Comments (7 posted)

Quotes of the week

choice
	prompt "BogoMIPs setting"
	default BOGOMIPS_MEDIUM
	help
	  The BogoMIPs value reported by Linux is exactly what it sounds
	  like: totally bogus. It is used to calibrate the delay loop,
	  which may be backed by a timer clocked completely independently
	  of the CPU.

	  Unfortunately, that doesn't stop marketing types (and even people
	  who should know better) from using the number to compare machines
	  and then screaming if it's less than some fictitious, expected
	  value.

	  So, this option can be used to avoid the inevitable amount of
	  pain and suffering you will endure when the chaps described above
	  start parsing /proc/cpuinfo.

	config BOGOMIPS_SLOW
		bool "Slow (older machines)"
		help
		  If you're comparing a faster machine with a slower machine,
		  then you might want this option selected on one of them.

	config BOGOMIPS_MEDIUM
		bool "Medium (default)"
		help
		  A BogoMIPS value for the masses.

	config BOGOMIPS_FAST
		bool "Fast (marketing)"
		help
		  Some people believe that software runs faster with this
		  setting so, if you're one of them, say Y here.

	config BOGOMIPS_RANDOM
		bool "Random (increased Bogosity)"
		help
		  Putting the Bogo back into BogoMIPs.

Comments (4 posted)

Extended stable support for the 3.8 kernel

Full Story (comments: 23)

copy_range()

There have been various proposals over the years for ways to speed up copy operations (reflink(), for example), but nothing has ever made it into the mainline. The latest attempt is Zach Brown's copy_range() patch. It adds a new system call:

    int copy_range(int in_fd, loff_t *in_offset,
		   int out_fd, loff_t *out_offset, size_t count);

The intent of the system call is fairly clear: copy count bytes from the input file to the output. It is not said anywhere, but it's implicit in the patch that the two files should be on the same filesystem.

Inside the kernel, a new copy_range() member is added to the file_operations structure; each filesystem is meant to implement that operation to provide a fast copy operation. There is no fallback at the VFS layer if copy_range() is unavailable, but that looks like the sort of omission that would be fixed before mainline merging. Whether merging will ever happen remains to be seen; this is an area that is littered with abandoned code from previous failed attempts.

Comments (9 posted)

The conclusion of the 3.10 merge window

• The bcache caching layer has been merged. Bcache allows a fast device (like an SSD) to provide fast caching in front of a slower device; it is designed for fast performance given the constraints of contemporary solid-state devices. See Documentation/bcache.txt for more information.

• The on-disk representation of extents in Btrfs has been changed to make the structure significantly smaller. "In practice this results in a 30-35% decrease in the size of our extent tree, which means we COW less and can keep more of the extent tree in memory which makes our heavy metadata operations go much faster." It is an incompatible format change that must be explicitly enabled when the filesystem is created (or after the fact with btrfstune).

• The MIPS architecture has gained basic support for virtualization with KVM. MIPS kernels can also now be built using the new "microMIPS" instruction set, with significant space savings.

• New hardware support includes Abilis TB10x processors, Freescale ColdFire 537x processors, Freescale M5373EVB boards, Broadcom BCM6362 processors, Ralink RT2880, RT3883, and MT7620 processors, and Armada 370/XP thermal management controllers.

Changes visible to kernel developers include:

• The block layer has gained basic power management support; it is primarily intended to control which I/O requests can pass through to a device while it is suspending or resuming. To that end, power-management-related requests should be marked with the net __REQ_PM flag.

• A lot of work has gone into the block layer in preparation for "immutable biovecs," a reimplementation of the low-level structure used to represent ranges of blocks for I/O operations. One of the key advantages here seems to be that it becomes possible to create a new biovec that contains a subrange of an existing biovec, leading to fast and efficient request splitting. The completion of this work will presumably show up in 3.11.

• The dedicated thread pool implementation used to implement writeback in the memory management subsystem has been replaced by a workqueue.

If this development cycle follows the usual pattern, the final 3.10 kernel release can be expected in early July. Between now and then, though, there will certainly be a lot of bugs to fix.

Comments (none posted)

Smarter shrinkers

A new shrinker API

In current kernels, a cache would implement a shrinker function that adheres to this interface:

    #include <linux/shrinker.h>

    struct shrink_control {
	gfp_t gfp_mask;
	unsigned long nr_to_scan;
    };

    int (*shrink)(struct shrinker *s, struct shrink_control *sc);

The shrink() function is packaged up inside a shrinker structure (along with some ancillary information); the whole thing is then registered with a call to register_shrinker().

When memory gets tight, the shrink() function will be called from the memory management subsystem. The gfp_mask will reflect the type of allocation that was being attempted when the shrink() call was made; the shrinker should avoid any actions that contradict that mask. So, for example, if a GFP_NOFS allocation is in progress, a filesystem shrinker cannot initiate filesystem activity to free memory. The nr_to_scan field tells the shrinker how many objects it should examine and free if possible; if, however, nr_to_scan is zero, the call is really a request to know how many objects currently exist in the cache.

The use of a single callback function for two purposes (counting objects and freeing them) irks some developers; it also makes the interface harder to implement. So, one of the first steps in the new shrinker patch set is to redefine the shrinker API to look like this:

    long (*count_objects)(struct shrinker *s, struct shrink_control *sc);
    long (*scan_objects)(struct shrinker *s, struct shrink_control *sc);

The roughly two-dozen shrinker implementations in the kernel have been updated to use this new API.

The current shrinker API is not NUMA-aware. In an effort to improve that situation, the shrink_control structure has been augmented with a new field:

    nodemask_t nodes_to_scan;

On NUMA systems, memory pressure is often not a global phenomenon. Instead, some nodes will have plenty of free memory while others are running low. The current shrinker interface will indiscriminately free memory objects; it pays no attention to which NUMA node any given object is local to As a result, it can dump a lot of cached data without necessarily helping to address the real problem. In the new scheme, shrinkers should observe the nodes_to_scan field and only free memory from the indicated NUMA nodes.

LRU lists

A maintainer of an existing shrinker implementation may well look at the new NUMA awareness requirement with dismay. Most shrinker implementations are buried deep within filesystems and certain drivers; these subsystems do not normally track their cached items by which NUMA node holds them. So it appears that shrinker implementations could get more complicated, but that turns out not to be the case.

While looking at the shrinker code, Dave Chinner realized that most implementations look very much the same: they maintain a least-recently-used (LRU) list of cached items. When the shrinker is called, a pass is made over the list in an attempt to satisfy the request. Much of that code looked well suited for a generic replacement; that replacement, in the form of a new type of linked list, is part of the larger shrinker patch set.

The resulting "LRU list" data structure encapsulates a lot of the details of object cache management; it goes well beyond a simple ordered list. Internally, it is represented by a set of regular list_head structures (one per node), a set of per-node object counts, and per-node spinlocks to control access. The inclusion of the spinlock puts the LRU list at odds with normal kernel conventions: low-level data structures do not usually include their own locking mechanism, since that locking is often more efficiently done at a higher level. In this case, putting the lock in the data structure allows it to provide per-node locking without the need for NUMA awareness in higher-level callers.

The basic API for the management of LRU lists is pretty much as one might expect:

    #include <linux/list_lru.h>

    int list_lru_init(struct list_lru *lru);
    int list_lru_add(struct list_lru *lru, struct list_head *item);
    int list_lru_del(struct list_lru *lru, struct list_head *item);

A count of the number of items on a list can be had with list_lru_count(). There is also a mechanism for walking through an LRU list that is aimed at the needs of shrinker implementations:

    unsigned long list_lru_walk(struct list_lru	*lru, 
			        list_lru_walk_cb isolate,
				void *cb_arg,
				unsigned long nr_to_walk);
    unsigned long list_lru_walk_nodemask(struct list_lru *lru, 
    	     	  	        list_lru_walk_cb isolate,
				void *cb_arg,
				unsigned long nr_to_walk,
				nodemask_t *nodes_to_walk);

Either function will wander through the list, calling the isolate() callback and, possibly, modifying the list in response to the callback's return value. As one would expect, list_lru_walk() will pass through the entire LRU list, while list_lru_walk_nodemask() limits itself to the specified nodes_to_walk. The callback's prototype looks like this:

    typedef enum lru_status (*list_lru_walk_cb)(struct list_head *item, 
    	    	 	    			spinlock_t *lock,
						void *cb_arg);

Here, item is an item from the list to be examined, lock is the spinlock controlling access to the list, and cb_arg is specified by the original caller. The return value can be one of four possibilities, depending on how the callback deals with the given item:

• LRU_REMOVED indicates that the callback removed the item from the list; the number of items on the list will be decremented accordingly. In this case, the callback does the actual removal of the item.

• LRU_ROTATE says that the given item should be moved to the ("most recently used") end of the list. The LRU list code will perform the move operation.

• LRU_RETRY indicates that the callback should be called again with the same item. A second LRU_RETRY return will cause the item to be skipped. A potential use for this return value is if the callback notices a potential deadlock situation.

• LRU_SKIP causes the item to be passed over with no changes.

With this infrastructure in place, a lot of shrinker implementations come down to a call to list_lru_walk_nodemask() and a callback to process individual items.

Memcg-aware LRU lists

While an improved shrinker interface is well worth the effort on its own, much of the work described above has been driven by an additional need: better support for memory control groups (memcgs). In particular, memcg developer Glauber Costa would like to be able to use the shrinker mechanism to free only memory that is associated with a given memcg. All that is needed to reach this goal is to expand the LRU list concept to include memcg awareness along with NUMA node awareness.

The result is a significant reworking of the LRU list API. What started as a simple list with some helper functions has now become a two-dimensional array of lists, indexed by node and memcg ID. A call to list_lru_add() will now determine which memcg the item belongs to and put it onto the relevant sublist. There is a new function — list_lru_walk_nodemask_memcg() — that will walk through an LRU list, picking out only the elements found on the given node(s) and belonging to the given memcg. The more generic functions described above have been reimplemented as wrappers around the memcg-specific versions. At this point, the "LRU list" is no longer a generic data structure (though one could still use it that way); it is, instead, a core component of the memory management subsystem.

Closing notes

A review of the current shrinker implementations in the kernel reveals that not all of them manage simple object caches. In many cases, what is happening is that the code in question wanted a way to know when the system is under memory pressure. In current kernels, the only way to get that information is to register a shrinker and see when it gets called. Such uses are frowned upon; they end up putting marginally related code into the memory reclaim path.

The shrinker patch set seeks to eliminate those users by providing a different mechanism for code that wants to learn about memory pressure. It essentially hooks into the vmpressure mechanism to set up an in-kernel notification mechanism, albeit one that does not use the kernel's usual notifier infrastructure. Interested code can call:

    int vmpressure_register_kernel_event(struct cgroup *cg, void (*fn)(void));

The given fn() will be called at the same time that pressure notifications are sent out to user space. The concept of "pressure levels" has not been implemented for the kernel-side interface, though.

Most of this code is relatively new, and it touches a fair amount of core memory management code. The latter stages of the patch set, where memcg awareness is added, could be controversial, but, then, it could be that developers have resigned themselves to memcg code being invasive and expensive. One way or another, most or all of this code will probably find its way into the mainline; the benefits of the shrinker API improvements will be nice to have. But the path to the mainline could be long, and this patch set has just begun, so it may be a while before it is merged.

Comments (none posted)

User-space page fault handling

One situation where user-space page fault handling can be useful is for the live migration of virtual machines from one physical host to another. Migration can be done by stopping the machine, copying its full address space to the new host, and restarting the machine there. But address spaces may be large and sparsely used; copying a full address space can result in a lot of unnecessary work and a noticeable pause before the migrated system restarts. If, instead, the virtual machine's address space could be demand-paged from the old host to the new, it could restart more quickly and the copying of unused data could be avoided.

Live migration with KVM is currently managed with an out-of-tree char device. This scheme works, but, once the device takes over a range of memory, that memory is removed from the memory management subsystem. So it cannot be swapped out, transparent huge pages don't work, and so on. Clearly it would be better to come up with a solution that, while allowing user-space handling of demand paging, does not remove the affected memory from the kernel's management altogether. A patch set recently posted by Andrea Arcangeli aims to resolve those issues with a couple of new system call options.

The first of those is to extend the madvise() system call, adding a new command called MADV_USERFAULT. Processes can use this operation to tell the kernel that user space will handle page faults on a range of memory. After this call, any access to an unmapped address in the given range will result in a SIGBUS signal; the process is then expected to respond by mapping a real page into the unmapped space as described below. The madvise(MADV_USERFAULT) call should be made immediately after the memory range is created; user-space fault handling will not work if the kernel handles any page faults before it is told that user space will be doing the job.

The SIGBUS signal handler's job is to handle the page fault by mapping a real page to the faulting address. That can be done in current kernels with the mremap() system call. The problem with mremap() is that it works by splitting the virtual memory area (VMA) structure used to describe the memory range within the kernel. Frequent mremap() calls will result in the kernel having to manage a large number of VMAs, which is an expensive proposition. mremap() will also happily overwrite existing memory mappings, making it harder to detect errors (or race conditions) in user-space handlers. For these reasons, mremap() is not an ideal solution to the problem.

Andrea's answer to this problem is a new system call:

    int remap_anon_pages(void *dest, void *source, unsigned long len);

This call will cause the len bytes of memory starting at source to be mapped into the process's address space starting at dest. At the same time, the source memory range will be unmapped — the pages previously found there will be atomically moved to the dest range.

Andrea has posted a small test program that demonstrates how these APIs are meant to be used.

As one might expect, some restrictions apply: source and dest must be page-aligned, len should be a multiple of the page size, the dest range must be completely unmapped, and the source range must be fully mapped. The mapping requirements exist to catch bugs in user-space fault handlers; remapping pages on top of existing memory has a high risk of causing memory corruption.

One nice feature of the patch set is that, on systems where transparent huge pages are enabled, huge pages can be remapped with remap_anon_pages() without the need to split them apart. For that to work, of course, the length and alignment of the range to move must be compatible with huge pages.

There are a number of limitations in the current patch set. The MADV_USERFAULT option can only be used on anonymous (swap-backed) memory areas. A more complete implementation could conceivably support this feature for file-backed pages as well. The mechanism offers support for demand paging of data into RAM, but there is no user-controllable mechanism for pushing data back out; instead, those pages are swapped with all other anonymous pages. So it is not a complete user-space paging mechanism; it's more of a hook for loading the initial contents of anonymous pages from an outside source.

But, even with those limitations, the feature is useful for the intended virtualization use case. Andrea suggests it could possibly have other uses as well; remote RAM applications come to mind. First, though, it needs to get into the mainline, and that, in turn, suggests that the proposed ABI needs to be reviewed carefully. Thus far, this patch set has not gotten a lot of review attention; that will need to change before it can be considered for mainline merging.

Comments (18 posted)

Always-releasable Debian

Debian 7.0 ("Wheezy") was released on May 5, at which point preparations started for 8.0 ("Jessie"). But the freeze period in the Wheezy development cycle (during which only release-critical (RC) bugs are meant to be fixed) took longer than average: ten months rather than the usually-expected six. That led to some understandable self-reflection about the development and release process, including a proposal from Lars Wirzenius and Russ Allbery to adopt several changes meant to keep the testing branch in an always-releasable state.

A modest proposal

The benefits of transitioning to a shorter release cycle would affect many groups: users would get updated packages more quickly, the release team would feel less swamped by the stack of RC bugs, and developers would feel less frustrated about the process. The crux of the proposal is to keep the packages in testing as free of RC bugs as possible, akin to maintaining a releasable trunk branch in an individual software project. Wirzenius and Allbery suggest four changes to bring this about:

• Making an attitude shift, so that maintainers of individual packages view making Debian releases as part of their job, not just the job of the release team.
• Keeping RC bugs out of testing.
• Using automatic testing and/or continuous integration to catch problems earlier.
• Limiting the number of packages treated as "core" packages that can hold up a release.

The attitude shift is a nebulous task, naturally; few on the debian-devel mailing list actually think of releases as the concern of the release team alone. But the gist is that individual package maintainers may need to change the way they handle certain changes, such as transitioning a fundamental package like glibc to a new version, which can have far-reaching effects on other packages.

Keeping RC bugs out of testing is a goal that could involve several possible steps. Wirzenius and Allbery note that in the current release model, "right after the release we open the floodgates, and large number of new packages and versions enter testing. The bug count sky-rockets, but we don't care a lot about that until the next freeze gets closer." They suggest removing RC-buggy packages as soon as possible, including dependent packages, devoting more developer-power to fixing RC bugs in those core packages that cannot be removed (e.g., GCC), and having bug-fix-only "mini-freezes" if the RC count rises above a particular threshold.

On the automatic testing front, they suggest setting up a suite of "reference installations" aimed at common Debian deployment scenarios (mail server, LAMP server, desktop machine, etc.) to determine which packages ought to be considered "core" and which ought to be considered optional. The project could then use these reference systems as targets for continuous integration and testing. They note that Debian already has several automatic testing tools (such as lintian, piuparts, adequate, and autopkgtest), but they do not currently guide the progress of a package into testing:

Holger Levsen has already been working on a continuous integration system for Debian at jenkins.debian.net, Wirzenius and Allbery observe, and there quite a few useful test already available in piuparts—it will just take some additional effort to build them into a reliable automatic testing framework.

The great debate

On the whole, most of the people on debian-devel seem supportive of the goal—which is to say they agree that the lengthy Wheezy freeze was less than ideal and that the process can be improved. Almost everyone who replied to Wirzenius and Allbery's email is in favor of increased automated testing and continuous integration. There is less unanimity on the question of limiting the number of packages that constitute the essential core of a release, however, and about how attitude changes may or may not affect the process.

For example, Paul Wise thought that the point about the attitude shift "essentially comes down to 'people don't do enough work and we want them to do more'", which does not account for several factors that impact developers' individual workloads, including available time, knowledge, confidence, and motivation level. In response, Neil Williams replied that the real intent was to solve a somewhat different problem: "people are not getting the work done, let's break down the problems and make working on them easier or the solutions more obvious".

On the other hand, Michael Gilbert felt that there was not a fundamental problem with the methodology used in the Wheezy release cycle, and that it in fact reflected a better bug-squashing effort:

Allbery replied that the project says almost the same thing after every release, and that it needs to "be sure that we're not just trying the same thing over and over again and expecting different results." Ultimately, however, Gilbert's suggestion for handling the increased RC bug count was more automated testing, so regardless of whether the increased RC bug count is intrinsically good or intrinsically bad, handling it better involves the same solution.

Where there is less consensus at present is on the subject of limiting the number of packages regarded as "core" components of a release—and pulling non-core components that introduce RC bugs (although the offending package would be re-admitted to testing once its RC bugs had been fixed). In their original proposal, Wirzenius and Allbery noted that making a core/non-core distinction could have negative side-effects, such as causing buggy packages to miss the release. Perhaps Debian could introduce new packages in subsequent point releases, they said—an idea which always-releasable testing makes possible.

But several felt that the existing process already sifts packages by importance, at least as "importance" is defined in practice. Vincent Bernat said: "If a package is important, an RC bug will get noticed and someone will step to fix the RC bug or ask for a delay. This avoids unnecessary debate on what is important and what is not and people fighting to get their packages in the right list." Helmut Grohne worried that frequently adding and removing packages from testing would destabilize it as much or more than the RC bugs introduced by those packages. Perhaps, he said, simply making the removal notification (which precedes any actual package removal) higher-profile would goad the developer into fixing the bug faster.

It is also not clear who would make the core/non-core determination; such a call could easily be a source of controversy. Wirzenius and Allbery assured everyone in their proposal that the non-core designation is not a black mark, but considering the practical effect it has—namely, that the release will not wait for the package to get fixed—it has the potential to spawn disagreement. Not everyone was sold on the "reference installations" idea, either; Thomas Goirand said that the suggested installation types (e.g., web server, mail server) denote packages that are pretty stable already, and not the sources of problems in Wheezy. "If you want to find out which tests would help, you would have to conduct a better analysis of the problems we had during the freeze, IMO."

Naturally, there are other "postmortem" discussions analyzing the Wheezy release cycle and how it could be improved. Ian Jackson, for example, started a thread of his own around the same time, analyzing (among other things) which RC bugs had proven "hard" to fix and what could be done to alleviate the time required. The always-releasable testing concept is being tracked on the Debian wiki; clearly the goal it sets out is grander in scope than a mere reduction in the length of the pre-release freeze. But regardless of where it eventually heads, it has already raised some ideas that seem likely to get more attention before the release of Jessie. Debian will likely remain a "release when ready" distribution, not following the time-based process popular among other distributions, but automated testing and continuous integration could go a long way to making that "ready" state a more frequent occurrence.

Comments (none posted)

Distribution quotes of the week

Comments (none posted)

Three Ubuntu releases reach end of life

Comments (9 posted)

A proposal for an always-releasable Debian

Full Story (comments: 49)

Fedora account system (FAS) potential information disclosure

Full Story (comments: 19)

Fedora Elections

Full Story (comments: none)

Distribution newsletters

• Debian Project News (May 13)
• DistroWatch Weekly, Issue 507 (May 13)
• Maemo Weekly News (May 13)
• Ubuntu Weekly Newsletter, Issue 316 (May 13)

Comments (none posted)

Raspberry Pi operating systems: 5 reviewed and rated (Techradar)

Comments (4 posted)

Cinnarch successor Antergos arrives (The H)

Comments (none posted)

How Mighty Mint became one of the most popular Linux distros (Techradar)

Comments (1 posted)

PostgreSQL 9.3 beta: Federated databases and more

In Berkeley, California — the birthplace of PostgreSQL — it's spring: plum and cherry blossoms, courting finches and college students, new plans for the summer, and the first beta release of the database system. Every year, the first beta of the next PostgreSQL version comes out in April or May, for a final release in September. PostgreSQL 9.3 beta 1 was released to the public on May 13th, and contains a couple dozen new features both for database administrators and application developers.

Of course, if you're in the southern hemisphere, it's not spring for you. Nor if you live in the north Midwest of the US or central Canada. Sorry about that; we just track the weather with PostgreSQL, we don't control it.

As usual, there are too many features in the new PostgreSQL to cover them all individually, so we're going to go over just a few of them here: database federation, writable foreign data sources, and the end of System V shared memory dependence.

Federated databases

Sharding and master-slave replication are the most common ways to horizontally scale a relational database, but they are not the only ones. A type of horizontal scaling which has been little explored among open source databases is federated databases. PostgreSQL 9.3 will introduce database federation as a standard feature of the database system, through the postgres_fdw extension, which will ship with the core code.

The idea of federated databases is that each database server can query other database servers on the network, effectively giving the user access to all databases and tables on all servers from any single database client. Tables can even be JOINed in queries with other tables from other servers. Federated databases are basically connected by the database engine at the query executor level. This type of horizontal scaling is mainly good for spreading reads of large amounts of data around several machines, and as such is primarily useful for data warehousing and analytics, rather than for transaction processing.

This concept has been implemented before, and is one of the most distinctive features of IBM's DB2. However, implementations within open source relational databases have not been extensively used. PostgreSQL has had federated databases though Skype's PL/Proxy extension since 2006, but that extension was never integrated into the core, and it forces database access via stored procedures, which didn't work for many users. MySQL introduced the FEDERATED storage engine in version 5.0, but for some reason it seems not to have been widely adopted as part of scalable MySQL solutions.

PostgreSQL introduced the foreign data wrapper (FDW) feature for accessing external data at the query level in version 9.1. That version's FDWs were read-only, however, and queried external data by copying the entire target data source to memory on the querying server, which necessarily limits how big of an external table you could query. Since August of 2010, a development team including major contributor Shigeru Hanada and committer Takahiro Itagaki, worked on making FDWs something more, culminating in the new postgres_fdw extension. Hanada described the use case and development of the feature as follows:

Creating a link to a table on another PostgreSQL server is relatively simple, although it has multiple steps. First, import the postgres_fdw extension:

    CREATE EXTENSION postgres_fdw;

    CREATE SERVER remotesrv foreign data wrapper postgres_fdw OPTIONS 
        ( host '127.0.01', port '5433', dbname 'bench');

    CREATE USER MAPPING FOR current_user SERVER remotesrv OPTIONS 
        ( user 'josh', password 'password' );

    CREATE FOREIGN TABLE remoteacct (aid int, bid int, abalance int, filler char(84)) 
        SERVER remotesrv OPTIONS ( table_name 'pgbench_accounts' );

Queries can be run against the tables on the attached server, including writing to it, just as if it were a table on the local server:

    EXPLAIN SELECT * FROM remoteacct WHERE bid = 5;
    INSERT INTO remoteacct ( aid, bid, abalance ) VALUES ( 10000000, 5, 100 );

The postgres_fdw extension permits the PostgreSQL query planner to "push down" query clauses, such as WHERE clauses, to the remote database, allowing for distributed, parallel execution of several portions of the federated query. This eliminates the requirement to copy the entire foreign table into memory, and with some architectures permits execution of federated queries over much larger data sets than could be queried on one server alone. More work needs to be done in this area to make this a full "big data" solution, however; Hanada hopes to add "push down" of joins, sorts, and aggregates in future versions of PostgreSQL.

Linking PostgreSQL to Redis

The improved foreign data wrappers aren't limited to PostgreSQL-to-PostgreSQL connections. FDWs can be used to connect PostgreSQL to any kind of external data, as long as it can be rendered in a tabular format: Oracle database tables, CSV files, process lists, or data from non-relational databases such as Redis and MongoDB, are all possibilities. Now that FDWs are writable as well as readable, PostgreSQL becomes much more useful as a data integration tool for multiple other databases and data sources.

In order to connect to a non-PostgreSQL data source, a developer needs to write a special driver (also called an FDW), which is then installable as a PostgreSQL extension. Existing FDWs will need to be enhanced to support writability. One which is likely to be ready for read-write access when PostgreSQL 9.3 is released is the Redis FDW. Andrew Dunstan, PostgreSQL committer, is working on a new version of the driver, because he uses Redis together with PostgreSQL. Redis is a non-relational, memory-only database which stores hashes, lists, and sets.

Many people use Redis alongside PostgreSQL. It functions well as an application cache, a buffer for rapid-fire writes, or to support queuing. The Redis FDW allows users to pull data directly from Redis into PostgreSQL without going through the application layer, saving both development time and system resources. Dunstan describes the work:

Redis can be attached as a foreign server as well. First you need to install the redis_fdw extension, and create the foreign server, in this case a Redis server on the same machine, and tell PostgreSQL what users are allowed to access it, as you do with a Postgres-to-Postgres link:

    CREATE EXTENSION redis_fdw;
    CREATE SERVER localredis FOREIGN DATA WRAPPER redis_fdw;
    CREATE USER MAPPING FOR public SERVER localredis;

Redis stores data in five forms: scalars, hashes, sets, lists and sorted sets (zsets). These objects can be mapped to local PostgreSQL tables, either singly or in groups. The code below, for example, makes a two-column table out of all of the lists whose key begins with "jobqueue":

    CREATE FOREIGN TABLE jobqueues(key text, list text[])
        SERVER localredis
        OPTIONS (database '0', tabletype 'list', tableprefix 'jobqueue');

The future read-write Redis FDW will permit pushing data to Redis as well as reading it, enabling new ways of using Redis as an integrated cache or queuing store with PostgreSQL. For example, cache invalidation and refresh can be controlled using a database trigger, making invalidation much more discriminating about which data it needs to replace. Or PostgreSQL can automatically push items onto a Redis queue whenever certain events in the database happen.

No more "shmmax"

PostgreSQL's use of SysV shared memory has been a frequent source of irritation for system administrators. Every time an administrator installs PostgreSQL on a brand-new system, they have to edit the sysctl.conf file in order to raise the kernel limits on shared memory by substantially increasing "shmmax" and "shmall"; otherwise, PostgreSQL is limited to using a few megabytes of RAM. As of version 9.3, this headache will go away, thanks to committer Robert Haas:

After debating several other approaches, Haas ended up moving all of PostgreSQL's dedicated memory to mmap(), using anonymous shared memory regions. This avoids shared memory limits, and is a widely supported interface that works on all of the operating systems supported by PostgreSQL, except for Windows. PostgreSQL for Windows, however, has always used a different system of memory allocation. The new mechanism works without users needing to make any changes to their PostgreSQL configurations. The one problem reported so far has been decreased performance on BSD operating systems, due to the loss of special optimizations those operating systems made for SysV shared memory.

Lots more features

As always, this PostgreSQL annual release has a lot more features to offer users. Included in the current beta are:

• A new CREATE MATERIALIZED VIEW declaration, which lets users generate precalculated summary tables using a simple SQL statement.
• Regular dynamic VIEWs (saved queries) are now automatically updatable as well as readable.
• A data page checksum option for users who need to detect disk integrity issues immediately.
• A new JOIN construct, LATERAL JOINs, which allow self-referencing table lists; this is especially useful with functions or foreign data wrappers.
• Additional built-in JSON manipulation functions for the JSON data type.
• Indexed regular expression search to speed up text matching.
• Sub-second "fast failover" option when switching replicated servers, for 99.999% high availability.

Plus many other features, some of them unique to PostgreSQL. The community has documented some of them, so that you can see if there is something for you in this beta.

Inevitably, these features also introduce lots of new bugs, which is why the PostgreSQL team wants you to download the beta and test it with your applications. Like most community-driven open source projects, PostgreSQL relies heavily on end-user testing to find bugs before the final release. The project will release multiple betas over the four month testing period. The final release for version 9.3 is expected to be sometime in September.

[ Josh Berkus is a member of the PostgreSQL Core Team. ]

Comments (32 posted)

Quotes of the week

Comments (6 posted)

PyPy 2.0 released

Comments (8 posted)

GFXprim-1.0.0 release candidate available

The first release candidate of the GFXprim 2D bitmap graphics library is now available, after more than four years of development. Python bindings, code generators, and several supported back-ends are all included with the release.

Full Story (comments: none)

PacketFence 4.0 released

Comments (3 posted)

Gawk 4.1.0 released

Full Story (comments: 21)

PostgreSQL 9.3 Beta 1 released

Full Story (comments: 4)

Go language 1.1 released

Comments (17 posted)

Tahoe-LAFS v1.10 is available

Version 1.10 of the Tahoe Least-Authority File System has been released. This update "adds a new Introducer protocol, improves the appearance of the web-based user interface, improves grid security by making introducer FURLs unguessable" and fixes several bugs while it's at it.

Full Story (comments: none)

GnuPg 2.0.20 available

Version 2.0.20 of GNU Privacy Guard (GnuPG) has been released. This update offers a number of improvements to smartcard support, as well as various fixes for other operations.

Full Story (comments: none)

Firefox 21 is now available

Firefox 21 has been released, for desktop systems and for Android. This version adds several new service providers to the Social API, initial support for the "Firefox Health Report" troubleshooting tool, and adds a "please track me" option to the Do Not Track (DNT) selector, which is sure to be a big hit.

Full Story (comments: none)

Development newsletters from the past week

• Caml Weekly News (May 14)
• What's cooking in git.git (May 9)
• CloudStack Weekly News (May 13)
• Openstack Community Weekly Newsletter (May 10)
• Perl Weekly (May 13)
• PostgreSQL Weekly News (May 12)
• Ruby Weekly (May 9)

Comments (none posted)

Results of the Apache OpenOffice 4.0 Logo Survey

Rob Weir has posted an analysis of the logo contest recently held for Apache OpenOffice. The main blog post showcases the leading vote-getters, but the real meat comes in the detailed report, which breaks down the survey by demographics and examines various ways of interpreting what boils down to a set of individual personal preferences. "With an ordinal interpretation we can look at histograms (counts of scores), at the mode (most frequent response), median (the middle value) and the variation ratio (fraction of scores not in the mode). With an interval interpretation we would assign each point on the scale a numeric value, e.g., 1 for Strongly Dislike to 5 for Strongly Like. Then we could take these scores and calculate means and standard deviations." The logo-selection process now moves to revisions by the leading candidates, aiming for the upcoming 4.0 release.

Comments (117 posted)

Reed: Eulogy for a Founding Father

At his blog, Mozilla's J. Paul Reed has posted a reflection on the recently-announced end-of-life for Tinderbox, the Mozilla continuous integration (CI) system. "It may have 'sucked,' but it facilitated a workflow and nurtured an ethos that is not only extremely important, but taken for granted today: namely the notion that individual developers should be 'on the hook' when checking in, and have a responsibility to their peers to monitor the build and make sure the tree 'stays green.'" Reed also notes that, despite the rise of other CI tools, Tinderbox still sports some features not found in newer systems.

Comments (none posted)

New Linux Foundation members

Full Story (comments: none)

New Zealand Government Announces That Software Will No Longer Be Patentable (Forbes)

Comments (50 posted)

Kiwi PyCon 2013 Call for Proposals

Full Story (comments: none)

Events: May 16, 2013 to July 15, 2013

If your event does not appear here, please tell us about it.