LWN.net Weekly Edition for March 13, 2014
The Linux Foundation invests in MOOCs
In addition to hosting events, shepherding software projects, and supporting kernel developers, the Linux Foundation (LF) has long offered training courses of various kinds—for developers, system administrators, and even managers. These courses are generally paid, "professional-development"–style training, but the LF recently announced that it would open up its introductory Linux course as a free online offering, built in conjunction with one of the leading sources for massive open online courses (MOOCs).
The announcement was made on March 6, noting the LF's partnership with edX, an online-courseware organization whose site hosts MOOCs for several major universities. As we saw in September 2013, edX has released the source code to its hosting platform and now maintains it as an active open-source project. Many of the universities who offer classes through edX contribute developer time (and even components), as does Google, who re-tasked the team from its own MOOC platform Course Builder, to edX development.
Although the edX platform can be self hosted, the LF course in
question, Introduction to Linux, will be offered through
the main edX site. The course
is already available for registration, though it will not come online
until the third quarter of 2014. As for the content of the course, it
comes from the systems administration section of LF's course catalog, and
the current page about the paid,
in-person variant of the course describes it as teaching "a
good working knowledge of Linux, from both a graphical and command
line perspective, allowing you to easily navigate through any of the
three major Linux distributions.
" As LF's Jerry Cooperstein explained in a comment to our earlier
blurb about the announcement, the "three major distributions" refer to
the Fedora/CentOS/RHEL family, the Debian/Ubuntu/Mint family, and the
SUSE/openSUSE family.
An outline offers a few more specific topics, but it is quite clearly a course meant for those with zero prior experience working with a Linux distribution. The paid version of the course costs $2400 and takes four days. When the edX version launches, it will be a work-at-your-own-pace endeavor.
For those people already familiar with Linux, there is probably little to see in Introduction to Linux. The more important aspect of the announcement is the fact that the LF has joined edX as a Member Institution. That suggests that LF will likely make more and more of its training courses available in MOOC form, which will open them up to far more participants. As it stands now, LF training courses are generally offered as on-site corporate offerings, co-located at LF events, or as periodic training events held in big cities. There may multiple opportunities for an interested individual to get to one of the in-person classes, but the availability and self-pacing of a MOOC allow considerably more flexibility.
In a March 11 blog post, LF's Amanda McPherson linked the decision to offer the course for free at edX to the current Linux talent shortage, and noted that more than 38,000 people had already registered for the course in advance. Considering that level of interest, reaching the wider pool of potential students seems to be the major benefit of the MOOC approach.
Furthermore, while the introductory course is understandably light in its subject matter, the other offerings already offered as LF training do include some pretty in-depth topics, like Optimizing Linux Device Drivers for Power Efficiency, and cover some specific software projects like Yocto, Tizen, and OpenStack that are distinct enough to warrant special attention.
The other wrinkle, of course, is the training classes are one of the ways that the LF raises funds. The edX site itself offers only free classes, so if the LF intends to offer more of its own content in MOOC form, it will probably do so at its own site. Some of its existing courses are offered in "virtual" form, but this means a combination of whiteboard software and live audio conferencing to connect to a classroom where an instructor is lecturing.
The MOOC format has proven itself to be a far more popular way to work through course materials, with the likes of Stanford, MIT, and Harvard (among many others) offering MOOC classes. So it is a wise move for the LF to embrace it for technical training; by partnering with the leading open-source MOOC project, it can do so while also supporting the development of an important software framework.
What's new with HTTP2
Changing any widely used network protocol is a risky affair—more so with a protocol as ubiquitous as HTTP. But the venerable web transport protocol is expected to see its first update of the century later this year when HTTP2 is slated for release. The new revision can be hard to characterize, since it changes the way HTTP is sent over the wire, but it does not alter the existing HTTP semantics. In essence, the revisions are designed to speed up how HTTP traffic is relayed between clients, servers, proxies, and switches, but to keep intact the format and meaning of HTTP requests and responses. The upshot will be faster throughput, even if few people outside of network operators will see the actual changes to the protocol.
HTTP has not been updated since 1999, when the Internet Engineering Task Force (IETF) published version 1.1 in RFC 2616. Version 1.1, notably, added support for persistent, pipelined TCP connections between clients and servers. Thus, it was no longer necessary to open and close a separate TCP connection for each HTTP request, and clients could send multiple requests without waiting for each request's response before sending the next.
Those changes did a lot to reduce latency in HTTP, and the concepts implemented arose from the real-world problems experienced with HTTP 1.0. The HTTP2 situation is similar; the vendors that make web software and networking hardware—as well as large-scale web service providers—identified a new set of pain points as the World Wide Web continued to evolve. Several players (including Google and Microsoft) started their own research projects to develop improvements, and by 2007 there was enough interest for the IETF to charter a new working group to address a new revision.
The working group is called the HTTP Bis group—where Bis is a perhaps uncommon designation meaning "twice" that is intended to communicate that the group's charter is not to invent new methods or headers, but just to improve on existing HTTP functionality. The notion that HTTP2 is semantically compatible with HTTP 1.1 is clearly important to the project. The group has recently taken to calling revised protocol HTTP2 or HTTP/2, rather than "HTTP 2.0," just to make such a distinction more prominent, although there is not yet universal agreement on the point.
Drafting
The most recent HTTP2 draft is version 10, which was published on February 13. The first draft was based directly on Google's SPDY (which we looked at in 2009), although as one would expect, it has evolved in a number of ways since then. The basic approach, however, remains the same.
HTTP2 is backward compatible with all of HTTP 1.1's request methods, responses, and headers. But it attempts to decrease the overall latency of serving a page to a client by multiplexing requests, compressing HTTP headers, and allowing the server to prioritize some requests over others.
Multiplexing requests is in many ways an extension to the pipelining feature of HTTP 1.1. Pipelining allows the client to send request #2 before request #1 has been fulfilled by the server, but it still has a weakness: request #2 is still processed by the server after request #1. So if request #1 takes a long time, it still introduces latency. This is called "head-of-line blocking." HTTP 1.1 is also hampered by the requirement that clients not open more than two simultaneous connections to the same server. HTTP2 does away with that limit, and averts head-of-line blocking by multiplexing several HTTP request connections over the same underlying TCP connection (testing of current implementations seems to indicate somewhere between four to eight streams as the best number on average). Since these multiplexed HTTP streams are independent of each other, it is far less likely for slow requests to block the delivery of the entire page.
In addition to making more efficient use of bandwidth, HTTP2's multiplexing will alleviate the need to do several other popular workarounds, such as resource sharding (i.e., splitting one image into several pieces each served from a different server) and the inlining of CSS and JavaScript. Both of these techniques have their costs; sharding introduces multiple DNS lookups for the various servers (and then opening lots of separate HTTP connections to them), and inline CSS and JavaScript cannot be cached by the browser. So the full benefits of HTTP2 multiplexing should exceed the raw gains seen just from reducing head-of-line blocking.
HTTP2 also compresses the HTTP headers, which typically contain a lot of redundancy—particularly between related requests for the same page. The earliest implementations used gzip for compression, and built a shared Huffman code dictionary for all of the requests on a single connection. But that scheme was found to be vulnerable to the CRIME (Compression Ratio Info-leak Made Easy) attack, in which an attacker can discover the bytes of a session cookie by injecting test bytes into HTTP requests—if the compressed result is the same length as the unmodified request, then the attacker knows the test byte is indeed part of the cookie. The HTTP Bis group has subsequently moved to a new compression scheme called HPACK.
There is little opposition to header compression as an idea, but
the same cannot be said of another HTTP2 optimization: binary
encoding. As befits a compressed format, HTTP2's frame format is
binary, both for the header and the payload. This decision has its
critics, such as John Graham-Cumming, who said
that he wondered "whether it was really necessary to add a
complex, binary protocol to the HTTP suite
". But, in response,
Mozilla's Patrick McManus commented
that the orderly message boundaries of the binary format make it
considerably more straightforward—and, therefore,
faster—to parse than HTTP 1.1's ASCII. Of course, it could also
be argued that the binary format is not human-readable, but the
counter-argument is that it is merely the binary encoding of existing
ASCII request/response semantics, rather than something new.
HTTP2 adds several other mechanisms that can speed up the process of fulfilling requests, but which (unlike compression and multiplexing) require some deployment-time tweaks to produce optimized results. For instance, servers can prioritize one request stream over another. The assumption is that some content (such as page text) is more critical to deliver to the client than others (such as CSS or decorative images), but, naturally, the decision of which resources to prioritize and which to leave untouched cannot be solved in the abstract. Similarly, HTTP2 allows servers to effectively "push" some resources, initiating the transfer before the client makes the actual request. The most effective implementation might begin sending linked-in CSS or image resources after the main page is requested, but the protocol itself does not dictate how servers decide what to push.
100 CONTINUE
Work continues on HTTP2, and the final draft is not expected until mid-2014. Most recently, discussion has included whether (and how) the new protocol should deal with encryption. At one time or another, dropping support for unencrypted http:// URIs was considered, as was dropping HTTPS as a separate protocol—in favor of requiring that HTTP2 run over TLS. For now, no such large steps have been taken. But the group has introduced HTTP Alternate Services (ALTSVC), a new HTTP header through which a server can move a URI request to a different host, port, or even protocol, without tearing down the connection and starting a new one.
This technique could be used to "upgrade" an http:// URI request from HTTP 1.1 to HTTP2, and thus move the connection to TLS. ALTSVC is a comparatively recent draft, which is still subject to change. Among other things, upgrading a connection to TLS relies on the presence of some specific TLS extensions, such as Server Name Indication (SNI), so while the idea of "opportunistic encryption" sounds appealing to many, there may be a fair amount of tire-kicking left before it is finalized.
The working group maintains a list of issues on GitHub and a publicly archived mailing list at the W3C site, which makes it easy to follow the development of the protocol. Just as importantly, the group tracks working implementations of HTTP2. Firefox and Chromium already support HTTP2, and Google, Twitter, and a few other large-scale application services support it on the server side. Since many of the benefits of the new protocol rely on implementations testing and getting the optimization details right, this is welcome news, and keeping in line with the IETF mantra of "rough consensus and working code." End users, of course, will notice only reduced page load times if they notice anything at all—but perhaps that is a good goal for a protocol update to set.
A false midnight
Interpreted, "duck typing" languages often have some idiosyncrasies in their definitions of "truth" and Python is no exception. But Python goes a bit further than some other languages in interpreting the True or False status of non-Boolean values. Even so, it often comes as a big surprise for programmers to find (sometimes by way of a hard-to-reproduce bug) that, unlike any other time value, midnight (i.e. datetime.time(0,0,0)) is False. A long discussion on the python-ideas mailing list shows that, while surprising, that behavior is desirable—at least in some quarters.
Python's notion of truth values defines False as various kinds of numeric zero (0, 0L, 0.0, 0j), "empty" objects ('', (), [], {}), the special value None, and, of course, False. User-defined classes can also have a False value, but many will not, which means that all objects of those classes are True. However, buried in the datetime module documentation for a time object is the following somewhat complicated explanation:
It all boils down to the idea that a time of 00:00 UTC (which could be something else entirely in the local time zone, of course) is False. If one were testing a variable meant to hold a time value as follows:
if current:
do_something(current) # we have a valid current time
The code will fail to do_something() if current is
None or is set to a time value that is exactly at
midnight UTC.
One can argue (and several in the thread did) that the test is wrong and
should be:
if current is not None:
do_something(current) # we have a valid current time
But "if var:" is a common idiom in Python code—it works
just fine for most objects, as they always evaluate to True, so
None can be used as a sentinel value. But
it doesn't work for time objects, at least for one second per day.
The datetime module in the standard library is a fairly simple way to represent dates, times, and timestamps (using the datetime class). The date class does not have a time associated with it, while the time class (with resolution in seconds) is undated. Oddly, a time can have a time zone associated with it (even though time zones are often date-related). A datetime has both time and date (and can include a time zone as well).
A Python bug was opened in 2012 about midnight as False, but it was closed soon thereafter as "invalid". The time class was documented to work that way and someone had created a __nonzero__() (now __bool__()) method for time to implement that behavior. But, the behavior still didn't sit well with some. Nine months after the bug was closed, Danilo Bargen reported that the False interpretation caused problems with filters in Django templates. He asked that the bug be reopened, but nothing happened—and that's where things stood until recently.
That's when Shai Berger asked again for the bug to be reopened. That request was seconded by Andreas Pelme before moving to python-ideas at the request of Alexander Belopolsky. Berger posted a summary of the issue and asked for a reconsideration. That led to a large thread, even by python-ideas standards.
Berger's argument was twofold. First, paraphrasing Bargen and Pelme, that
the current behavior is "surprising and useless
" since users
generally don't want special behavior at midnight. Second, his personal
argument is that midnight "is not a 'zero value', it is just a
value
". He went on to say that midnight as False made as
much sense as datetime.date(1,1,1) (the minimum date
value) evaluating to False (and it doesn't).
While Paul Moore was unhappy with the
practice of using Boolean tests for date validity
(as it should be "is not None" in his mind),
he did think the current behavior was wrong. Oscar Benjamin went further, noting that if this was new code
being added, returning False for midnight in a Boolean context
would never pass muster. "The question is surely whether the issue is worth a backwards
compatibility break not whether the current behaviour is a good idea
(it clearly isn't).
"
On the other hand, Skip Montanaro is convinced that the problem is all in the application code and that the bug should remain closed. There weren't too many voices joining in with that sentiment, though Tim Peters and Belopolsky, who were both involved in the creation of the datetime module, were opposed to making any change. But Nick Coghlan posted an analysis of the current behavior that made it clear to many that it was completely inconsistent and not really tenable:
Coghlan had already reopened the bug, suggesting that anyone interested in seeing it fixed could post a patch to do so. The only question was how to handle deprecating the existing behavior. Python 3.4 is feature-frozen, so it is out of the picture; any change would come in 3.5 and subsequent releases. Coghlan's suggestion was to issue a DeprecationWarning whenever a time returns False in a Boolean context for 3.5, then to change the behavior so that all time values are True for 3.6. That means the change won't actually be released for roughly three years, which is too long for some.
One of those who might be in that "change it already" camp is BDFL Guido van Rossum. He posted an unequivocal statement that False time values were a mistake made a long time ago:
Furthermore, Van Rossum said, he would be inclined to fix the problem "immediately" (for Python 3.5) but would, perhaps, like to see a search for legitimate uses of the feature. Both Peters and Belopolsky noted that they had used the False midnight in programs, but nothing that was widely distributed; no other examples were mentioned in the thread. Meanwhile, multiple reports of problems stemming from the feature did come up.
While there is a danger that some legitimate programs might run afoul of the change, Van Rossum's intuition is probably right: there are likely few instances of real False midnight users, while the unexpected False problem is likely to be fairly common, with many if var: instances going undetected (or at least undiagnosed) for years. Backward compatibility is always a balancing act, but in this case it would seem that making a clean break from the existing behavior—and relatively soon—is probably the right approach.
Security
The status of Wayland security
Unlike its predecessor, X, Wayland was created with an eye toward security. In part, that is due to the threat landscape present at the time each was created—the twenty-first century is a much more dangerous time. While Wayland is designed to provide confidentiality and integrity, there are times when applications will need or want to circumvent those protections. How to safely allow that is the topic of some recent postings to the wayland-devel mailing list.
The discussion started back at the end of December in a thread on authorizing clients to use restricted interfaces for accessibility and other tools, though it had been touched on in the context of screen-grabbing applications even earlier. In mid-February, Martin Peres summarized the discussion thus far both on the mailing list and on his blog. He also referred back to a talk that he and Timothée Ravier gave at the 2012 X.Org Developer Conference. The current state of Wayland input security is good, Peres said, and there is work in progress for its output security, but it is the mechanisms to allow applications to violate those security constraints (under user control) that are still being worked out.
Unlike X, the Wayland input stack doesn't allow applications to snoop on the input of other programs (preserving confidentiality), to generate input events that appear to come from the user (preserving input integrity), or to capture all the input events to the exclusion of the user's application (preserving availability). Wayland clients are still susceptible to LD_PRELOAD-style attacks, but that is not something the Wayland protocol itself can preclude. It is no different than the situation with X clients today, though, so existing solutions can presumably be reused.
On the output side, Wayland does a lot better than X, in general, but there is still some work to do. The use of "graphics execution manager" (GEM) buffers between the client and server is known to be problematic because buffers are identified by 32-bit handles; any program that can guess a buffer's handle can access the buffer itself. Work has been done in Wayland to use DMA-buf sharing, which does not suffer from the same confidentiality and integrity problems because it relies on file descriptor passing, rather than a globally visible handle. In Mesa 10.0, released in November 2013, the Wayland client EGL code does have support for secure DMA-buf sharing.
But there is still the question of what to do about programs that need to
circumvent some of those protections. For example, virtual keyboards need
to be able to generate input events, screen readers and screen shot
programs need to get access to the output buffers, screen lockers need to
capture all input, and so on. As Peres put it, if users don't have the
ability to allow these actions "people may refuse to use Wayland because it 'takes freedom away from them'
".
There are two different ways to achieve the goal of letting users (broadly defined to include distributions as well) break the security barriers: either static capability-like assignment to binaries, or determining the user's intent at runtime. Peres thinks that both should be implemented for Wayland. In addition, he sees a need for users to be able to determine whether any programs are currently running with elevated Wayland permissions (via static assignment) and to revoke those permissions if so.
Clients requiring extra privileges can be launched by the compositor so that it can provide a known, secure environment for those programs. A file descriptor of a socket that will allow the client to connect back to the compositor can be passed to the client using the WAYLAND_SOCKET environment variable. The actual descriptor is inherited by the client across the fork() and exec() calls, but its integer value is passed via the environment. Since the compositor has opened the socket to itself, and controls which client is executed, it can apply any privileges that have been assigned to the client to any operations made over that socket.
Handling the security policy for programs is a difficult problem. Different types of users and distributions will want to be able to implement their own policies. As we have seen with the Linux kernel, finding consensus on security policies is somewhere between really difficult and impossible. Peres expects to see the equivalent of the modular approach taken by Linux Security Modules (LSMs) for Wayland (perhaps Wayland Security Modules or WSMs). That way, different kinds of Wayland consumers will be able to choose a policy framework, much like distributions today choose SELinux, Smack, AppArmor, or to create their own.
Peres has plans to start working on WSMs (or whatever the name ends up being) soon. He has outlined some of the considerations that would need to go into a cross-compositor security framework as part of the discussion. It is a wide-open topic, in many ways, and one that is in its earliest planning stages; those interested in those kinds of problems (and their solutions) should probably get involved.
As can be seen, Wayland starts from a much stronger security position than X, but that doesn't mean that all of the work is done. Beyond the inevitable bugs, there are still some fundamental issues to consider—and work on.
Brief items
Security quotes of the week
However, this story is about how something the way around has happened, in that AnonGhost, a hacker group, believe that they’ve defaced “Yorkshire Bank, one of the largest United Kingdom bank” and there’s some boasting about this to be found at http://www.p0ison.com/ybs-bank-got-hacked-by-team-anonghost/.
However, it rather looks to me as if they’ve hacked an imitation bank instead! A rather less glorious exploit from the point of view of potential admirers.
The internal post – titled “I hunt sys admins” – makes clear that terrorists aren’t the only targets of such NSA attacks. Compromising a systems administrator, the operative notes, makes it easier to get to other targets of interest, including any “government official that happens to be using the network some admin takes care of.”
Given the energies and opportunities out there, it’s possible to end repressive Internet censorship within a decade. If we want the next generation of users to be free, we don’t see any other option.
FSF: Replicant developers find and close Samsung Galaxy back-door
The Free Software Foundation has put out a release claiming that developers working on the Replicant fork of Android have found a backdoor on Samsung Galaxy handsets. "While working on Replicant, a fully free/libre version of Android, we discovered that the proprietary program running on the applications processor in charge of handling the communication protocol with the modem actually implements a back-door that lets the modem perform remote file I/O operations on the file system. This program is shipped with the Samsung Galaxy devices and makes it possible for the modem to read, write and delete files on the phone's storage. On several phone models, this program runs with sufficient rights to access and modify the user's personal data."
New vulnerabilities
apache2: update SSLCipherSuite to avoid CRIME attack
| Package(s): | apache2 | CVE #(s): | |||||
| Created: | March 10, 2014 | Updated: | March 12, 2014 | ||||
| Description: | From the openSUSE advisory:
This low-profile update introduces a backport of the SSLCompression directive (added to /etc/apache2/ssl-global.conf) that helps mitigating the CRIME attack if set to off (default). Also added to /etc/apache2/ssl-global.conf: "SSLHonorCipherOrder on". /etc/apache2/vhosts.d/vhost-ssl.template now contains a new SSLCipherSuite string. Even though GCM mode of AES is not supported in openssl-1.0.0l, the string works well and may be useful elsewhere, too. | ||||||
| Alerts: |
| ||||||
cups-filters: multiple vulnerabilities
| Package(s): | cups-filters | CVE #(s): | CVE-2013-6474 CVE-2013-6475 CVE-2013-6476 | ||||||||||||||||||||||||||||
| Created: | March 12, 2014 | Updated: | March 13, 2014 | ||||||||||||||||||||||||||||
| Description: | From the Debian advisory:
Florian Weimer of the Red Hat Product Security Team discovered multiple vulnerabilities in the pdftoopvp CUPS filter, which could result in the execution of arbitrary code if a malformed PDF file is processed. | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
cups-filters: code execution
| Package(s): | cups-filters | CVE #(s): | CVE-2013-6473 | ||||||||||||||||
| Created: | March 12, 2014 | Updated: | March 12, 2014 | ||||||||||||||||
| Description: | From the Ubuntu advisory:
Florian Weimer discovered that cups-filters incorrectly handled memory in the urftopdf filter. An attacker could possibly use this issue to execute arbitrary code with the privileges of the lp user. This issue only affected Ubuntu 13.10. | ||||||||||||||||||
| Alerts: |
| ||||||||||||||||||
fail2ban: denial of service
| Package(s): | fail2ban | CVE #(s): | CVE-2013-7176 CVE-2013-7177 | ||||||||||||||||||||
| Created: | March 10, 2014 | Updated: | July 18, 2014 | ||||||||||||||||||||
| Description: | From the CVE entries:
config/filter.d/postfix.conf in the postfix filter in Fail2ban before 0.8.11 allows remote attackers to trigger the blocking of an arbitrary IP address via a crafted e-mail address that matches an improperly designed regular expression. (CVE-2013-7176) config/filter.d/cyrus-imap.conf in the cyrus-imap filter in Fail2ban before 0.8.11 allows remote attackers to trigger the blocking of an arbitrary IP address via a crafted e-mail address that matches an improperly designed regular expression. (CVE-2013-7177) | ||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||
file: code execution
| Package(s): | file | CVE #(s): | CVE-2014-2270 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 7, 2014 | Updated: | April 9, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Magiea advisory: A flaw was found in the way the file utility determined the type of Portable Executable (PE) format files, the executable format used on Windows. A malicious PE file could cause the file utility to crash or, potentially, execute arbitrary code (CVE-2014-2270). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
imapsync: information leak
| Package(s): | imapsync | CVE #(s): | CVE-2013-4279 | ||||||||||||||||||||||||||||
| Created: | March 10, 2014 | Updated: | December 1, 2015 | ||||||||||||||||||||||||||||
| Description: | From the Red Hat bugzilla:
By default imapsync runs a "release check" when executed, this causes imapsync to connect to http://imapsync.lamiral.info and send information about the version of imapsync, the operating system and perl. This feature is not well documented. It is enabled by default. The only hint it exists is the "--noreleasecheck" which is not documented anywhere other then running the program with the help option. | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
kernel: three vulnerabilities
| Package(s): | kernel | CVE #(s): | CVE-2014-0100 CVE-2014-0101 CVE-2014-0049 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 6, 2014 | Updated: | April 25, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Red Hat bugzilla entries [1, 2, 3]: CVE-2014-0100: A very subtle race condition between inet_frag_evictor, inet_frag_intern and the IPv4/6 frag_queue and expire functions (basically the users of inet_frag_kill/inet_frag_put) was found. What happens is that after a fragment has been added to the hash chain but before it's been added to the lru_list (inet_frag_lru_add), it may get deleted (either by an expired timer if the system load is high or the timer sufficiently low, or by the fraq_queue function for different reasons) before it's added to the lru_list, then after it gets added it's a matter of time for the evictor to get to a piece of memory which has been freed leading to a number of different bugs depending on what's left there. CVE-2014-0101: A flaw was found in the way Linux kernel processed authenticated COOKIE_ECHO chunks. A remote attacker could use this flaw to crash the system by sending a maliciously prepared SCTP handshake in order to trigger a NULL pointer dereference on the server. CVE-2014-0049: The problem occurs when the guest performs a pusha with the stack address pointing to an mmio address (or an invalid guest physical address) to start with, but then extending into an ordinary guest physical address. When doing repeated emulated pushes emulator_read_write sets mmio_needed to 1 on the first one. On a later push when the stack points to regular memory, mmio_nr_fragments is set to 0, but mmio_is_needed is not set to 0. As a result, KVM exits to userspace, and then returns to complete_emulated_mmio. In complete_emulated_mmio vcpu->mmio_cur_fragment is incremented. The termination condition of vcpu->mmio_cur_fragment == vcpu->mmio_nr_fragments is never achieved. The code bounces back and fourth to userspace incrementing mmio_cur_fragment past it's buffer. If the guest does nothing else it eventually leads to a a crash on a memcpy from invalid memory address. However if a guest code can cause the vm to be destoryed in another vcpu with excellent timing, then kvm_clear_async_pf_completion_queue can be used by the guest to control the data that's pointed to by the call to cancel_work_item, which can be used to gain execution. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kernel: two information leaks
| Package(s): | EC2 kernel | CVE #(s): | CVE-2014-1444 CVE-2014-1445 | ||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 6, 2014 | Updated: | March 12, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Ubuntu advisory:
An information leak was discovered in the Linux kernel's SIOCWANDEV ioctl call. A local user with the CAP_NET_ADMIN capability could exploit this flaw to obtain potentially sensitive information from kernel memory. (CVE-2014-1444) An information leak was discovered in the wanxl ioctl function the Linux kernel. A local user could exploit this flaw to obtain potentially sensitive information from kernel memory. (CVE-2014-1445) | ||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||
libpng16: denial of service
| Package(s): | libpng16 | CVE #(s): | CVE-2014-0333 | ||||||||||||||||||||||||
| Created: | March 12, 2014 | Updated: | March 17, 2014 | ||||||||||||||||||||||||
| Description: | From the CVE entry:
The png_push_read_chunk function in pngpread.c in the progressive decoder in libpng 1.6.x through 1.6.9 allows remote attackers to cause a denial of service (infinite loop and CPU consumption) via an IDAT chunk with a length of zero. | ||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||
libssh: private key leak
| Package(s): | libssh | CVE #(s): | CVE-2014-0017 | ||||||||||||||||||||||||||||||||||||||||
| Created: | March 6, 2014 | Updated: | August 12, 2014 | ||||||||||||||||||||||||||||||||||||||||
| Description: | From the Mageia advisory:
When using libssh before 0.6.3, a libssh-based server, when accepting a new connection, forks and the child process handles the request. The RAND_bytes() function of openssl doesn't reset its state after the fork, but simply adds the current process id (getpid) to the PRNG state, which is not guaranteed to be unique. The most important consequence is that servers using EC (ECDSA) or DSA certificates may under certain conditions leak their private key (CVE-2014-0017). | ||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||
mediawiki: multiple vulnerabilities
| Package(s): | mediawiki | CVE #(s): | CVE-2014-2242 CVE-2014-2243 CVE-2014-2244 | ||||||||||||||||||||
| Created: | March 10, 2014 | Updated: | March 12, 2014 | ||||||||||||||||||||
| Description: | From the CVE entries:
includes/upload/UploadBase.php in MediaWiki before 1.19.12, 1.20.x and 1.21.x before 1.21.6, and 1.22.x before 1.22.3 does not prevent use of invalid namespaces in SVG files, which allows remote attackers to conduct cross-site scripting (XSS) attacks via an SVG upload, as demonstrated by use of a W3C XHTML namespace in conjunction with an IFRAME element. (CVE-2014-2242). includes/User.php in MediaWiki before 1.19.12, 1.20.x and 1.21.x before 1.21.6, and 1.22.x before 1.22.3 terminates validation of a user token upon encountering the first incorrect character, which makes it easier for remote attackers to obtain access via a brute-force attack that relies on timing differences in responses to incorrect token guesses. (CVE-2014-2243). Cross-site scripting (XSS) vulnerability in the formatHTML function in includes/api/ApiFormatBase.php in MediaWiki before 1.19.12, 1.20.x and 1.21.x before 1.21.6, and 1.22.x before 1.22.3 allows remote attackers to inject arbitrary web script or HTML via a crafted string located after http:// in the text parameter to api.php. (CVE-2014-2244). | ||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||
mutt: code execution
| Package(s): | mutt | CVE #(s): | CVE-2014-0467 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 12, 2014 | Updated: | June 5, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Debian advisory:
Beatrice Torracca and Evgeni Golov discovered a buffer overflow in the mutt mailreader. Malformed RFC2047 header lines could result in denial of service or potentially the execution of arbitrary code. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
net-snmp: multiple vulnerabilities
| Package(s): | net-snmp | CVE #(s): | CVE-2014-2284 CVE-2014-2285 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 7, 2014 | Updated: | March 25, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Mageia advisory: Remotely exploitable denial of service vulnerability in Net-SNMP, in the Linux implementation of the ICMP-MIB, making the SNMP agent vulnerable if it is making use of the ICMP-MIB table objects (CVE-2014-2284). Remotely exploitable denial of service vulnerability in Net-SNMP, in snmptrapd, due to how it handles trap requests with an empty community string when the perl handler is enabled (CVE-2014-2285). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
owncloud: multiple unspecified vulnerabilities
| Package(s): | owncloud | CVE #(s): | |||||||||||||
| Created: | March 12, 2014 | Updated: | May 9, 2014 | ||||||||||||
| Description: | From the Mageia advisory:
Owncloud versions 5.0.15 and 6.0.2 fix several unspecified security vulnerabilities, as well as many other bugs. | ||||||||||||||
| Alerts: |
| ||||||||||||||
percona-toolkit, xtrabackup: code execution
| Package(s): | percona-toolkit,xtrabackup | CVE #(s): | CVE-2014-2029 | ||||||||||||
| Created: | March 6, 2014 | Updated: | March 14, 2014 | ||||||||||||
| Description: | From the openSUSE advisory:
CVE-2014-2029: Can be used by owner of a Percona Server (or an attacker who can control this destination for the client) to collect arbitrary MySQL configuration parameters and execute commands (with -v). Now the version check needs to be requested via command line or global/tool specific/user configuration. (--version-check) | ||||||||||||||
| Alerts: |
| ||||||||||||||
php-sabre-dav: unspecified vulnerability
| Package(s): | php-sabre-dav | CVE #(s): | |||||||||
| Created: | March 12, 2014 | Updated: | March 12, 2014 | ||||||||
| Description: | From the Fedora advisory:
XEE issue: Previous SabreDAV versions had a security issue, if running on the following PHP versions: PHP 5.3, older than 5.3.23, PHP 5.4, older than 5.4.13, PHP 5.5 is not affected by this. | ||||||||||
| Alerts: |
| ||||||||||
rubygem-actionpack: cross-site scripting
| Package(s): | rubygem-actionpack | CVE #(s): | CVE-2013-6416 | ||||
| Created: | March 7, 2014 | Updated: | March 12, 2014 | ||||
| Description: | From the CVE entry: Cross-site scripting (XSS) vulnerability in the simple_format helper in actionpack/lib/action_view/helpers/text_helper.rb in Ruby on Rails 4.x before 4.0.2 allows remote attackers to inject arbitrary web script or HTML via a crafted HTML attribute. | ||||||
| Alerts: |
| ||||||
rubygem-actionpack: multiple vulnerabilities
| Package(s): | rubygem-actionpack | CVE #(s): | CVE-2014-0080 CVE-2014-0081 CVE-2014-0082 | ||||||||||||||||||||||||||||
| Created: | March 11, 2014 | Updated: | April 25, 2014 | ||||||||||||||||||||||||||||
| Description: | From the CVE entries:
SQL injection vulnerability in activerecord/lib/active_record/connection_adapters/postgresql/cast.rb in Active Record in Ruby on Rails 4.0.x before 4.0.3, and 4.1.0.beta1, when PostgreSQL is used, allows remote attackers to execute "add data" SQL commands via vectors involving \ (backslash) characters that are not properly handled in operations on array columns. (CVE-2014-0080) Multiple cross-site scripting (XSS) vulnerabilities in actionview/lib/action_view/helpers/number_helper.rb in Ruby on Rails before 3.2.17, 4.0.x before 4.0.3, and 4.1.x before 4.1.0.beta2 allow remote attackers to inject arbitrary web script or HTML via the (1) format, (2) negative_format, or (3) units parameter to the (a) number_to_currency, (b) number_to_percentage, or (c) number_to_human helper. (CVE-2014-0081) actionpack/lib/action_view/template/text.rb in Action View in Ruby on Rails 3.x before 3.2.17 converts MIME type strings to symbols during use of the :text option to the render method, which allows remote attackers to cause a denial of service (memory consumption) by including these strings in headers. (CVE-2014-0082) | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
rubygems: gems not getting security updates
| Package(s): | rubygems | CVE #(s): | |||||
| Created: | March 6, 2014 | Updated: | March 12, 2014 | ||||
| Description: | From the openSUSE advisory:
fix rubygem patches are not applied to the gem but only to the tree. Packages embedding rubygems via their .gem files were not receiving security updates. | ||||||
| Alerts: |
| ||||||
sudo: privilege escalation
| Package(s): | sudo | CVE #(s): | CVE-2014-0106 | ||||||||||||||||||||||||||||||||||||||||
| Created: | March 6, 2014 | Updated: | June 27, 2014 | ||||||||||||||||||||||||||||||||||||||||
| Description: | From the sudo advisory:
Due to a logic bug in the validate_env_vars() function, if the env_reset option is disabled, environment variables specified on the command line are permitted when they should not be (and vice versa). This can be used by a malicious user to run arbitrary programs by manipulating the environment of a command the user is legitimately allowed to run. For example, on many systems the LD_PRELOAD environment variable is used to load a dynamic shared object before any shared libraries are loaded. By either replacing a library function called by the program, or by including an _init() function in the shared object, the user can execute arbitrary commands with elevated privileges. | ||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||
tomcat: multiple vulnerabilities
| Package(s): | tomcat6, tomcat7 | CVE #(s): | CVE-2013-4286 CVE-2013-4322 CVE-2014-0033 CVE-2014-0050 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 6, 2014 | Updated: | September 26, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Ubuntu advisory:
It was discovered that Tomcat incorrectly handled certain inconsistent HTTP headers. A remote attacker could possibly use this flaw to conduct request smuggling attacks. (CVE-2013-4286) It was discovered that Tomcat incorrectly handled certain requests submitted using chunked transfer encoding. A remote attacker could use this flaw to cause the Tomcat server to stop responding, resulting in a denial of service. (CVE-2013-4322) It was discovered that Tomcat incorrectly applied the disableURLRewriting setting when handling a session id in a URL. A remote attacker could possibly use this flaw to conduct session fixation attacks. This issue only applied to Ubuntu 12.04 LTS. (CVE-2014-0033) It was discovered that Tomcat incorrectly handled malformed Content-Type headers and multipart requests. A remote attacker could use this flaw to cause the Tomcat server to stop responding, resulting in a denial of service. This issue only applied to Ubuntu 12.10 and Ubuntu 13.10. (CVE-2014-0050) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
udisks: privilege escalation
| Package(s): | udisks | CVE #(s): | CVE-2014-0004 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 10, 2014 | Updated: | March 29, 2015 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Debian advisory:
Florian Weimer discovered a buffer overflow in udisks's mount path parsing code which may result in privilege escalation. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
wireshark: multiple vulnerabilities
| Package(s): | wireshark | CVE #(s): | CVE-2014-2281 CVE-2014-2283 CVE-2014-2299 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | March 10, 2014 | Updated: | March 17, 2014 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Debian advisory:
Moshe Kaplan discovered that the NFS dissector could be crashed, resulting in denial of service. (CVE-2014-2281) It was discovered that the RLC dissector could be crashed, resulting in denial of service. (CVE-2014-2283) Wesley Neelen discovered a buffer overflow in the MPEG file parser, which could lead to the execution of arbitrary code. (CVE-2014-2299) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
wireshark: denial of service
| Package(s): | wireshark | CVE #(s): | CVE-2014-2282 | ||||||||||||||||||||
| Created: | March 10, 2014 | Updated: | March 12, 2014 | ||||||||||||||||||||
| Description: | From the Mageia advisory:
The M3UA dissector could crash. | ||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||
Page editor: Jake Edge
Kernel development
Brief items
Kernel release status
The current development kernel is 3.14-rc6, released on March 9. Linus says: "There haven't been any huge problems, but there's been quite a few small bumps that shouldn't happen this late in the release cycle. And rc6 is noticeably bigger than rc5 was, as well. So I'm really hoping that the upcoming week will be calmer, because otherwise I'll start thing rc8 and even rc9."
Stable updates: 3.13.6 and 3.10.33 were released on March 6; 3.4.83 followed on March 11.
A discussion between database and kernel developers
On March 27, there will be a meeting between database and kernel developers at the Linux Foundation's Collaboration Summit; interested developers are invited to attend. "If there are developers attending Collaboration Summit that work in the database or kernel communities, it would be great if you could come along. Previous discussions were on the PostgreSQL list and that should be expanded in case we accidentally build postgres-only features. The intent is to identify the problems encountered by databases and where relevant, test cases that can be used to demonstrate them if they exist. While the kernel community may be aware of some of the problems, they are not always widely known or understood."
Kernel development news
3.14 development statistics
Normally, by the time the -rc6 prepatch is released in a given development cycle, the final release is thought to be not too far away. This time around, Linus is making noises about needing to go as far as -rc8 or even -rc9 due to the number of outstanding problems. Even if we are still a few weeks from release, the situation with 3.14 will be stable enough that a look at where the changes in this release came from makes sense. The picture that emerges can be described as "business as usual," with the continuation of some longstanding trends.As of this writing, the 3.14 development cycle has seen the addition of just over 12,000 non-merge changesets from exactly 1,400 developers. These changes added 591,000 lines of code while removing 250,000 lines, for a net growth of 341,000 lines. The most active developers this time around were:
Most active 3.14 developers
By changesets H Hartley Sweeten 278 2.3% Laurent Pinchart 232 1.9% Jingoo Han 174 1.4% Rashika Kheria 162 1.3% Geert Uytterhoeven 138 1.1% Tejun Heo 123 1.0% Sachin Kamat 122 1.0% Kuninori Morimoto 110 0.9% Sujith Manoharan 97 0.8% Linus Walleij 89 0.7% Wei Yongjun 86 0.7% Alex Deucher 82 0.7% Stephen Warren 81 0.7% Lars-Peter Clausen 79 0.7% Ville Syrjälä 78 0.6% Namhyung Kim 76 0.6% Thierry Reding 74 0.6% Johannes Berg 73 0.6% Christoph Hellwig 73 0.6% Ding Tianhong 71 0.6%
By changed lines Greg Kroah-Hartman 73035 10.5% Micky Ching 23657 3.4% Stephen Boyd 17511 2.5% Paul Gortmaker 17511 2.5% Greg Rose 16428 2.4% Tero Kristo 14509 2.1% Ben Skeggs 13320 1.9% Sergio Aguirre 8388 1.2% Ben Hutchings 8002 1.2% Tejun Heo 7975 1.2% Laurent Pinchart 7799 1.1% Frank Haverkamp 7424 1.1% Takashi Iwai 6247 0.9% Thomas Hellstrom 6148 0.9% Tom Lendacky 6103 0.9% Upinder Malhi 6012 0.9% Sujith Manoharan 5837 0.8% Peter De Schrijver 5680 0.8% H Hartley Sweeten 5586 0.8% Rob Clark 5345 0.8%
After a brief absence, H. Hartley Sweeten has returned to the top of the "by changesets" list with more work on the seemingly interminable project of fixing up the Comedi drivers in the staging tree. Laurent Pinchart did a lot of work in the Video4Linux and ARM architecture trees, while Jingoo Han continues to work on cleaning up the driver tree. Rashika Kheria, an Outreach Program for Women participant, contributed lots of driver cleanup patches, and Geert Uytterhoeven did a lot of work in the m68k architecture and embedded driver subsystems.
Greg Kroah-Hartman often appears near the top of the "lines changed" column; this time, it is as the result of the addition of the Realtek 8821 PCI WIFI driver and more than the usual number of reverted patches. Micky Ching added the rts5208 and rts5288 drivers to the staging tree, Stephen Boyd added a bunch of Qualcomm hardware support, Paul Gortmaker did (among other things) a bunch of header file cleanups, and Greg Rose contributed the Intel i40evf network driver.
At least 210 employers supported work on the 3.14 kernel. The most active of these employers were:
Most active 3.14 employers
By changesets Intel 1233 10.2% (None) 1075 8.9% Red Hat 877 7.3% (Unknown) 701 5.8% Linaro 528 4.4% Samsung 523 4.3% SUSE 396 3.3% IBM 351 2.9% Renesas Electronics 339 2.8% 324 2.7% Texas Instruments 288 2.4% Vision Engraving Systems 278 2.3% (Consultant) 268 2.2% NVIDIA 248 2.1% FOSS Outreach Program for Women 237 2.0% Huawei Technologies 211 1.8% Freescale 210 1.7% Qualcomm 157 1.3% Oracle 152 1.3% Broadcom 144 1.2%
By lines changed Linux Foundation 78675 11.4% Intel 69526 10.0% (None) 47083 6.8% Red Hat 46371 6.7% Texas Instruments 28274 4.1% (Unknown) 25716 3.7% IBM 25427 3.7% Realsil Microelectronics 23676 3.4% SUSE 22686 3.3% NVIDIA 20720 3.0% Samsung 19988 2.9% Wind River 19946 2.9% Code Aurora Forum 17878 2.6% 13452 1.9% Linaro 12945 1.9% Cisco 12747 1.8% (Consultant) 12301 1.8% Qualcomm 10806 1.6% Renesas Electronics 9655 1.4% Freescale 9533 1.4%
There are few surprises here; instead, this table shows the continuation of some trends we have been seeing for a while. After a short-lived jump in 3.13, the number of contributions from volunteers is back to its long-term decline. Intel seems to have taken a permanent place at the top of the list of changeset contributors. Contributions from the mobile and embedded industry continue to grow. It's tempting to call out the fact that 3.14 will contain a fix to the nouveau driver that came directly from NVIDIA, but this turns out to be the second time that has happened; the first fix from NVIDIA was quietly merged for 3.9 in early 2013.
A slightly different picture emerges if one looks at non-author signoffs — Signed-off-by tags applied to patches by developers other than the author. Such tags are applied by subsystem maintainers as they accept patches; the statistics can thus indicate who the gatekeepers to the kernel are. Associating signoffs with employers leads to these results:
Most signoffs in 3.14
By developer Greg Kroah-Hartman 1516 13.1% David S. Miller 1128 9.7% Mark Brown 502 4.3% Andrew Morton 465 4.0% Mauro Carvalho Chehab 370 3.2% John W. Linville 352 3.0% Simon Horman 256 2.2% Arnaldo Carvalho de Melo 237 2.0% Daniel Vetter 225 1.9% Rafael J. Wysocki 173 1.5% Jeff Kirsher 166 1.4% Chris Mason 160 1.4% Linus Walleij 148 1.3% Ingo Molnar 146 1.3% Brian Norris 140 1.2% John Crispin 129 1.1% Jesse Brandeburg 127 1.1% Josef Bacik 126 1.1% Johannes Berg 125 1.1% Benjamin Herrenschmidt 121 1.0%
By employer Red Hat 2336 20.2% Linux Foundation 1548 13.4% Intel 1367 11.8% Linaro 1093 9.4% 649 5.6% Samsung 647 5.6% (None) 417 3.6% 283 2.4% SUSE 270 2.3% Renesas Electronics 265 2.3% IBM 262 2.3% Texas Instruments 200 1.7% Broadcom 183 1.6% (Unknown) 159 1.4% Fon 129 1.1% NVIDIA 97 0.8% Cisco 91 0.8% Pure Storage 91 0.8% (Consultant) 82 0.7% University of Cambridge 77 0.7%
The concentration of companies here is reduced from what it once was, but it is still true that more than half of the patches that were merged into 3.14 went by way of the employees of just four organizations. Linaro and Facebook are moving up the list; in both cases, this has mostly been done by hiring developers who were already working as subsystem maintainers. Here, too, the presence of volunteer developers has been slowly declining over time.
All told, the kernel development machine appears to be in a relatively steady state, with things running smoothly and changes happening over relatively long periods of time. Your editor is thus curious to know whether these reports remain useful on a per-release basis. Perhaps it makes sense to scale them back to, say, an annual exercise where the long-term trends might be more pronounced. This is a question that will be considered over the course of the 3.15 development cycle.
Unmixing the pool
One of the more useful outcomes from the Snowden revelations may well be the increased scrutiny of the security of our systems. While no one would (sensibly) claim that all of the problems have been found and fixed, there has been improvement in many different areas over the last year or so. One of the areas that has received some attention recently has been the kernel's random number generation. Two recent patches continue that trend, though it is hard to claim that they are the direct result of the exposure of NSA (and other secret agency) spying.
Both patches update the state of the "entropy pool" that is used to generate random numbers for both /dev/random and /dev/urandom. That pool is associated with a (conservative) estimate of the amount of entropy (i.e. state unknowable by an attacker) stored within it. Anything that is believed to truly add entropy gets credited in that estimate, while other, possibly even attacker-controlled, input is simply mixed into the pool without entropy credit. The entropy estimate is used to block /dev/random readers when the amount of data requested is larger than the amount of entropy in the pool.
Adding RDSEED
The first patch is from H. Peter Anvin and it simply adds support for the RDSEED instruction to the
kernel. RDSEED is an instruction being added to Intel processors
that returns "fully
conditioned entropy that is suitable for use as seeds to a PRNG
[pseudo-random number generator]
". The patches use four bytes of
RDSEED output
to mix into the entropy pool at boot time (with
no entropy credit). In addition, four bytes are generated using the
instruction once per second and then mixed into the pool.
It is also used to do an "emergency refill" with 64 bytes of RDSEED output
if /dev/random is
about to block due to a lack of entropy credit to fulfill a
request. In both of the latter cases, four bits of credit are given for
each byte of RDSEED output that gets mixed into the pool.
Some may not be convinced that a black-box hardware random number generator (RNG) buried inside an Intel chip should be given that much (or any) entropy credit. It is a difficult question, as there is no technical barrier to the instruction returning known-to-the-NSA sequences and there is no way for anyone (at least anyone outside of Intel) to know for sure. While that may seem paranoid, many formerly paranoid scenarios have moved into the "plausible" category over the last year. That concern has not been raised about the RDSEED patches, however.
Mixing and unmixing
The other patch, from Kees Cook, would add some
output from a newly instantiated hardware RNG into the entropy pool.
When the RNG is
registered (via hwrng_register()), sixteen bytes of its output
would get mixed into the pool, but without any entropy credit. Jason
Cooper was concerned that even mixing these
bytes into the pool could lead to problems: "By adding this patch, even without crediting entropy to the pool, a
rogue hwrng now has significantly more influence over the initial state
of the entropy pools.
"
But Cook didn't see it as any different than mixing in other random or system-specific data at initialization time:
In addition, former random subsystem maintainer Matt Mackall brought up an important aspect of the design of the mixing function. Because it can be reversed, mixing even attacker-controlled data into the pool can never remove randomness that was there at the outset:
That means, if I have an initial secret pool state X, and hostile attacker controlled data Y, then we can do:
X' = mix(X, Y)
and
X = unmix(X', Y)
We can see from this that the combination of (X' and Y) still contain the information that was originally in X. Since it's clearly not in Y.. it must all remain in X'.
That didn't entirely mollify Cooper, who was still concerned that built-in hardware RNGs would have their output mixed in early in the boot sequence. He was worried that those bytes could pollute the pool, but Mackall reiterated his argument, putting it in starker terms:
Put another way: mixing can't ever [remove] unknownness from the pool, it can only add more. So the only reason you should ever choose not to mix something into the pool is performance.
While the mixing function design with reversibility in mind (and its implications) was clear in Mackall's mind, it would seem that others in the kernel community did not know about it. It's an interesting property that makes perfect sense, once you know about it, but is rather counter-intuitive otherwise. In any case, Cooper's objections were withdrawn, and hardware RNG maintainer Herbert Xu queued the patch. We should see it in 3.15.
MCS locks and qspinlocks
Impressive amounts of effort have gone into optimizing the kernel's low-level locking mechanisms over the years, but that does not mean that there is no room for improving their performance further. Some work that will be in theConceptually, a spinlock is a simple mechanism. The lock can be as small as a single bit; if that bit is clear, the lock is available. A thread wanting to acquire the lock will attempt to set that bit with an atomic compare-and-swap instruction, "spinning" repeatedly if the lock is not available at the time. Over the years, spinlocks have become more complex; ticket spinlocks added fairness to the mechanism in 2008, and 2013 saw the addition of better paravirtualization support, for example.
Spinlocks still suffer from a fundamental problem beyond the fact that simply spinning for a lock can be painful, though: every attempt to acquire a lock requires moving the cache line containing that lock to the local CPU. For contended locks, this cache-line bouncing can hurt performance significantly. So it's not surprising that developers have been working to reduce cache contention with spinlocks; an attempt to add automatic backoff to spinlocks in early 2013 was working toward that goal, for example, but this work was never merged.
MCS locks
More recently, Tim Chen put together a different approach based on a primitive called an "MCS lock" (described in this paper [PDF]). By expanding a spinlock into a per-CPU structure, an MCS lock is able to eliminate much of the cache-line bouncing experienced by simpler locks, especially in the contended case.
In 3.14 the tip tree for 3.15, an MCS lock is defined by
an instance of this structure:
struct mcs_spinlock {
struct mcs_spinlock *next;
int locked; /* 1 if lock acquired */
};
If one is willing to put up with your editor's second-rate drawing skills, one could envision an unlocked MCS spinlock as follows:
![[Unlocked MCS lock]](https://static.lwn.net/images/2014/mcslock1.png)
When a CPU comes along with a desire to acquire this lock, it will provide an mcs_spinlock structure of its own. Using an unconditional atomic exchange operation, it stores the address of its own structure in the lock's next field and marks the lock as taken, yielding a situation that looks like this:
![[Locked MCS lock]](https://static.lwn.net/images/2014/mcslock2.png)
The atomic exchange will return the previous value of the next pointer. Since that pointer was null, the acquiring CPU knows that it was successful in acquiring the lock. Once that is done, the lock is busy, but there is no contention for it. Should a second CPU come along and try to acquire the lock, it will start in the same way, storing a pointer to its mcs_spinlock structure in the next pointer of the main lock:
![[Contending an MCS lock]](https://static.lwn.net/images/2014/mcslock3.png)
When the second CPU does this atomic swap on the main lock, it too will get back the previous contents of the next field — the pointer to the first CPU's mcs_spinlock structure. The non-NULL value tells the second CPU that the lock is not available, while the specific pointer value says who is ahead in line for the lock. The second CPU will respond to this situation by storing a pointer to its mcs_spinlock structure in the next field of CPU 1's structure:
![[Contending an MCS lock]](https://static.lwn.net/images/2014/mcslock4.png)
Note that the use of an atomic swap operation on the main lock means that only CPU 2 can have a pointer to CPU 1's mcs_spinlock structure. So there is no need for atomic operations when making changes to that structure, though some careful programming is still needed to make sure that changes are visible to CPU 1 at the right times.
Once this assignment is done, CPU 2 will spin on the locked value in its own mcs_spinlock structure rather than the value in the main lock. Its spinning will thus be entirely CPU-local, not touching the main lock at all. This process can go on indefinitely as contention for the lock increases, with each CPU placing itself in line behind those that are already there, and each CPU spinning on its own copy of the lock. The pointer in the "main" lock, thus, always indicates the tail of the queue of waiting CPUs.
When CPU 1 finally finishes with the lock, it will do a compare-and-swap operation on the main lock, trying to set the next pointer to NULL on the assumption that this pointer still points to its own structure. If that operation succeeds, the lock was never contended and the job is done. If some other CPU has changed that pointer as shown above, though, the compare-and-swap will fail. In that case, CPU 1 will not change the main lock at all; instead, it will change the locked value in CPU 2's structure and remove itself from the situation:
![[Transferred MCS lock]](https://static.lwn.net/images/2014/mcslock5.png)
Once its copy of locked changes, CPU 2 will break out of its spin and become the new owner of the lock.
An MCS lock, thus, is somewhat more complicated than a regular spinlock. But that added complexity removes much of the cache-line bouncing from the contended case; it also is entirely fair, passing the lock to each CPU in the order that the CPUs arrived.
Qspinlocks
In the tip tree, MCS locks are used in the implementation of mutexes, but they do not replace the existing ticket spinlock implementation. One reason for this is size: ticket spinlocks fit into a single 32-bit word, while MCS locks do not. That turns out to be important: spinlocks are embedded into many kernel structures, some of which (notably struct page) cannot tolerate an increase in size. If the MCS lock technique is to be used throughout the kernel, some other approach will be needed.
The version of that approach which is likely to be merged can be seen in the "qspinlock" patch series from Peter Zijlstra which, in turn, is based on an implementation by Waiman Long. In this patch set, each CPU gets an array of four mcs_spinlock structures in a well-known location. Four structures are needed because a CPU could be trying to acquire more than one spinlock at a time: imagine what happens if a hardware interrupt comes in while a thread is spinning on a lock, and the interrupt handler tries to take a lock of its own, for example. The array of structures allows lock acquisition attempts from the normal, software interrupt, hardware interrupt, and non-maskable interrupt contexts to be kept apart.
The 32-bit qspinlock is divided into a number of fields:
- an integer counter to function like the locked field described above,
- a two-bit "index" field saying which entry in the per-CPU mcs_spinlock array is used by the waiter at the tail of the list,
- a single "pending" bit, and
- an integer field to hold the CPU number indicating the tail of the queue.
The last field is arguably the key: by storing a CPU number rather than a pointer, the qspinlock patch set allows all of the relevant information to be crammed into a single 32-bit word. But there are a couple of other optimizations to be found in this patch set as well.
One has to do with the value used by each CPU for spinning. When a CPU is next in line to acquire a lock, it will spin on the lock itself instead of spinning on its per-CPU structure. In this way, the per-CPU structure's cache line need not be touched when the lock is released, removing one cache line miss from the equation. Any subsequent CPUs will spin on their own structures until they reach the head of the queue.
The "pending" bit extends that strategy a bit further. Should a CPU find that a lock is busy but that no other CPUs are waiting, it will simply set the pending bit and not bother with its own mcs_spinlock structure at all. The second CPU to come along will see the pending bit, begin the process of building the queue, and spin on its local copy of the locked field as usual. Cache-line bouncing between waiters is still eliminated, but the first waiter is also able to avoid the cache-miss penalty associated with accessing its own mcs_spinlock array.
Performance-oriented patches should, of course, always be accompanied by benchmark results. In this case, Waiman included a set of AIM7 benchmark results with his patch set (which did not include the pending-bit optimization). Some workloads regressed a little, but others shows improvements of 1-2% — a good result for a low-level locking improvement. The disk benchmark runs, however, improved by as much as 116%; that benchmark suffers from especially strong contention for locks in the virtual filesystem layer and ext4 filesystem code.
The qspinlock patch can, thus, improve performance in situations where locks are highly contended, though, as Waiman noted in the patch posting, it is not meant to be an actual solution for contention problems. Importantly, qspinlocks also perform better in the uncontended case. Releasing a ticket spinlock requires a read-modify-write operation, while a qspinlock can be released with a simple write. So, while the main performance benefits of qspinlocks are to be seen on large systems, most systems should see at least a small improvement. That should be enough to get this code merged as soon as the 3.15 development cycle.
Patches and updates
Kernel trees
Architecture-specific
Core kernel code
Development tools
Device drivers
Documentation
Filesystems and block I/O
Memory management
Networking
Security-related
Miscellaneous
Page editor: Jonathan Corbet
Distributions
A setuid wrapper for X.org
X11 has long been a source of anxiety for the security-conscious, and understandably so: the X.org codebase is complicated, several decades old, and when there are security problems they can be massive in scope (such as in the May 2013 X client vulnerability disclosure). But despite the fact that most Linux distributions are working toward replacing the X server, the X.org project itself is still making incremental improvements. One such improvement is the ability to automatically run the X server as an unprivileged user for most GPUs, a change that looks like it could ship in Fedora 21.
With the availability of kernel mode setting (KMS) for the majority of graphics cards, it has been possible to run the X server as a non-root UID for quite some time. Intel's Moblin project did so (for Intel GPUs) in 2009; Ubuntu made it an option in 2010, and several other distributions have followed suit. But it has never been the default on a desktop distribution, perhaps because proprietary, binary-only drivers have remained too popular with users.
Let's wrap about X
On March 5, Hans de Goede posted a patch to the Xorg development list that offered a viable solution: a small "setuid wrapper" for the X server binary. If the wrapper detects that only KMS-supporting hardware and drivers are in use, it can launch the X server as a non-root UID. If KMS cannot be used, the wrapper can launch the X server as root.
In more detail, the patch starts with a shell script that looks for the wrapper; if there is no wrapper, the script launches the X server directly. But when the wrapper gets started instead, it first checks to see if all of the video adapters in use have KMS support and if the KMS drivers are loaded and working. There is, after all, the possibility that one KMS-supported card and one non-KMS card will both be in use, as well as the chance that the user wants to run a binary-only driver on a KMS-supported card. If the KMS conditions are met, then the wrapper drops the root privileges and launches the X server binary. If there are non-KMS components (or if there are no graphics cards present at all), then the wrapper starts the X server as root just as has traditionally been done in years past.
De Goede asked for feedback, which several other X.org developers supplied. Alan Coopersmith pointed out that Debian already has a utility called Xwrapper that is used to control the X server's execution environment. De Goede replied that he had not heard of it before, took a look, then pointed out what he saw as the good and bad points about it.
On the plus side, he said, it has a configuration file, supports a setting in which only local console users can start X, checks to see if the real X server is executable by the chosen UID, and performs some checks on /tmp/.X11. On the down side, he said, it installs the wrapper executable at /usr/bin/X rather than /usr/bin/Xorg, checks to see that the real X server is not just a symlink back to the wrapper, and is released under the GPL, which De Goede noted was unsuitable for X.org. Thus, he decided to duplicate the features he liked from Xwrapper, implement an Xwrapper-compatible configuration file, and hoped that Debian would switch over to his wrapper in the future.
Mark Kettenis from OpenBSD replied
that OpenBSD's Xenocara X server
has had its own approach to privilege separation from quite some time,
and questioned the wrapper approach. The Xenocara server, he said,
still runs with the setuid bit set, but it drops as many privileges as
it can. " De Goede responded that the log
file issue was not particularly important, and that with binary-only
drivers, it is simply not possible to know which rights the driver
needs and which can be dropped. " De Goede has also proposed adopting the wrapper as a change
for Fedora 21. The Fedora Engineering Steering Committee (FESCo) has
yet to decide on the proposal, but, if Fedora successfully implements the
feature and includes it, that would be the first mainstream desktop
distribution to offer a non-root X server by default.
Implementing the change will require alterations to
display managers like GNOME's GDM and KDE's SDDM, since the display
manager is the process responsible for starting X. Specifically, the
display manager must start the X server inside a user session;
fortunately Wayland also needs to be started inside a user session, so
display managers also interested in offering Wayland support can
handle both servers in the same fashion.
The feature tracking page notes that GDM and SDDM developers have
both started working on the required change; the XFCE, LXDE, and Mate
teams have been contacted, but so far there is no indication of
whether or not they plan to implement the change. De Goede's wrapper
is anticipated to land in X.org's xserver 1.16 release, which is
expected in July.
The benefits of such a change are, of course, provisory: if a
security bug is found in the X server, having the server run with
minimal permissions limits the scope of the potential damage. But it
is always good policy to grant the least possible set of privileges to
a component, so the change is still welcome—even if the X server
is due to be replaced in its entirety before too long. Then again,
when it comes to replacing components, one rarely gets in trouble for
overestimating how long X will continue to live on.
What you care about is dropping as many access
rights as possible, and being setuid you might actually be able to
drop more of them
", he said, adding also that this approach
allowed the server to use the traditional log files.
From the Linux POV the wrapper
is a workaround for these drivers; I hope that in the future we can
stop shipping the wrapper by default.
"
The Fedora change
Brief items
Distribution quotes of the week
Distribution News
Debian GNU/Linux
Bits from the Security Team
The Debian Security Team presents a summary of last month's team meeting. Topics include the Debian Security Tracker, security release workflow, security archive, and more. "At the moment it seems likely that an extended security support timespan for squeeze is possible. The plan is to go ahead, sort out the details as as it happens, and see how this works out and whether it is going to be continued with wheezy."
Newsletters and articles of interest
Distribution newsletters
- DistroWatch Weekly, Issue 549 (March 10)
- Tails report (February)
- Ubuntu Weekly Newsletter, Issue 358 (March 9)
Peppermint introduces cloud-based open source desktop to Africa (opensource.com)
Opensource.com talks with Kendall Weaver and Shane Remington about Peppermint Linux and how it's being used in Cameroon and elsewhere. "Remington: One of the key areas that we wanted to address with Peppermint, as a cornerstone, was that we had to be the absolute best Linux distribution when it came to inclusion. Everything we would do--community, social, support--would have to have a much different tone than the Linux that I grew up learning. And I mean having to prove your worth in the forums, even to ask simple questions, and being made to feel inadequate frequently. Kendall and I both agreed that those elements needed to be buried forever and I feel we've done a great job of that. Nowadays, the distance to the other side of the globe doesn't exist anymore with the Internet that's always on no matter where you are. We also had to attract the properly tempered support administrators and moderators who saw our vision clearly and would vow to maintain the positive and open vibe to the entire world. I can't say enough about how lucky we are for these guys who have been with us from the beginning. Its amazing what these guys can do."
Page editor: Rebecca Sobol
Development
Glibc feature test macros
As evidenced by the frequency of misreported bugs to the Linux man-pages project, feature test macros (FTMs) would appear to be one of the topics that are most widely misunderstood by C programmers. The fundamental idea of FTMs is simple, but there are quite a few subtle details to master, and at least one pitfall that regularly traps programmers. This article will look at how FTMs are used in the GNU C library (glibc) and how to avoid some common problems.FTMs are a technique used to control the definitions exposed by C header files. For example, in the clock_getres(3) manual page, we see the following text in the SYNOPSIS section:
Feature Test Macro Requirements for glibc (see feature_test_macros(7)):
clock_getres(), clock_gettime(), clock_settime():
_POSIX_C_SOURCE >= 199309L
This text tells us that the <time.h> header file will expose the declarations of these three system calls (or, more precisely, their C library wrapper functions) if _POSIX_C_SOURCE is defined with the value 199309L or greater. Beginning with man-pages version 2.64 in 2007, I started adding FTM annotations such as the above to various manual pages. By now, all of the pages that require such annotations (some 350) have them. (Any omissions or errors are bugs that should be reported.)
Feature test macros and formal standards
There are several different FTMs, and a full list can be found in the feature_test_macros(7) manual page. Many of the FTMs fall into two broad classes. The first of these are the FTMs that relate to various formal standards—for example, the C language standards or Unix standards such as POSIX and the Single Unix Specification. These macros have names such as _ISOC99_SOURCE, _POSIX_C_SOURCE, and _XOPEN_SOURCE. The purpose of these macros is to cause a header file to expose definitions in a manner that conforms to a particular standard.
To take an example, we can define _POSIX_C_SOURCE with different values in order to have header files expose definitions and declarations associated with different versions of POSIX. This FTM was specified in the first (1990) edition of POSIX; defining it with the value 1 causes various header files to expose definitions associated with that POSIX version. As that standard evolved, a more coherent numbering scheme was devised, based on the year and month that the standard was released. Thus, the 1993 POSIX.1b (realtime) standard specifies that the value 199309L should cause header files to expose definitions relating to that standard. Similarly, the value 200112L causes definitions associated with the POSIX.1-2001 standard to be exposed, and the value 200809L causes definitions associated with the POSIX.1-2008 standard to be exposed.
In the snippet from the clock_getres(2) manual page shown above, the value 199309L is required because those three APIs were first standardized in POSIX.1b (1993). Standards tend to be cumulative, so that most of the APIs specified in POSIX.1b were also specified in POSIX.1-2001 and POSIX.1-2008, which is why defining _POSIX_C_SOURCE with any value greater than 199309L will also cause the POSIX.1b declarations to be exposed.
There are a number of other FTMs associated with various standards. For example, defining _ISOC99_SOURCE (with any value) causes header files to expose definitions associated with the C99 standard. The _XOPEN_SOURCE macro causes header files to expose definitions associated with various versions of the Single Unix Specification (SUS): 1 (or greater) to get definitions for SUSv1, 500 (or greater) to get definitions associated with SUSv2, 600 (or greater) to get definitions associated with SUSv3, and so on.
The "00" numbering scheme used for the _XOPEN_SOURCE values relates to the XPG (X/Open Portability Guide) standards out of which SUS grew. Thus, SUSv2 is also (occasionally) known as XPG5, and so the number 500 was used for the corresponding FTM. The next SUS release, version 3, adopted the value 600, and SUSv4 adopted the value 700. Adding a couple of zeroes to the numbers presumably permits the standard to assign meaning to intermediate values between those numbers, for something like minor revisions of the standard, although in practice no intermediate values have so far been defined.
Feature test macros and implementation standards
A second class of glibc-specific FTMs relate to historical implementation standards, in particular System V and BSD. Defining one of these macros was sometimes helpful when porting older software from one of those implementations to Linux.
Defining the macro _BSD_SOURCE (with any value) causes header files to expose definitions derived from BSD. Thus, the setgroups(2) manual page says that we must define _BSD_SOURCE to obtain the declaration of that system call:
Feature Test Macro Requirements for glibc (see feature_test_macros(7)):
setgroups(): _BSD_SOURCE
This corresponds to the fact that setgroups() is originally a BSD invention that is not specified in any formal standard.
Explicitly defining the _BSD_SOURCE macro has one other effect: in a very few cases where standards conflict, it causes BSD definitions to be favored. One example of such a conflict is the getpgrp() function, which exists in two forms in glibc: a POSIX version and a BSD version. By default, <unistd.h> exposes the POSIX version, unless _BSD_SOURCE is explicitly defined.
Analogously, the _SVID_SOURCE macro is used to obtain some definitions derived from the System V Interface Definition, a vendor-defined standard for System V.
The _GNU_SOURCE macro is an FTM invented by the glibc developers. Defining this macro (with any value) serves two main purposes. First, this macro is commonly used to guard Linux-specific and GNU-specific definitions that are part of standard header files. Thus, to get the declaration of the Linux-specific ppoll() system call from <poll.h>, which also provides the poll() call specified by various Unix standards, we must define _GNU_SOURCE.
The second purpose of _GNU_SOURCE is to say "give me everything". Defining this macro causes almost all of the other FTMs to be defined (with their maximum meaningful values), with the result that header files expose (almost) all of the definitions they contain.
It's complicated
Standards evolve and sometimes conflict; glibc has evolved to follow the standards, and occasionally changed its use of FTMs to fix bugs or be more consistent. Furthermore, sometimes standards are subtractive, meaning that a standard can remove an interface. A notable example occurred in the C11 standard, which removed the dangerous gets() function. Thus, recent versions of glibc don't expose this function if the _ISOC11_SOURCE FTM is defined.
The syntax used in the man pages reflects these complexities. Thus, for example, in the dprintf(3) page we see:
Feature Test Macro Requirements for glibc (see feature_test_macros(7)):
dprintf(), vdprintf():
Since glibc 2.10:
_XOPEN_SOURCE >= 700 || _POSIX_C_SOURCE >= 200809L
Before glibc 2.10:
_GNU_SOURCE
This tells us that before glibc version 2.10, it was necessary to define _GNU_SOURCE to obtain the declarations of these functions. However, these functions were eventually standardized in the combined POSIX.1-2008/SUSv4 standard, and glibc 2.10 and later adjusted to that fact, so that now it suffices either to define _XOPEN_SOURCE with a value greater than or equal to 700 or to define _POSIX_C_SOURCE with a value greater than or equal to 200809L.
The <features.h> header file
The logic dealing with FTMs is implemented within the <features.h> header file. That header file defines various internal macros which are used within each of the other glibc header files to control which definitions are exposed by the header file. (Programs never need to explicitly include <features.h>: it is implicitly included by each glibc header file that makes use of FTMs.)
The internal macros have names such as __USE_GNU (for _GNU_SOURCE), __USE_BSD (for _BSD_SOURCE), __USE_SVID (for _SVID_SOURCE), and __USE_MISC (defined if either _BSD_SOURCE or _SVID_SOURCE is defined). FTMs that can take different values are implemented as multiple internal macros. So, for example, there are __USE_POSIX199309, __USE_POSIX199506, __USE_XOPEN2K, and __USE_XOPEN2K8, each set when _POSIX_C_SOURCE is greater than or equal to, respectively, 199309L, 199506L, 200112L, and 200809L.
Certain FTMs are defined by default, principally _BSD_SOURCE, _SVID_SOURCE, and _POSIX_C_SOURCE with the value 200909L (which causes the definitions for the POSIX.1-2008 base standard to be exposed).
Individual macros can be explicitly defined, using either #define in the C source code or -D directives on the compiler command line. Multiple definitions are additive. However, there is a potential surprise: in most cases, if any FTM is explicitly defined, then the default FTMs are not defined by default, and they too must be explicitly defined. The details of the default definitions, as well as certain other implicit definitions that take place when certain FTMs are explicitly defined, have varied across glibc versions. Most of those details can be found in the feature_test_macros(7) manual page.
A regular pitfall
Over the years, probably the single most common recurrent bug report to the man-pages project is the claim that defining a particular FTM does not expose a definition as documented in a manual page. Often, the reporter has inspected the header files, and discovered that explicitly defining one of glibc's internal macros such as __USE_MISC (which should never be done outside of the glibc code), causes the desired definition to be exposed, and so the report includes a claim that the manual page should document the use of that macro.
Notwithstanding occasional bugs in the documented FTM requirements, the problem is always the same: FTMs must be defined before including any header files. The reason for this requirement is that various header files may implicitly include other header files, and any of those files may in turn include <features.h>. If a macro is defined after <features.h> is included, then it has no effect, since that header file (like most library header files) includes a guard macro that ensures that the contents of the file are processed only on the first inclusion of the file.
Changes in glibc 2.19 and 2.20
The original intent seems to have been that, within each of the glibc header files that employs FTMs, only one of the __USE_* internal macros should govern the exposure of any particular definition. Additionally, the macros should not be used in nested #ifdef directives. An inspection of the glibc header files quickly shows that the reality is far from the intent, a situation that led Roland McGrath to suggest that it was time for a major cleanup to bring things back to the intended situation. Roland thought that task could be simplified by eliminating the _BSD_SOURCE and _SVID_SOURCE FTMs, which, although they had a purpose historically, have ceased to be useful these days. Anymore, he said, the only macros that are needed for modern source code are those that relate to formal standards plus _GNU_SOURCE.
Joseph Myers duly obliged with a series of patches to implement the first steps in this work. The conservative approach encouraged by Roland meant that the deprecation of the _BSD_SOURCE and _SVID_SOURCE FTMs is taking place across two glibc versions. Version 2.19 of glibc added a new FTM, _DEFAULT_SOURCE. Defining this macro causes the default definitions to be exposed even when the explicit definition of other macros would cause that not to happen. The effect of defining this macro is equivalent to the effect of explicitly defining three macros in earlier glibc versions:
cc -D_BSD_SOURCE -D_SVID_SOURCE -D_POSIX_C_SOURCE=200809C
In addition, glibc 2.19 removed the behavior that explicitly defining _BSD_SOURCE caused BSD definitions to be favored in the case of standards conflicts.
Some further patches from Joseph complete the deprecation of _BSD_SOURCE and _SVID_SOURCE for the upcoming glibc 2.20 release. The first of these patches converts _BSD_SOURCE and _SVID_SOURCE into synonyms for _DEFAULT_SOURCE. The deprecated macros remain effective, but cause compile-time warnings to be generated if they are used (unless _DEFAULT_SOURCE is also defined). The second patch (mechanically) converts all of the uses of the __USE_SVID and __USE_BSD internal macros in glibc header files into __USE_MISC.
The final patch does most of the real work: Joseph inspected all of the glibc header files to eliminate redundant uses of the __USE_MISC internal macro. These redundant uses range from doubled instances of __USE_MISC created by the preceding patch through to cases where __USE_MISC was combined (with a logical OR) with other macros that logically imply that __USE_MISC must also be defined. For example, in all cases where __USE_MISC is defined in glibc 2.20, __USE_XOPEN2K8 is also defined, meaning that conditionals such as the following in the glibc headers:
#if defined __USE_MISC || defined __USE_XOPEN2K8
can be simplified to:
#ifdef __USE_XOPEN2K8
From here on, there will probably be some further cleanups in the glibc headers. While it may not be practical in all cases to reach Roland's goal that only one of the internal __USE_* macros should govern the exposure of any particular definition, some simplifications can probably be achieved. There are likely also other cases similar to those involving __USE_MISC where there are conditionals involving two or more mutually dependent __USE_* macros that can be simplified by removing one of the macros.
Joseph has also hinted that there may be scope for obsoleting some interfaces or removing the __USE_MISC guard from conditionals such as __USE_MISC || __USE_UNIX98, so that some interfaces are not exposed by default. Certainly, there are currently a number of anomalous situations where some Linux-specific definitions are guarded by __USE_GNU and thus are exposed only if _GNU_SOURCE is defined (for example, the msgget() MSG_EXCEPT flag) while other (in some cases, closely related) Linux-specific definitions are guarded by __USE_MISC and are thus exposed by default (for example, the msgctl() MSG_STAT flag). Quite how the glibc developers will deal with issues like this, and what other FTM rationalizations and simplifications will be made for individual definitions, is matter that we are likely to see play out over the next couple of glibc releases.
Brief items
Quotes of the week
"I am altering the deal. Pray I don't alter it any further." There's a reason Darth Vader wasn't proclaimed Python's BDFL ;-)
For some people, sometimes, they are essentially synonymous, because they are often encountered together. But of course they are fundamentally different things
In this analogy, Email is like Wine: it’s the other universally popular use of the Internet/Alcohol.
But there are lots of other uses, too, somewhat more obscure. We could say the various chat protocols are the various Whiskeys. IRC is Scotch; XMPP is Bourbon.
gopher is obscure and obsolete, …. maybe melomel.
ssh is potato vodka.
Calligra 2.8 released
The Calligra office suite has announced its 2.8 release. In addition to the Krita painting program's 2.8 release (which we reviewed in the March 6 edition of LWN), other components have new features and bug fixes as well. Words and Author gained support for document comments, Kexi (visual database tool) has many fixes and improvements based on user feedback, Sheets now has pivot tables, and so on. "There are also some general improvements in all apps. It is now possible to copy and paste any shape between any documents in Calligra applications. Moreover, copying and pasting of images and rich text is now more advanced."
Xen Project Hypervisor version 4.4 available
The Xen Project has released version 4.4 of its hypervisor. This release enhances support for the ARM architecture, including support for AppliedMicro’s X-Gene, the Arndale Board, Calxeda ECX-2000, TI OMAP5 and the Allwinner SunXi boards, and 64-bit guest support on ARMv8 machines. Other changes in this release include a new event channel implementation contributed by Citrix and an experimental paravirtualization mode contributed by Oracle.
Mozilla cutting support for Persona
Mozilla has announced that it is "transitioning Persona to community ownership" — or, in other words, dropping development support for this identity management project. "For a variety of reasons, Persona has received less adoption than we were hoping for by this point. However, we do still believe that Persona offers a unique and useful alternative to passwords, and we intend to support it as such. Reducing the scope of Persona and stabilizing its core APIs over the last quarter has shown us that adding more features was not the way forward." (LWN reported on Persona in March 2013).
Newsletters and articles
Development newsletters from the past week
- What's cooking in git.git (March 11)
- LLVM Weekly (March 10)
- OCaml Weekly News (March 11)
- OpenStack Community Weekly Newsletter (March 7)
- Perl Weekly (March 10)
- PostgreSQL Weekly News (March 10)
- Python Weekly (March 6)
- Qt Weekly (March 11)
- Ruby Weekly (March 6)
- This Week in Rust (March 12)
- Tor Weekly News (March 12)
Packard: Glamor hacking
At his blog, Keith Packard has posted a write-up of
recent work by Eric Anholt and himself to accelerate 2D functions in X with
OpenGL, with promising results. "These two cases, text and points, are probably the
hardest operations to accelerate with a GPU and yet a small amount of
OpenGL code was able to meet or beat software easily. The advantage of
this work over traditional GPU 2D acceleration implementations should
be pretty clear — this same code should work well on any GPU which
offers a reasonable OpenGL implementation. That means everyone shares
the benefits of this code, and everyone can contribute to making 2D
operations faster.
"
Page editor: Nathan Willis
Announcements
Brief items
Linux Foundation teams up with edX to build free online Linux course
The Linux Foundation has announced that it is building a massive open online course (MOOC) with edX, the non-profit learning platform created by Harvard University and Massachusetts Institute of Technology (MIT). "The Linux Foundation and edX are partnering to develop a MOOC program that will help address this issue by making basic Linux training materials available to all for free. Previously a $2,400 course, Introduction to Linux will be the first class available as a MOOC and will be free to anyone, anywhere. The Linux Foundation is among a new group of member organizations edX announced today who will contribute courses to the platform. EdX’s MOOC’s are an increasingly popular way to provide for unlimited participation and open access to learning material to people anywhere in the world via the web. These programs also provide interactive users forums where students and professors can build communities, similar to the way in which the Linux community collaborates. MOOCs have recently generated enrollments for individual classes of 60,000 or more students."
Articles of interest
FSFE Newsletter – March 2014
This month the Free Software Foundation Europe Newsletter looks at vendor lock-in in the UK, why Open Standards aren't enough, motivation for Free Software contributors, and much more.Google's coding internship summer program reaches 10th year (opensource.com)
The 2014 edition of Google's Summer of Code (GSoC) marks the tenth year of the program. Opensource.com covers a few of this year's 190 mentoring organizations. "What is likely to remain the same this year is the overwhelming response from students from all over the world who want the chance to work on free and open source projects with mentoring organizations that Google has hand-picked. Carol Smith, Open Source Programs Manager at Google, tells us that to date GSoC students have helped generate over 50 million lines of open source code to date, from over 8,500 student developers."
SCALE 12X: A long-overdue Thank You
The Southern California Linux Expo team presents a postmortem of last month's event, and a thank you to all involved. "The SCALE Team would like to thank everyone who made it to the Hilton Los Angeles Airport hotel for the three days that made up one great show. The team would also like to thank the speakers and session leads who made the show a learning experience for all. We would also like to recognize all the people who make the show run – the volunteers who put in what seems like endless hours – to make this the best, and largest, community-run show in North America."
New Books
"Is parallel programming hard?", first edition
Paul McKenney has announced that the first edition of his 500-page book Is Parallel Programming Hard, And, If So, What Can You Do About It? is available in electronic form; a printed version will follow soon. The entire book is available under the CC-BY-SA 3.0 license.
Calls for Presentations
Call for Proposals: PyCon AU 2014
PyCon Australia will take place August 1-3 in Brisbane. The call for proposals deadline is April 25. "Presentation subjects may range from reports on open source, academic or commercial projects; or even tutorials and case studies. If a presentation is interesting and useful to the Python community, it will be considered for inclusion in the program."
CFP Deadlines: March 13, 2014 to May 12, 2014
The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.
| Deadline | Event Dates | Event | Location |
|---|---|---|---|
| March 14 | May 20 May 22 |
LinuxCon Japan | Tokyo, Japan |
| March 14 | July 1 July 2 |
Automotive Linux Summit | Tokyo, Japan |
| March 14 | May 23 May 25 |
FUDCon APAC 2014 | Beijing, China |
| March 16 | May 20 May 21 |
PyCon Sweden | Stockholm, Sweden |
| March 17 | June 13 June 15 |
State of the Map EU 2014 | Karlsruhe, Germany |
| March 21 | April 26 April 27 |
LinuxFest Northwest 2014 | Bellingham, WA, USA |
| March 31 | July 18 July 20 |
GNU Tools Cauldron 2014 | Cambridge, England, UK |
| March 31 | September 15 September 19 |
GNU Radio Conference | Washington, DC, USA |
| March 31 | June 2 June 4 |
Tizen Developer Conference 2014 | San Francisco, CA, USA |
| March 31 | April 25 April 28 |
openSUSE Conference 2014 | Dubrovnik, Croatia |
| April 3 | August 6 August 9 |
Flock | Prague, Czech Republic |
| April 4 | June 24 June 27 |
Open Source Bridge | Portland, OR, USA |
| April 5 | June 13 June 14 |
Texas Linux Fest 2014 | Austin, TX, USA |
| April 7 | June 9 June 10 |
DockerCon | San Francisco, CA, USA |
| April 14 | May 24 | MojoConf 2014 | Oslo, Norway |
| April 17 | July 9 | PGDay UK | near Milton Keynes, UK |
| April 17 | July 8 | CHAR(14) | near Milton Keynes, UK |
| April 18 | November 9 November 14 |
Large Installation System Administration | Seattle, WA, USA |
| April 18 | June 23 June 24 |
LF Enterprise End User Summit | New York, NY, USA |
| April 24 | October 6 October 8 |
Operating Systems Design and Implementation | Broomfield, CO, USA |
| April 25 | August 1 August 3 |
PyCon Australia | Brisbane, Australia |
| April 25 | August 18 | 7th Workshop on Cyber Security Experimentation and Test | San Diego, CA, USA |
| May 1 | July 14 July 16 |
2014 Ottawa Linux Symposium | Ottawa, Canada |
| May 1 | May 12 May 16 |
Wireless Battle Mesh v7 | Leipzig, Germany |
| May 2 | August 20 August 22 |
LinuxCon North America | Chicago, IL, USA |
| May 2 | August 20 August 22 |
CloudOpen North America | Chicago, IL, USA |
| May 3 | May 17 | Debian/Ubuntu Community Conference - Italia | Cesena, Italy |
| May 4 | July 26 August 1 |
Gnome Users and Developers Annual Conference | Strasbourg, France |
| May 9 | June 10 June 11 |
Distro Recipes 2014 - canceled | Paris, France |
If the CFP deadline for your event does not appear here, please tell us about it.
Upcoming Events
LUG 2014 agenda announced
Open Scalable File Systems (OpenSFS) has announced the agenda for the 12th Annual Lustre User Group (LUG) conference, to be held April 8-10 in Miami, Florida. The event will feature 29 sessions including the inaugural Lustre Annual Report from OpenSFS and "deep technical presentations from DataDirect Networks, Intel, Lawrence Livermore National Laboratory (LLNL), Los Alamos National Laboratory (LANL), NetApp, Oakridge National Laboratory (ORNL), Texas Advanced Computing Center (TACC), TITECH (Japan), and Xyratex."
Sneak Preview of oSC14 Sessions
The openSUSE conference will take place April 24-28, in Dubrovnik, Croatia. The conference organizers present a sneak preview of some of the approved sessions. The call for proposals is still open for the business track. "For end users, the openSUSE Conference features a series of interesting talks. For starters, ownCloud fans will enjoy the talk by Jos Poortvliet about ownCloud at home and the cloud features in more talks with sessions on databases (MySQL) and groupware tools. Desktop users will appreciate an update on what is going on in and to be expected from the KDE community (also Jos Poortvliet), the Jolla phone by Michal Hrušecký and more is coming."
Events: March 13, 2014 to May 12, 2014
The following event listing is taken from the LWN.net Calendar.
| Date(s) | Event | Location |
|---|---|---|
| March 15 March 16 |
Chemnitz Linux Days 2014 | Chemnitz, Germany |
| March 15 March 16 |
Women MiniDebConf Barcelona 2014 | Barcelona, Spain |
| March 18 March 20 |
FLOSS UK 'DEVOPS' | Brighton, England, UK |
| March 20 | Nordic PostgreSQL Day 2014 | Stockholm, Sweden |
| March 21 | Bacula Users & Partners Conference | Berlin, Germany |
| March 22 March 23 |
LibrePlanet 2014 | Cambridge, MA, USA |
| March 22 | Linux Info Tag | Augsburg, Germany |
| March 24 March 25 |
Linux Storage Filesystem & MM Summit | Napa Valley, CA, USA |
| March 24 | Free Software Foundation's seminar on GPL Enforcement and Legal Ethics | Boston, MA, USA |
| March 26 March 28 |
Collaboration Summit | Napa Valley, CA, USA |
| March 26 March 28 |
16. Deutscher Perl-Workshop 2014 | Hannover, Germany |
| March 29 | Hong Kong Open Source Conference 2014 | Hong Kong, Hong Kong |
| March 31 April 4 |
FreeDesktop Summit | Nuremberg, Germany |
| April 2 April 4 |
Networked Systems Design and Implementation | Seattle, WA, USA |
| April 2 April 5 |
Libre Graphics Meeting 2014 | Leipzig, Germany |
| April 3 | Open Source, Open Standards | London, UK |
| April 7 April 8 |
4th European LLVM Conference 2014 | Edinburgh, Scotland, UK |
| April 7 April 9 |
ApacheCon 2014 | Denver, CO, USA |
| April 8 April 10 |
Open Source Data Center Conference | Berlin, Germany |
| April 8 April 10 |
Lustre User Group Conference | Miami, FL, USA |
| April 11 April 13 |
PyCon 2014 | Montreal, Canada |
| April 11 | Puppet Camp Berlin | Berlin, Germany |
| April 12 April 13 |
State of the Map US 2014 | Washington, DC, USA |
| April 14 April 17 |
Red Hat Summit | San Francisco, CA, USA |
| April 25 April 28 |
openSUSE Conference 2014 | Dubrovnik, Croatia |
| April 26 April 27 |
LinuxFest Northwest 2014 | Bellingham, WA, USA |
| April 29 May 1 |
Embedded Linux Conference | San Jose, CA, USA |
| April 29 May 1 |
Android Builders Summit | San Jose, CA, USA |
| May 1 May 4 |
Linux Audio Conference 2014 | Karlsruhe, Germany |
| May 2 May 3 |
LOPSA-EAST 2014 | New Brunswick, NJ, USA |
| May 8 May 10 |
LinuxTag | Berlin, Germany |
If your event does not appear here, please tell us about it.
Page editor: Rebecca Sobol