The Grumpy Editor's guide to surviving the systemd debate

The systemd debate appears to be one of those gifts that keeps on giving; whenever it seems that the issues are approaching resolution and things might calm down, a new flame war starts up. Most recently, this dispute is threatening to tear apart the Debian community. Your editor would like to suggest that such an outcome would be far too high a cost to pay for a disagreement over a system initialization and service management system.

In the end, it comes down to this: it just is not that important. It is just a system initialization utility. There is plenty of experience by now to show that Linux systems still work when systemd is installed; indeed, they are still even recognizable as Linux systems. The distributions that have moved to systemd have not been destroyed or rendered less useful; many would argue that they have been improved by the change. But even if systemd turns out to be a wrong turn in the end, the current level of anxiety and heat is not called for.

Some of us will remember the multi-year dispute over the devfs filesystem, which was perhaps the first serious attempt to deal with the failure of static /dev directories to deal with contemporary hardware. After numerous heated discussions, some of which made the systemd debates look calm indeed, devfs was merged into the 2.3.46 kernel. Some distributions adopted it fully, others held back. Over time, devfs turned out to not be the best of ideas, but, by then, it had worked its way deeply into the kernel. The patch set that removed devfs required 22 individual patches touching over 200 files in the kernel tree; we'll never know how much work was required to wean user space away from devfs. The removal took some time and effort, but, since we had all the code and sufficient will, we achieved it.

If systemd turns out to be an equally bad idea, it can be removed in the same manner. It is entirely possible that we'll eventually look at it the way we look at devfs — as an experiment in the wrong way to solve a problem. Or perhaps it will be seen more like sendmail: the best solution available at the time, but one that few of us wish to deal with now. However things turn out, if it becomes clear that there is a better solution than systemd available, we will be able to move to it.

Of course, it could also happen that systemd works out well and becomes just another part of our software mix.

With this in mind, your editor would like to venture forth with some humble suggestions for the people involved in the systemd debate. Starting with the systemd developers themselves: things are going for now, systemd has been adopted by most of the major distributions out there. Please try to be gracious in your success and to take into account the concerns felt by those who are not fully on board with your vision. Working harder to accommodate those who want a more loosely coupled plumbing layer — or, even better, encouraging them to join your community to make that looser coupling work better — would be a good thing. Avoiding "not our problem" responses when things break would be helpful.

And, it should be said, postings like this can easily be viewed as trolling. It should be possible to celebrate one's successes without being snide.

For those who are not systemd fans, it would be good to get away from the siege mentality that many of you seem to have adopted. This is not the final battle to save Linux from the depredations of the evil systemd empire. Nobody is forcing you to put systemd on your computers; they are simply offering you distributions using systemd that you are free to not install. This is not a conspiracy by any group to destroy Linux as we know it.

It is a group of developers with a vision for where they would like to improve Linux going forward. Linux has always moved quickly; these developers are simply continuing that tradition. The difference, arguably, is that we are long past the days of trying to catch up with other systems and are, instead, exploring new paths with little to guide us. It is natural that there will be more disagreement in this phase, and more mistakes too. Perhaps systemd is a mistake, but, if it is, we'll survive it, just like we have overcome our mistakes in the past.

It would be better if we all could assume that the systemd developers are acting in good faith to try to make a better Linux. If some of their ideas seem misguided to you, there are (at least) a couple of useful ways to respond. One is to join their community and work to push things in new directions; that is how free software works, and the systemd community seems to be reasonably inclusive. The alternative is to show a different solution that works better. Note that talking about such things tends not to impress people in this community; working code is the way to demonstrate a better idea.

For the trolls out there who seem to delight in trying to stir up systemd flame wars, we could ask you to go away but we know that's a futile request. Meanwhile, the rest of the community has, for the most part, been doing an admirable job of ignoring the inflammatory emails, comments, "songs," and more. Keep up the good work, please.

There will come a day when we will wonder what all of the fuss was about. Systemd will either be established and successful, or it will have been replaced by something better. Either way, we will be unable to comprehend why we subjected ourselves to so much stress over an init system; it couldn't possibly be as important as the battle over (say) choice in C libraries or Wayland compositors that we might be fighting at the time. If we can all relax a bit, we'll figure this one out without tearing our communities apart. We are, after all, still working on the same goal: producing the best free operating system we can. We'll never all agree on the best tactics to get to that goal, but we don't have to; it all tends to work out in the end.

Comments (330 posted)

GNOME versus Groupon

While the patent system isn't exactly popular among free-software developers, the privileges accorded by trademark law remain valuable for some free-software projects. For example, Mozilla has benefited from being able to control its brands. There remains a conflict, though, between the redistribution and modification rights that free software accords its users, and the restrictions imposed by a trademark on that software, with the dispute between Debian and Mozilla (that led Debian to rebrand Firefox as Iceweasel) as a classic example. A recent (and apparently resolved) disagreement between the GNOME Foundation and the daily-deals web site Groupon reflects how valuable trademarks can be in the open-source world, as the conflict illustrated the goodwill in the marketplace that the "GNOME" trademark enjoys.

Last May, Groupon issued a press release announcing "Gnome, a new tablet-based platform that will provide sophisticated tools to local merchants to run their businesses more effectively and understand their customers better". Groupon decided to trademark its use of the word "Gnome", variants like "G-NOME", and the phrase "GNOME BY GROUPON" in association with goods and services involving, among other uses, "computer hardware and software for processing point of sale transactions".

But, back in 2006, the GNOME Foundation had already registered a trademark on the word "GNOME" used in association with goods and services involving "downloadable computer software", "computer software development", and other related products. According to the GNOME Foundation, on a call-for-help page it posted online on November 10, efforts to communicate with Groupon about its concerns over the GNOME brand identity were made soon after Groupon's product announcement. The Foundation alleged that these talks broke down and that Groupon continued to seek the registration of additional trademarks.

The Foundation faced a deadline of December 3 to file legal paperwork starting formal opposition proceedings against some of Groupon's trademark applications. On the call-for-help page, it asked the open-source community to donate $80,000 to fund its battle: "Our counsel has advised us that we will need $80,000 to oppose the registration of the first set of 10 applications. If we are able to defend the mark without spending this amount, we will use the remaining funds to bolster and improve GNOME". At the end of the Foundation's statement, there was a link to PayPal to donate, as well as a note that "GNOME can also accept donations by check, bank/wire transfer, Flattr and Bitcoin". As of the morning of November 12, over $100,000 has been raised.

After the issue became more public, and donations started to roll in, Groupon responded by announcing its stance:

As the outcry continued, Groupon relented. A few hours after the initial announcement, Groupon declared it was no longer interested in registering trademarks on the word "Gnome": "After additional conversations with the open source community and the Gnome Foundation, we have decided to abandon our pending trademark applications for 'Gnome.' We will choose a new name for our product going forward". GNOME and Groupon released a joint statement soon after, announcing that the two organizations will be collaborating to ensure that there will be no trademark issues between them in the future:

An examination of US trademark law shows that it was likely prudent of Groupon to abandon its pursuit of these trademarks. In the US, a trademark is considered to be infringed by a similar trademark if there would be a "likelihood of confusion" caused in the marketplace by the trademarks co-existing. There are a number of different tests for "likelihood of confusion" across the different Circuits of the US court system, with considerable variety between them. An empirical study [PDF] of these tests, done by New York University law professor Barton Beebe, criticized both their lack of harmony and inconsistencies in their application by the courts. The US Supreme Court will likely clear up the confusion within the next year when it makes a ruling in a case that addresses the issue.

Until the Supreme Court makes a decision, we can turn to Beebe's study to see how this case might have played out in the courts. Beebe found that likelihood-of-confusion cases tend to turn on the following factors: the similarity of the marks; the likelihood consumers would believe, incorrectly, that the products branded with the trademarks are sold from the same organization; and bad faith on the part of the defendant. If Groupon hadn't capitulated, it would be reasonable to predict that Groupon would not fare well in a legal battle over trademarks with the GNOME Foundation.

The trademarks use the same word, disposing of the first factor in the Foundation's favor. Groupon's brand presence with the general public in the US is much stronger than GNOME's: if Groupon launched "Gnome"-branded devices and services, it would be more likely than not that an "average" consumer would assume, on first encounter, that the GNOME desktop environment was designed by Groupon. This disposes of the second factor against Groupon. While GNOME might not have been able to show bad faith (the third factor) on the part of Groupon, its strong case for likelihood-of-confusion on the first two factors would probably have been enough for the Foundation to prevail in court.

Nonetheless, this David and Goliath story could have been avoided, if Groupon had acted more ethically. Any search for the use of the word "GNOME" in connection with software would have quickly led to the GNOME project and proof of its trademark rights. Groupon must have been aware of that trademark while it decided to apply for its own marks. The fact that Groupon almost immediately caved in after the GNOME Foundation went public with the conflict is strong evidence that either Groupon did not have a case, or it was unwilling to bear the sharp criticism from the open-source community. If Groupon had acted honorably in its discussions with GNOME from the beginning, it would seem that this could have all been resolved quietly—or avoided altogether.

GNOME's call-for-help page remains online, with an update at the top indicating that the matter has been settled. This hasn't deterred GNOME's supporters from continuing to donate to the Foundation through that page, however. It remains to be seen if Groupon will indeed follow through on its commitment to rename the offending goods and services, but it would seem difficult for the company to back out now. Nevertheless, open-source enthusiasts and GNOME fans can breathe a sigh of relief: a potential threat to the name and brand of a major open-source project has been defused.

Comments (5 posted)

High-DPI displays and Linux

Your editor's laptop recently started to fall apart, to the point that some sort of action was required. Duct tape was considered as a solution; it would show that LWN is making careful use of its subscription revenues, but would also convey an impression that is, arguably, not 100% professional. The alternative was to get a new laptop; that costs money, but the idea turned out to be awfully shiny and distracting. So your editor ended up with a ThinkPad X1 Carbon. This machine has a number of nice features and a few weird quirks, but one of the more interesting options it came with is a high-DPI, 2560x1440 display. That was an ideal opportunity to investigate the state of Linux support of high-DPI screens; it turns out that, while quite a bit of progress has been made, this problem has not yet been fully solved.

There are certainly a lot of advantages to high-DPI displays (generally considered to be those with at least 200 pixels per linear inch). Text is rendered in an especially clear and readable fashion. Photos from digital cameras can be displayed at something resembling their native resolution. And, for those of us with sufficiently sharp eyes, high-DPI allows more work to be crammed onto a screen, increasing productivity. But, if the desktop environment and associated applications are not aware that a high-DPI screen is in use, the result can be unreadable text and unusable, microscopic screen elements.

High-DPI display support

So how have the Linux desktop environments responded to high-DPI displays? There appear to be two different basic approaches. The one used by GNOME and some web browsers is to hide the display resolution behind pixel scaling; in essence, the environment lies to applications, then turns each application pixel into a 2x2 array of screen pixels. It's a relatively quick solution, but, as we'll see, it leads to some unfortunate behaviors.

The alternative is to have applications deal with display-independent units (points, centimeters, inches, whatever) and hide the scaling required. KDE appears to have taken this approach. It seems to work well enough as long as applications play along, but here, too, the problem is not entirely solved.

Your editor's experiments have mostly been with GNOME, so this article will have an unavoidable bias in that direction. In GNOME, pixel scaling is managed with the inevitable gsettings registry value; to turn on 2-to-1 scaling, a command like this can be used:

    gsettings set org.gnome.desktop.interface scaling-factor 2

Setting the scaling factor back to one turns off scaling. Scaling should be enabled by default if GNOME detects a high-DPI display; otherwise a command like the above can be used. This value can also be adjusted in gnome-tweak-tool. Should you run into bugs or weirdness (like only a quarter of the screen being used), setting the scaling factor to one then back to two will often straighten them out; your editor has ended up creating a fixscaling script for just this eventuality.

Firefox implements a similar sort of pixel scaling, but it doesn't seem to figure out when to use it on its own. The solution is to go into about:config and set the layout.css.devPixelsPerPx knob to a value like two. Alternatively, one can make things work by configuring the use of 30-point fonts, but that will make a mess of a lot of web sites.

In the KDE world, there seem to be a couple of parameters to set if the desktop doesn't figure things out on its own. The "application appearance" dialog has an option to set the dots-per-inch resolution used for font rendering; setting that to a value close to your screen's pixel density (or lower; 150 seems to be a recommended value) will make text readable. The size of icons is also nicely configurable in KDE; its use of SVG icons helps to make this option work well. Obviously, one will want to make icons larger on high-DPI displays.

Difficulties

So it seems that some support for high-DPI displays is in place for Linux. But there are still a lot of nagging little details — some perhaps not so little — that pop up when using a system with such a display.

For example, what happens when one plugs an external monitor (or projector) into such a system? That display almost certainly is not also high-DPI. The techniques used to make the high-DPI screen will have the effect of turning the lower-DPI screen into an extra-low-DPI screen. In your editor's case, this problem has made the use of external monitors on the new laptop nearly impossible. The word is that this behavior is the results of limitations built deeply into the X Window System which, after all, was developed when 75dpi was a high-DPI screen. Things will be much better under Wayland, we are told, but there doesn't appear to be a lot of relief in store before then.

Pixel scaling can also cause a lot of confusion with applications that are not prepared for it. Here is an old version of gthumb (the version shipped with Fedora 20) with scaling in effect:

With scaling turned off, the result is rather more satisfying:

(It should be noted that gthumb 3.14.x handles high-DPI displays in a much better manner. Your editor will reserve comment on some of the other user interface changes made during that time, though.)

Over time your editor has found a few applications that do better with pixel scaling turned off. Happily, there is an easy way to do that on a per-application basis: simply set the GDK_SCALE environment variable to the desired scaling value.

Another problem with pixel scaling is that it can cause even the high-DPI screen to appear to have a low pixel density. As a result, for example, gnome-terminal believes it's running with 7-point fonts, something that would normally be illegibly small. The sizing is confusing, but it also makes it hard to get that perfect size: seven points is too large, but a 6-point font, being quite a bit smaller, is illegible. All of this is somewhat unfortunate, since points are supposed to be real-world physical dimensions, but that connection has been lost for now.

There is no shortage of web sites that come through badly on high-DPI displays — including this one in many cases. User-interface elements can be so small as to be unusable, and images are reduced down to postage stamps. Xkcd becomes unreadable in such a setting. Turning on pixel scaling in Firefox makes a lot of these problems go away at the cost of losing a fair amount of screen space to the expanded icon bars at the top — and to larger advertisements. The Chromium browser, surprisingly, appears to have no high-DPI awareness at all at the moment; the sad result is that even Google sites can look terrible there.

In general, any web site that is trying to manage its page layout using pixel sizes is going to run into trouble on high-DPI displays. Fixing that problem will take a long time, and involves some interesting tradeoffs. The tiny-image problem can be fixed by shipping higher-resolution images and getting the browser to scale them to the desired size, but that can increase bandwidth consumption considerably. The alternative — sensing the screen resolution and shipping appropriately sized images — is a lot more work to support.

Some closing thoughts

Naturally, the new machine came with Windows preinstalled, so your editor had the chance to see how that system copes with the high-DPI display. The situation there is perhaps a bit better, but Windows has not fully solved the problem either. The basic system user interface works well, but web pages can run into trouble there, and some applications, even popular, high-profile applications, have not yet made the adjustment.

Making applications work properly in an environment where displays can have widely varying densities — even on the same system — is not an easy problem to solve. The good news is that we are much of the way there; the bad news is that there is still a lot to be done, and, in some cases, a change of approach may be needed. In particular, approaches based on pixel scaling have all the appearance of being a short-term kludge that gets things working while a real solution is being worked on.

That real solution, it seems, almost has to involve divorcing applications from the individual pixels on the screen. Once applications are working in pixel-independent units almost all of the time (there may need to be exceptions for image editing, video playback, etc.), the underlying environment can work on rendering in a visually similar way on all possible screens.

Until then, it is good to see that developers are working on making high-DPI screens work better with Linux. A fast rate of progress is arguably not surprising; after all, desktop developers hardly seem like a crowd that would be resistant to the allure of a beautiful new screen. So we can probably count on those developers to fix up the remaining problems in relatively short order.

Comments (83 posted)

The security state of KVM

One of the benefits of virtualization is security; applications running in separate virtual machines are isolated from each other and, ideally, it is very hard for a compromised guest to damage other virtual machines running on the same host. The hypervisor itself is the place where most attacks on a virtualization system will be aimed. At the 2014 KVM Forum, Andrew Honig presented his analysis of which parts of KVM are more likely to have problems, and proposed ways to limit the attack surface.

An insecure hypervisor does not provide much security to the virtual machines it hosts. Luckily, hypervisors are typically small pieces of software, and a smaller size means a reduced attack surface and a higher feasibility of auditing the code. In the case of KVM, the hypervisor runs in the same address space as the rest of the Linux kernel, including device drivers and the network stack, but only a small amount of code deals with untrusted input from the virtual machine.

Therefore, the Linux kernel is substantially insulated from possible malicious behavior of the virtual machine. Device drivers in the virtual machine talk to a user-space process (typically QEMU), and this process talks to the kernel through the regular system call interface or through special devices such as /dev/tap. QEMU is exposed to all the evil that could come from a malicious virtual machine, but only limited and low-level interfaces can be used to attack it. This makes it hard to use QEMU as a vector to exploit kernel vulnerabilities in the host. And, since QEMU is a user-space program, Linux Security Modules (LSMs) such as SELinux or AppArmor can be used to substantially mitigate the effect of arbitrary code execution if QEMU itself is subverted.

This makes the hypervisor much more interesting to attack than QEMU is. So there was a great deal of interest in Honig's talk, "Security Hardening of KVM", (slides [PDF], video [YouTube]) at the KVM Forum, which was held in Düsseldorf, Germany in October. Honig has been working on hypervisor security for about ten years. He used to try to break VMware, and found six CVEs, but his attention has shifted to KVM since he switched employers. He now works at Google, where his team takes care of securing Google Compute Engine (GCE). This is a cloud platform that uses KVM as the hypervisor. Interestingly, the user-space part of GCE is not QEMU; Google wrote its own.

So far, the team has found nine vulnerabilities in KVM. That is not all that many compared to the effort that he and his team is putting into breaking it. In Honig's words, few other parts of Linux have probably had as many "engineer-hours per line of code" spent looking for security problems. Forty thousand lines of C code can certainly be expected to have bugs.

Vulnerability types

What kind of vulnerabilities can you encounter? Privilege escalation or denial of service (DoS) in the host can happen of course, since hypervisors expose a relatively rich ioctl() API to user space; this kind of vulnerability is not really specific to hypervisors. It is slightly more interesting to have a bug that lets an unprivileged program running in the guest crash the whole virtual machine. A bug of this kind was fixed recently.

Crashing the host is worse and mostly happens because of null pointer dereferences (with the panic_on_oops=1 setting); and in some rare cases, a hypervisor bug can facilitate privilege escalation for an unprivileged program running within a guest. Which of these is worse? For a cloud provider such as Google, crashing the host is worse; its customers, however, might value the integrity of their virtual machines.

Higher up in the rankings are vulnerabilities that let guests read data from other guests or from the hypervisor. The recently discovered Xen vulnerability, XSA-108, let guests read a few kilobytes of hypervisor memory. Despite being hard to exploit, and despite the existence of worse kinds of hypervisor vulnerabilities, it received considerable press and forced major cloud providers to reboot all of their hosts.

Of course, the worst bugs of all happen when the guest can write to hypervisor memory and, in all likelihood, execute arbitrary code in hypervisor context soon after. Of the fifteen CVEs that Honig mentioned, five were of this kind: two in KVM and three in VMware.

In order to find these bugs, Honig's team resorts to fuzzing and a lot of code review. They have gained some experience and by now they know what and where to look for every time they upgrade GCE to a newer hypervisor.

Most of the problems stem from either race conditions or buffer overflows, and some are downright embarrassing. In one case for KVM, the code used an ASSERT() macro to verify the validity of an index in an array:

    u32 redir_index = (ioapic->ioregsel - 0x10) >> 1;
    u64 redir_content;
    ASSERT(redir_index < IOAPIC_NUM_PINS);
    redir_content = ioapic->redirtbl[redir_index].bits;

Unfortunately, the bounds check is buried inside the ASSERT() call that is compiled out by default. That means the guest can read arbitrary host memory. Or, if you choose to enable it, as is the case for debug builds, an assertion failure will crash the host—pick your poison.

The code above is part of the emulation of the IOAPIC, an interrupt controller device. It turns out that device emulation is the area where Google reported most bugs, but it is not the only one.

Improving KVM security

The main task of the hypervisor is to drive execution of the virtual CPUs. Some actions of the virtual CPUs, such as reads and writes to model specific registers (MSRs) and I/O registers, cannot be done by the processor; the hypervisor will then either emulate the operation itself or ask a user-space process to complete it. MSRs right now are always handled in kernel space, and are one source of bugs. Performance-critical devices such as interrupt controllers and timers are also handled in kernel space; the IOAPIC is not really performance-critical anymore, but it used to be in 2007-2008 operating systems when KVM was being developed.

In order to process loads and stores to I/O registers, KVM includes a small x86 instruction emulator. The emulator actually has a second purpose: it is needed to handle processor states that are not supported by older Intel processors, such as the so-called "big real mode" and hardware task switching. The good news is that this second purpose is becoming obsolete, as newer processors can do almost all of this in hardware. The bad news is that, unlike RISC architectures where only a handful of instructions have to be emulated, x86 has dozens of instructions that can access memory-mapped I/O registers, and KVM has to recognize and execute them all. Thus, the emulator consists of roughly 5,000 lines of code, and has its own share of bugs.

The more these parts can be moved to user space, the more the attack surface can be reduced, Honig said. As mentioned earlier, user space is naturally confined, and it offers a wealth of mitigation techniques that do not apply to the kernel.

As newer processors include more and more virtualization features, Google is targeting fairly new Intel processors only, and high-end ones at that. In particular, the Xeon E5 v2, also known as Ivy Bridge-E, supports big real mode virtualization and can also virtualize large parts of the local APIC inside the processor.

In a perfect world, everything else would then move to user space. In practice, parts of the local APIC support will almost definitely remain in the kernel. For example, inter-processor interrupts (IPIs) are performance-critical and, in general, not virtualized by the CPU. The only accelerated special case is "self-IPIs", that is IPIs sent to the same processor that triggered them. This sounds weird but is used extensively by Windows.

Still, this means the emulator, the legacy i8259 interrupt controller, the legacy i8254 programmable timer, and the almost-legacy IOAPIC would no longer be part of the hypervisor's attack surface. Most MSR emulation could also move to user space. Honig stated a fairly ambitious goal: to reduce the attack surface by 50% (measured in lines of code and "number of pages of the Intel manual" emulated in the kernel) with at most 0.1% performance impact on macro-benchmarks.

The team's plan has been to start with everything in user space, and re-enable kernel acceleration as much as needed to satisfy their goal. This makes sense for a research project, but it is backwards compared to how this maintainer would like to see the work pushed upstream. As far as I am concerned, in fact, it would be preferable to receive many small series, each one moving a piece of KVM out of the kernel. Also, since Google has not been using either QEMU or kvmtool for the user-space part of the work, the team also has to develop patches for one of them before its improvements can be accepted upstream.

That said, this kind of hurdle should probably be expected, and it did not make the presentation any less interesting. Compared to containers, one of the strengths in virtualization is (or should be) the smaller attack surface. It is important that hypervisors keep up with the promises, and Honig's ideas are definitely going in the right direction.

Comments (6 posted)

Security quotes of the week

— "Ms. Smith" visits a site that aggregates the feeds from security cameras with default credentials

— Dragos Ruiu on a presentation about BadUSB

— Jacob Hoffman-Andrews of the EFF

Comments (16 posted)

Ubuntu, ownCloud, and a hidden dark side of Linux software repositories (PC World)

Here's a PC World article on the old, insecure version of ownCloud shipped in Ubuntu 14.04 — and the difficulties in getting it updated or removed.

The writing is a little breathless, but there is a valid issue here; the software found in the more remote corners of distribution repositories may not be particularly well maintained.

Comments (68 posted)

cinder: information disclosure

Package(s):	cinder	CVE #(s):	CVE-2014-7230
Created:	November 12, 2014	Updated:	December 3, 2014
Description:	From the CVE entry: The processutils.execute function in OpenStack oslo-incubator, Cinder, Nova, and Trove before 2013.2.4 and 2014.1 before 2014.1.3 allows local users to obtain passwords from commands that cause a ProcessExecutionError by reading the log.
Alerts:	Red Hat RHSA-2014:1939-01 openstack-trove 2014-12-02 Ubuntu USN-2407-1 nova 2014-11-11 Ubuntu USN-2405-1 cinder 2014-11-11

Comments (none posted)

curl: information leak

Package(s):	curl	CVE #(s):	CVE-2014-3707
Created:	November 7, 2014	Updated:	January 5, 2015
Description:	From the Debian advisory: Symeon Paraschoudis discovered that the curl_easy_duphandle() function in cURL, an URL transfer library, has a bug that can lead to libcurl eventually sending off sensitive data that was not intended for sending, while performing a HTTP POST operation. This bug requires CURLOPT_COPYPOSTFIELDS and curl_easy_duphandle() to be used in that order, and then the duplicate handle must be used to perform the HTTP POST. The curl command line tool is not affected by this problem as it does not use this sequence.
Alerts:	Scientific Linux SLSA-2015:2159-6 curl 2015-12-21 Oracle ELSA-2015-2159 curl 2015-11-23 Red Hat RHSA-2015:2159-06 curl 2015-11-19 Scientific Linux SLSA-2015:1254-2 curl 2015-08-03 Oracle ELSA-2015-1254 curl 2015-07-29 Red Hat RHSA-2015:1254-02 curl 2015-07-22 Mandriva MDVSA-2015:098 curl 2015-03-28 openSUSE openSUSE-SU-2015:0248-1 curl 2015-02-10 Fedora FEDORA-2014-16690 curl 2015-01-03 Fedora FEDORA-2014-17601 mingw-curl 2015-01-02 Fedora FEDORA-2014-16538 curl 2014-12-13 Fedora FEDORA-2014-16605 curl 2014-12-15 Fedora FEDORA-2014-17596 mingw-curl 2015-01-02 Fedora FEDORA-2014-15706 curl 2014-12-01 Mandriva MDVSA-2014:213 curl 2014-11-18 Mageia MGASA-2014-0444 curl 2014-11-14 Ubuntu USN-2399-1 curl 2014-11-10 Fedora FEDORA-2014-14354 curl 2014-11-10 Debian DSA-3069-1 curl 2014-11-07

Comments (none posted)

deluge: deluge-web is vulnerable to POODLE

Package(s):	deluge	CVE #(s):
Created:	November 12, 2014	Updated:	November 12, 2014
Description:	From the Red Hat bugzilla: The web plugin creates an SSLv3 socket. The latest version of deluge, 1.3.10, updates the web plugin to use TLSv1.
Alerts:	Fedora FEDORA-2014-12991 deluge 2014-11-12

Comments (none posted)

gnutls28: code execution

Package(s):	gnutls28	CVE #(s):	CVE-2014-8564
Created:	November 11, 2014	Updated:	November 19, 2014
Description:	From the Ubuntu advisory: Sean Burford discovered that GnuTLS incorrectly handled printing certain elliptic curve parameters. A malicious remote server or client could use this issue to cause GnuTLS to crash, resulting in a denial of service, or possibly execute arbitrary code.
Alerts:	Mandriva MDVSA-2015:072 gnutls 2015-03-27 openSUSE openSUSE-SU-2014:1472-1 gnutls 2014-11-21 Mandriva MDVSA-2014:215 gnutls 2014-11-19 Mageia MGASA-2014-0458 gnutls 2014-11-15 Fedora FEDORA-2014-14760 gnutls 2014-11-13 Scientific Linux SLSA-2014:1846-1 gnutls 2014-11-12 Oracle ELSA-2014-1846 gnutls 2014-11-12 CentOS CESA-2014:1846 gnutls 2014-11-12 Red Hat RHSA-2014:1846-01 gnutls 2014-11-12 Ubuntu USN-2403-1 gnutls28 2014-11-11

Comments (none posted)

ImageMagick: multiple vulnerabilities

Package(s):	ImageMagick	CVE #(s):	CVE-2014-8354 CVE-2014-8355 CVE-2014-8562
Created:	November 12, 2014	Updated:	April 13, 2015
Description:	From the openSUSE advisory: - Out-of-bounds memory access in PCX parser (CVE-2014-8355). - Out-of-bounds memory access in resize code (CVE-2014-8354). - Out-of-bounds memory error in DCM decode (CVE-2014-8562).
Alerts:	Ubuntu USN-3131-1 imagemagick 2016-11-21 Fedora FEDORA-2015-3612 ImageMagick 2015-04-13 Debian-LTS DLA-242-1 imagemagick 2015-06-11 Mandriva MDVSA-2015:105 imagemagick 2015-03-29 Mandriva MDVSA-2014:226 imagemagick 2014-11-25 Mageia MGASA-2014-0484 graphicsmagick 2014-11-25 Mageia MGASA-2014-0482 imagemagick 2014-11-22 openSUSE openSUSE-SU-2014:1396-1 ImageMagick 2014-11-12

Comments (none posted)

kde-workspace: privilege escalation

Package(s):	kde-workspace	CVE #(s):	CVE-2014-8651
Created:	November 11, 2014	Updated:	December 31, 2015
Description:	From the Ubuntu advisory: David Edmundson discovered that the KDE Clock KCM policykit helper did not properly guard against untrusted input. Under certain circumstances, a process running under the user's session could exploit this to run programs as the administrator. See also this KDE advisory.
Alerts:	Gentoo 201512-12 systemsettings 2015-12-30 Mageia MGASA-2014-0480 kdebase4-workspace 2014-11-21 Fedora FEDORA-2014-14865 kde-workspace 2014-11-17 Fedora FEDORA-2014-14813 kde-workspace 2014-11-15 Mageia MGASA-2014-0445 kdebase4-workspace 2014-11-14 Ubuntu USN-2402-1 kde-workspace 2014-11-10

Comments (none posted)

libreoffice: code execution

Package(s):	libreoffice	CVE #(s):	CVE-2014-3693
Created:	November 6, 2014	Updated:	December 4, 2014
Description:	From the Ubuntu advisory: It was discovered that LibreOffice incorrectly handled the Impress remote control port. An attacker could possibly use this issue to cause Impress to crash, resulting in a denial of service, or possibly execute arbitrary code.
Alerts:	Gentoo 201603-05 libreoffice 2016-03-09 openSUSE openSUSE-SU-2016:0588-1 LibreOffice 2016-02-26 Scientific Linux SLSA-2015:0377-1 libreoffice 2015-03-25 Oracle ELSA-2015-0377 libreoffice 2015-03-12 Red Hat RHSA-2015:0377-01 libreoffice 2015-03-05 Mageia MGASA-2014-0505 libreoffice 2014-12-03 openSUSE openSUSE-SU-2014:1443-1 libreoffice 2014-11-18 openSUSE openSUSE-SU-2014:1412-1 libreoffice 2014-11-13 Ubuntu USN-2398-1 libreoffice 2014-11-05

Comments (none posted)

libvirt: information disclosure

Package(s):	libvirt	CVE #(s):	CVE-2014-7823
Created:	November 11, 2014	Updated:	January 6, 2015
Description:	From the Ubuntu advisory: Eric Blake discovered that libvirt incorrectly handled permissions when processing the qemuDomainFormatXML command. An attacker with read-only privileges could possibly use this to gain access to certain information from the domain xml file.
Alerts:	Mandriva MDVSA-2015:115 libvirt 2015-03-29 Oracle ELSA-2015-0323 libvirt 2015-03-12 Oracle ELSA-2015-0008 libvirt 2015-01-05 Red Hat RHSA-2015:0008-01 libvirt 2015-01-05 Scientific Linux SLSA-2015:0008-1 libvirt 2015-01-06 CentOS CESA-2015:0008 libvirt 2015-01-05 Gentoo 201412-04 libvirt 2014-12-09 Mandriva MDVSA-2014:222 libvirt 2014-11-21 Fedora FEDORA-2014-15228 libvirt 2014-11-22 Oracle ELSA-2014-1873 libvirt 2014-11-20 openSUSE openSUSE-SU-2014:1471-1 libvirt 2014-11-21 Mageia MGASA-2014-0470 libvirt 2014-11-21 Scientific Linux SLSA-2014:1873-1 libvirt 2014-11-18 CentOS CESA-2014:1873 libvirt 2014-11-18 Red Hat RHSA-2014:1873-01 libvirt 2014-11-18 Ubuntu USN-2404-1 libvirt 2014-11-11

Comments (none posted)

php: code execution

Package(s):	php	CVE #(s):	CVE-2014-8626
Created:	November 7, 2014	Updated:	November 12, 2014
Description:	From the Red Hat advisory: A stack-based buffer overflow flaw was found in the way the xmlrpc extension parsed dates in the ISO 8601 format. A specially crafted XML-RPC request or response could possibly cause a PHP application to crash or execute arbitrary code with the privileges of the user running that PHP application.
Alerts:	CentOS CESA-2014:1824 php 2014-11-06 Oracle ELSA-2014-1824 php 2014-11-06 Scientific Linux SLSA-2014:1824-1 php 2014-11-06 Red Hat RHSA-2014:1825-01 php 2014-11-06

Comments (none posted)

Pound: HTTP request smuggling

Package(s):	Pound	CVE #(s):	CVE-2005-2090
Created:	November 7, 2014	Updated:	November 12, 2014
Description:	From the CVE entry: Jakarta Tomcat 5.0.19 (Coyote/1.1) and Tomcat 4.1.24 (Coyote/1.0) allows remote attackers to poison the web cache, bypass web application firewall protection, and conduct XSS attacks via an HTTP request with both a "Transfer-Encoding: chunked" header and a Content-Length header, which causes Tomcat to incorrectly handle and forward the body of the request in a way that causes the receiving server to process it as a separate HTTP request, aka "HTTP Request Smuggling."
Alerts:	Fedora FEDORA-2014-13777 Pound 2014-11-12 Fedora FEDORA-2014-13764 Pound 2014-11-07

Comments (none posted)

qemu: multiple vulnerabilities

Package(s):	qemu	CVE #(s):	CVE-2014-3689 CVE-2014-7815
Created:	November 7, 2014	Updated:	November 12, 2014
Description:	From the Debian advisory: CVE-2014-3689 – The Advanced Threat Research team at Intel Security reported that guest provided parameter were insufficiently validated in rectangle functions in the vmware-vga driver. A privileged guest user could use this flaw to write into qemu address space on the host, potentially escalating their privileges to those of the qemu host process. CVE-2014-7815 – James Spadaro of Cisco reported insufficiently sanitized bits_per_pixel from the client in the QEMU VNC display driver. An attacker having access to the guest's VNC console could use this flaw to crash the guest.
Alerts:	SUSE SUSE-SU-2016:1785-1 kvm 2016-07-11 SUSE SUSE-SU-2016:1745-1 xen 2016-07-06 SUSE SUSE-SU-2016:1698-1 kvm 2016-06-28 SUSE SUSE-SU-2016:1560-1 qemu 2016-06-13 SUSE SUSE-SU-2016:1445-1 Xen 2016-05-30 SUSE SUSE-SU-2016:1318-1 xen 2016-05-17 SUSE SUSE-SU-2016:1154-1 xen 2016-04-26 openSUSE openSUSE-SU-2016:0995-1 xen 2016-04-08 SUSE SUSE-SU-2016:0955-1 xen 2016-04-05 openSUSE openSUSE-SU-2016:0914-1 xen 2016-03-30 SUSE SUSE-SU-2016:0873-1 xen 2016-03-24 SUSE SUSE-SU-2015:1782-1 qemu 2015-10-20 Scientific Linux SLSA-2015:0349-1 qemu-kvm 2015-03-25 Mandriva MDVSA-2015:061 qemu 2015-03-13 Oracle ELSA-2015-0349 qemu-kvm 2015-03-12 Red Hat RHSA-2015:0349-01 qemu-kvm 2015-03-05 Gentoo 201412-37 qemu 2014-12-24 Gentoo 201412-01 qemu 2014-12-08 Mandriva MDVSA-2014:220 qemu 2014-11-21 Mageia MGASA-2014-0467 qemu 2014-11-21 Ubuntu USN-2409-1 qemu, qemu-kvm 2014-11-13 Fedora FEDORA-2014-14033 qemu 2014-11-10 Debian DSA-3067-1 qemu-kvm 2014-11-06 Debian DSA-3066-1 qemu 2014-11-06

Comments (none posted)

sssd: restriction bypass

Package(s):	sssd	CVE #(s):	CVE-2014-0249
Created:	November 12, 2014	Updated:	October 27, 2016
Description:	From the CVE entry: The System Security Services Daemon (SSSD) 1.11.6 does not properly identify group membership when a non-POSIX group is in a group membership chain, which allows local users to bypass access restrictions via unspecified vectors.
Alerts:	openSUSE openSUSE-SU-2016:2651-1 sssd 2016-10-26 openSUSE openSUSE-SU-2014:1407-1 sssd 2014-11-12

Comments (none posted)

tnftp: command execution

Package(s):	tnftp	CVE #(s):	CVE-2014-8517
Created:	November 11, 2014	Updated:	November 15, 2016
Description:	From the Red Hat bug report: It was reported that tnftp, an FTP client from NetBSD, could be forced to run arbitrary commands if an output file is not specified.
Alerts:	Gentoo 201611-05 tnftp 2016-11-15 Fedora FEDORA-2014-14113 tnftp 2014-11-12 openSUSE openSUSE-SU-2014:1383-1 tnftp 2014-11-10

Comments (none posted)

wss4j: authentication spoofing

Package(s):	wss4j	CVE #(s):	CVE-2014-3623
Created:	November 7, 2014	Updated:	December 29, 2014
Description:	From the CVE entry: Apache WSS4J before 1.6.17 and 2.x before 2.0.2, as used in Apache CXF 2.7.x before 2.7.13 and 3.0.x before 3.0.2, when using TransportBinding, does properly enforce the SAML SubjectConfirmation method security semantics, which allows remote attackers to conduct spoofing attacks via unspecified vectors.
Alerts:	Mageia MGASA-2014-0552 wss4j 2014-12-26 Fedora FEDORA-2014-13720 wss4j 2014-11-07

Comments (none posted)

xml-security: denial of service

Package(s):	xml-security	CVE #(s):	CVE-2013-4517
Created:	November 7, 2014	Updated:	December 31, 2014
Description:	From the CVE entry: Apache Santuario XML Security for Java before 1.5.6, when applying Transforms, allows remote attackers to cause a denial of service (memory consumption) via crafted Document Type Definitions (DTDs), related to signatures.
Alerts:	Mageia MGASA-2014-0558 xml-security 2014-12-31 Fedora FEDORA-2014-13879 xml-security 2014-11-07

Comments (none posted)

zarafa: multiple vulnerabilities

Package(s):	zarafa	CVE #(s):
Created:	November 10, 2014	Updated:	November 12, 2014
Description:	From the Fedora advisory: This R1 release of the 7.1.11 final release addresses the WebAccess install problem on RPM-based systems and resolves the dependencies problems under Ubuntu 14.04. Downstream changes Removed bundled PHP PEAR files/libraries Added patch to allow mitigation of SSLv3/POODLE vulnerability Added patch to implement ECDHE support (depending on OpenSSL) Added patch to allow plaintext authentication from 127.0.0.1
Alerts:	Fedora FEDORA-2014-13017 zarafa 2014-11-10 Fedora FEDORA-2014-12989 zarafa 2014-11-10

Comments (none posted)

zeromq: man-in-the-middle attack

Package(s):	zeromq	CVE #(s):	CVE-2014-7202 CVE-2014-7203
Created:	November 11, 2014	Updated:	November 25, 2014
Description:	From the CVE entries: stream_engine.cpp in libzmq (aka ZeroMQ/C++)) 4.0.5 before 4.0.5 allows man-in-the-middle attackers to conduct downgrade attacks via a crafted connection request. (CVE-2014-7202) libzmq (aka ZeroMQ/C++) 4.0.x before 4.0.5 does not ensure that nonces are unique, which allows man-in-the-middle attackers to conduct replay attacks via unspecified vectors. (CVE-2014-7203)
Alerts:	openSUSE openSUSE-SU-2014:1493-1 zeromq 2014-11-25 openSUSE openSUSE-SU-2014:1381-1 zeromq 2014-11-10

Comments (none posted)

Kernel release status

The current development kernel is 3.18-rc4, released on November 9. "Hey, things are finally calming down. In fact, it looked *really* calm until yesterday, at which point some people clearly realized 'hey, I should push my stuff to Linus so that it makes it into -rc4', and then a third of all changes came in the last day, but despite that, rc4 finally looks like things are falling into place, and we'll get to stabilize this release after all."

Stable updates: no stable kernel updates have been released in the last week. The 3.17.3 (319 patches!), 3.14.24, and 3.10.60 updates are in the review process as of this writing; they can be expected on or after November 14.

Comments (none posted)

An introduction to compound pages

Your editor was digging through a patch set that makes changes involving compound pages when he realized that his understanding of these pages was a bit on the weak side. After some time digging through the source to rectify that situation, a thought surfaced: the world must be full of people wishing they knew more about compound pages. For all of you whose list of desired lifetime accomplishments includes a better understanding of this subject, here is a quick introduction to compound pages in the Linux kernel.

A compound page is simply a grouping of two or more physically contiguous pages into a unit that can, in many ways, be treated as a single, larger page. They are most commonly used to create huge pages, used within hugetlbfs or the transparent huge pages subsystem, but they show up in other contexts as well. Compound pages can serve as anonymous memory or be used as buffers within the kernel; they cannot, however, appear in the page cache, which is only prepared to deal with singleton pages.

Allocating a compound page is a matter of calling a normal memory allocation function like alloc_pages() with the __GFP_COMP allocation flag set and an order of at least one. It is not possible to create an order-zero (single-page) compound page due to the way compound pages are implemented. (The "order" of an allocation is the base-2 logarithm of the number of pages to allocate; zero thus corresponds to a single page, one to two pages, etc.).

Note that a compound page differs from the pages returned from a normal higher-order allocation request. A call like:

    pages = alloc_pages(GFP_KERNEL, 2);  /* no __GFP_COMP */

will return four physically contiguous pages, but they will not be a compound page. The difference is that creating a compound page involves the creation of a fair amount of metadata; much of the time, that metadata is unneeded so the expense of creating it can be avoided.

So what does that metadata look like? Since most of it is stored in the associated page structures, one can assume that it's complicated. Let's start with the page flags. The first (normal) page in a compound page is called the "head page"; it has the PG_head flag set. All other pages are "tail pages"; they are marked with PG_tail. At least, that is the case on systems where page flags are not in short supply — 64-bit systems, in other words. On 32-bit systems, there are no page flags to spare, so a different scheme is used; all pages in a compound page have the PG_compound flag set, and the tail pages have PG_reclaim set as well. The PG_reclaim bit is normally used by the page cache code, but, since compound pages cannot be represented in the page cache, that flag can be reused here.

Code dealing with compound pages need not worry about the different marking conventions, though. No matter which convention is in use, a call to PageCompound() will return a true value if the passed-in page is a compound page. Head and tail pages can be distinguished, should the need arise, with PageHead() and PageTail().

Every tail page has a pointer to the head page stored in the first_page field of struct page. This field occupies the same storage as the private field, the spinlock used when the page holds page table entries, or the slab_cache pointer used when the page is owned by a slab allocator. The compound_head() helper function can be used to find the head page associated with any tail page.

There is a bit of information describing the compound page as a whole: the order (size) of the page, and a destructor used to return the page to the system when it is no longer needed. One might first think to store that information in the head page's struct page, but there is no room for it there. Instead, the order is stored in the lru.prev field in the page structure for the first tail page. While unions are used for many of the overlaid fields in struct page, here the order is simply cast into a pointer type before being stored in a pointer field. Similarly, a pointer to the destructor is stored in the lru.next field of the first tail page's struct page. This extension of compound-page metadata into the second page structure is why compound pages must consist of at least two pages.

Incidentally, there are only two compound page destructors declared in the kernel. By default, free_compound_page() is used; all it does is return the memory to the page allocator. The hugetlbfs subsystem, though, uses free_huge_page() to keep its accounting up to date.

In most cases, compound pages are unnecessary and ordinary allocations can be used; calling code needs to remember how many pages it allocated, but otherwise the metadata that would be stored in a compound page is unneeded. A compound page is indicated, though, whenever it is important to treat the group of pages as a whole even if somebody references a single page within it. Transparent huge pages are a classic example; if user space attempts to change the protections on a portion of a huge page, the entire huge page will need to be located and broken up. Various drivers also use compound pages to ease the management of larger buffers.

And that is pretty much everything that distinguishes a compound page from an ordinary, higher-order allocation. Most developers will not encounter compound pages in their area of the kernel. In cases where it is truly necessary to treat a set of pages as a single unit, though, compound pages may well be part of the solution toolkit.

Comments (3 posted)

Transparent huge page reference counting

While most architectures supported by Linux use a 4KB page size, most of them also are able to work with much larger pages, varying from 2MB to 1GB in size. These "huge pages" offer significant performance advantages for many workloads, the biggest of which is usually the reduction of pressure on the translation lookaside buffer which short-circuits the process of turning a virtual address into a physical address. The kernel's transparent huge pages (THP) feature enables the use of huge pages without the need for any sort of developer or user intervention. THP suffers from some limitations that prevent it from being used as fully as one might like, though; a patch set from Kirill A. Shutemov aims to reduce those limitations, at the cost of making changes to some fairly complex code.

Transparent huge pages

THP works by quietly substituting huge pages into a process's address space when (1) those page are available and (2) it appears that the process would benefit from the change. When Andrea Arcangeli first added this feature to the 2.6.38 kernel, he had a formidable problem to face: there were many places in the kernel's memory-management code that were not prepared to cope with huge pages scattered randomly in a process's address space. Andrea dealt with some of those problems by avoiding them entirely; that is why, for example, page-cache pages (those backed by files on disk) cannot be huge pages in current kernels.

In many other situations, Andrea placed a call to split_huge_page(), a function which breaks a huge page down into its component small pages. Whenever he encountered code that could not cope with a huge page, and that he wasn't able to fix at the time, he put in a split_huge_page() call to simply make the huge page go away. These calls have a clear performance cost, since they undo the work that put the huge page into place to begin with. But they reduced the problem to something more tractable; split_huge_page() can thus be thought of as a crutch similar to the big kernel lock. It is almost never the optimal solution to the problem, but it is a solution that can be made to work now, deferring a hard problem for a later time.

Over time, some of the split_huge_page() calls in the kernel have been replaced with code that can handle huge pages. But they still exist in the page migration code, in the implementation of kernel samepage merging (KSM), in the bad-memory poisoning code, in the implementation of mprotect() and mlock(), and in the swap code, among other places. Some of these cases are unlikely to change; KSM will probably never be able to merge duplicate pages if it cannot split them out of huge pages. Others might seem just as resistant to change; what does it mean to change the protection of, say, half of a huge page with mprotect()? It turns out that this latter case can be optimized, though, as part of a bigger effort to rework and simplify how the management of transparent huge pages and their references counts is done.

PMD- and PTE-level mappings

To understand Kirill's patch set, it is good to remember that huge pages are represented in the kernel as compound pages. Readers who are unfamiliar with how compound pages work may want to take a look at this article for some basic background and terminology.

Kirill's ultimate goal is to enable the use of transparent huge pages with the page cache. Currently, only anonymous pages can be replaced with huge pages, limiting their use to only a fraction of total memory. That is an ambitious goal, and the current patch set does not even try to approach it. Instead, Kirill has worked to simplify the management of transparent huge pages and make them more flexible.

In particular, he has eliminated the hard separation between normal and huge pages in the system. In current kernels, a specific 4KB page can be treated as an individual page, or it can be part of a huge page, but not both. If a huge page must be split into individual pages, it is split completely for all users, the compound page structure is torn down, and the huge page no longer exists. The fundamental change in Kirill's patch set is to allow a huge page to be split in one process's address space, while remaining a huge page in any other address space where it is found.

Time for a quick reminder of how page tables are structured on Linux systems; this diagram was taken from this 2005 LWN article on the subject:

A huge page is represented in a process's page table with a single entry at the page middle directory (PMD) level. Individual pages, instead, have entries at the bottom page-table entry (PTE) level, as shown in the diagram. But there is nothing that says that the same memory must be mapped in the same way in all processes; it is perfectly legitimate for one process to see a 2MB range as a single huge page while another has it mapped as 512 individual PTEs. If this type of different mapping were supported, one process could call mprotect() to change the protections on a portion of a huge page (causing the mapping to be split in that process's address space) while not disturbing the huge page mapping in other processes, which are not affected by the protection change.

In other words, if split_huge_page() could be replaced by a new function, call it split_huge_pmd(), that would only split up a single process's mapping of a huge page, code needing to deal with individual pages could often be accommodated while preserving the benefits of the huge page for other processes. But, as noted above, the kernel currently does not support different mappings of huge pages; all processes must map the memory in the same way. This restriction comes down to how various parameters — reference counts in particular — are represented in huge pages.

Huge-page reference counting

A reference count tracks the number of users an object (such as a page in memory) has, allowing the kernel to determine when the object is free and can be deleted. There are actually two types of reference counts for a normal page. The first, stored in the _count field of struct page, is the total number of references held to the page. The second, kept in _mapcount, is the number of page table entries referring to this page. A page-table mapping is a reference, so every such reference counted in _mapcount is also tracked in _count; the latter should thus always be greater than or equal to the former. Situations where _count can exceed _mapcount include pages mapped for DMA and pages mapped into the kernel's address space with a function like get_user_pages(). Locking a page into memory with mlock() will also increase _count. The relative value of these two counters is important; if _count equals _mapcount, the page can be reclaimed by locating and removing all of the page table entries. But if _count is greater than _mapcount, the page is "pinned" and cannot be reclaimed until the extra references are removed.

The reference count rules change with compound pages, though. In that case, the value of _count is held at zero for all tail pages to avoid confusing other parts of the memory management code. Reference counts are, as a rule, tracked in the head page for the compound page as a whole, but it is still necessary to track references on individual (small) pages within the compound page. Imagine a situation where part of such a page is used for an I/O operation, for example. The trick that is used is to keep that reference in _mapcount instead and to have the various helper functions that access reference counts pick the right field depending on whether a given page is a tail page or not.

This trick works because huge pages are mapped and unmapped as a unit, so there is no need to track the mapping of tail pages separately. If, however, one wants to allow the mapping of individual pages within a huge page, things fall apart, because it will become necessary to track the mappings to those individual pages. So this trick must go; it must be replaced by a scheme that can track both the mappings to the huge page as a whole and the individual pages that make up that huge page.

The first part, tracking mappings to the huge page as a whole, is done with another increasingly familiar (to those who read the article on compound pages) trick: this count is placed into the mapping field of the first tail page in the huge page. This field is normally used to track the file that has been mapped into this page of memory; since huge pages are not used in the page cache, there is no file mapping and this space is available for other uses. The count is an atomic_t, while mapping is a pointer to struct address_space, so yet another cast is used. One might argue that another union field should be added to make the overloading explicit, but that was not done here.

There is still the matter of tracking non-mapping reference counts to tail pages — the reason _mapcount was used in those pages to begin with. This tracking is done so that, should the huge page be split, the individual pages with elevated reference counts can be properly marked. Kirill's approach is to give up on that objective and, in the process, change how non-mapping references to tail pages are tracked. Instead, a call to get_page() on a tail page will increment the reference count on the head page. So the knowledge that a specific tail page is pinned is replaced with the knowledge that some page, somewhere within the compound page, is pinned.

That, naturally, will destroy the ability to mark individual pages as being pinned if the huge page is split. Kirill's answer to that is to simply cause split_huge_page() to fail if any pages within the huge page are pinned. Note that it will not cause split_huge_pmd(), which just splits the mapping for one process, to fail. So most cases that formerly had to call split_huge_page() will still work since the huge page is no longer truly being split. The cases where a split is absolutely necessary will simply fail, but, Kirill says, the code is prepared for that eventuality in all cases.

Removing tail page reference counting enables the removal of a surprising amount of special-case code. It also frees up _mapcount for its original purpose: to track the number of page-table mappings to the page. At that point, it becomes possible for one process to map a huge page as a single page, while another maps it as a set of individual pages.

Kirill claims that the simplification of the code yields performance improvements on their own, though no benchmark results have been posted. Allowing some processes to retain a huge-page mapping when others need a split view should also make things go faster in cases where memory is shared. This feature should make a bigger difference, though, in the future when THP can be used with the page cache, where sharing of pages happens rather more often. That day will not come soon, though; first this set of patches must find its way into the mainline. Given that the work is being done in low-level memory-management code, and given that Kirill thinks there are probably a few surprises (code that doesn't expect an individual page to be a tail page, for example) yet to be found, that probably is not going to happen in the immediate future.

Comments (none posted)

Recent read-mostly research

One of my more unusual hobbies is occasionally reviewing academic papers. It should come as no surprise that I pay special attention to papers related to read-copy update (RCU), which is a category that I am happy to see has been growing significantly over the past few years. This article gives a quick summary of some recent papers in this category.

But first, what is RCU? It is a synchronization mechanism that was added to the Linux kernel in October 2002. RCU is most frequently described as a replacement for reader-writer locking, but has also been used in a number of other ways. It is notable in that RCU readers do not directly synchronize with updaters, which makes RCU read paths extremely fast. It also permits readers to accomplish useful work even when running concurrently with updaters—and vice versa.

Although the earliest known work on mechanisms resembling RCU was carried out by academics (see for example Kung and Lehman's landmark 1980 paper), by the early 1990s, with a few notable exceptions (Ben Gamsa, University of Toronto, et al. [PDF], Robert Olsson, Uppsala University, et al. [PDF], Thomas E. Hart, University of Toronto, et al. [PDF], and Jonathan Appavoo, IBM T.J. Watson Research Center, et al. [PDF]), much of the work in this area was carried out by practitioners. By the early 2000s, the initiative had passed to open-source projects, most notably to the Linux kernel community.

However, there are welcome signs that some in academia are showing interest in RCU; see, for example, Michael L. Scott's textbook that includes a chapter on RCU. One reason that this interest is welcome is that academics, unlike large open-source projects, are unconstrained by the need to develop production-quality code. This allows them to try crazier and more experimental ideas.

Although many of these ideas might seem to have no value outside of academia, the fact is that many of the best yet-undiscovered ideas are protected by wide moats of insanity—with RCU itself being a case in point. Therefore, if you are unwilling to deal with insanity, the odds on your discovering something new decrease, and, of course, those of us who deal with production-quality code must be quite careful with any insanity we encounter. Furthermore, even an otherwise useless idea might inspire a developer to come up with something that is both valuable and useful, or failing that, to avoid a pitfall. Although there are no guarantees, the hope is that increased academic work in the area of read-mostly concurrency will result in new breakthroughs.

Validation

Validation is one important area in which breakthroughs would be quite welcome. Peter Sewell's and Susmit Sarkar's groups at the University of Cambridge have done some important work in validating memory barriers and atomic instructions, as was reported earlier. This work has been extended with greatly increased performance by some of Sewell's and Sarkar's collaborators at University College London (Jade Alglave), INRIA (Luc Maranget), and Queen Mary University of London (Michael Tautschnig), most notably in this paper [PDF], which was described in this LWN article. Researchers at Stony Brook University have produced an RCU-aware data-race detector, described by Abhinav Duggal [PDF] and Justin Seyster [PDF]. Alexey Gotsman of IMDEA, Noam Rinetzky of Tel Aviv University, and Hongseok Yang of the University of Oxford have published a paper [PDF] expressing the formal semantics of RCU in terms of separation logic, and have continued with other aspects of concurrency. With some luck, all of this validation work will eventually result in more and better tools for validating concurrent code.

Using RCU

Phil Howard and Jon Walpole of Portland State University (PSU) have applied RCU to red-black trees [PDF] combined with updates synchronized using software transactional memory. Josh Triplett and Jon Walpole (again of PSU) applied RCU to resizable hash tables [PDF], reported on in two parts: Part 1 and Part 2. (Other RCU-protected resizable hash tables have been created by Herbert Xu and by Mathieu Desnoyers.)

Austin Clements, Frans Kaashoek, and Nickolai Zeldovich of MIT created an RCU-optimized balanced binary tree (Bonsai) [PDF], and applied this tree to the Linux kernel's VM subsystem (Git trees) in order to reduce read-side contention on mmap_sem. This work resulted in order-of-magnitude speedups and scalability up to at least 80 CPUs for a microbenchmark featuring large numbers of minor page faults. This is similar to a patch developed earlier by Peter Zijlstra, and both were limited by the fact that, at the time, filesystem data structures were not safe for RCU readers. Clements et al. avoided this limitation by optimizing the page-fault path for anonymous pages only. More recently, filesystem data structures have been made safe for RCU readers (covered in a 2010 article and another from 2011), so perhaps this work can be implemented for all page types, not just anonymous pages—Peter Zijlstra has, in fact, recently prototyped exactly this. In any case, this use of the Linux kernel itself as a testbed for cutting-edge academic research is a welcome development.

Yandong Mao and Robert Morris of MIT and Eddie Kohler of Harvard University created another RCU-protected tree named Masstree [PDF] that combines ideas from B+ trees and tries. Although this tree is about 2.5x slower than an RCU-protected hash table, it supports operations on key ranges, unlike hash tables. In addition, Masstree supports efficient storage of objects with long shared key prefixes and, furthermore, provides persistence via logging to mass storage.

The paper notes that Masstree's performance rivals that of memcached, even given that Masstree is persistently storing updates and memcached is not. The paper also compares Masstree's performance to the persistent datastores MongoDB, VoltDB, and Redis, reporting significant performance advantages for Masstree, in some cases exceeding two orders of magnitude. Another paper [PDF], by Stephen Tu, Wenting Zheng, Barbara Liskov, and Samuel Madden of MIT and Kohler, applies Masstree to an in-memory database named Silo, achieving 700K transactions per second (42M transactions per minute) on a well-known transaction-processing benchmark. Interestingly enough, Silo guarantees linearizability without incurring the overhead of grace periods while holding locks.

For my part, I have been doing some work enabling complex atomic updates to RCU-protected data structures with minimal copying, which I recently presented [PDF] at the C++ Conference (CppCon). (Next step: Deal with bottlenecks in user-mode memory allocators.)

Using and implementing RCU

Maya Arbel and Hagit Attiya of Technion took a more rigorous approach [PDF] to an RCU-protected search tree that, like Masstree, allows concurrent updates. This paper includes a proof of correctness, including proof that all operations on this tree are linearizable. Unfortunately, this implementation achieves linearizability by incurring the full latency of grace-period waits while holding locks, which degrades scalability of update-only workloads. One way around this problem is to abandon linearizability (as advocated here [PDF] and here [PDF]), however, Arbel and Attiya instead created an RCU variant that reduces low-end grace-period latency. Of course, nothing comes for free, and this RCU variant appears to hit a scalability limit at about 32 CPUs. Although I personally favor dropping linearizability, thus gaining both performance and scalability, it is very good to see academics experimenting with alternative RCU implementations. Researchers at Charles University in Prague have also been working on RCU implementations, including dissertations by Andrej Podzimek [PDF] and Adam Hraska [PDF].

RCU-like mechanisms are also finding their way into Java. Sivaramakrishnan et al. use an RCU-like mechanism to eliminate the read barriers that are otherwise required when interacting with Java's garbage collector, resulting in significant performance improvements.

A specialized RCU implementation and its use

The final piece of academic work that is described in this article was carried out at the Shanghai Jiao Tong University by Ran Liu, Heng Zhang, and Haibo Chen, who created a specialized variant of RCU that they used for an optimized reader-writer lock. This is important because traditional reader-writer lock implementations, such as the Linux kernel's rwlock_t, do not scale well. The reason for this is that all readers must atomically update the single memory location that represents the rwlock_t, both with entering and when leaving their read-side critical sections. The cache line containing this memory location will shuttle among all of these CPUs. Each access to this memory location will therefore result in a cache miss, which will in turn result in the accessing CPU stalling for hundreds or even thousands of clock cycles.

The more CPUs attempting to acquire or release the lock, the longer the stalls. Unless the read-side critical sections are extremely long (hundreds of thousands or millions of instructions), the result will be poor performance and scalability. Furthermore, some of the Linux kernel's lock implementations favor readers over writers (which can be OK), but to the extent of starving waiting writers (which can be a big problem).

This is, of course, why RCU is often used, but RCU has different semantics than do reader-writer locks. In a surprisingly large number of situations, this difference in semantics is not a problem. However, there are some parts of the Linux kernel that are notoriously difficult to apply RCU to. This situation has motivated a number of attempts to find more scalable implementations of reader-writer locking, going all the way back to brlock in the 2.4 Linux kernel (which has since been replaced by lglocks). These implementations provide a lock for each CPU. To read-acquire a lock, a CPU acquires its own lock, which still requires an atomic instruction and memory barrier on most architectures, but does not incur a cache miss in the absence of writers. Writers must acquire all CPUs' locks, which requires an atomic instruction and usually a cache miss per CPU. This overhead increases linearly with the number of CPUs, which can result in high overhead for updates.

In some cases, decreased read-side overhead is required. One scheme was put forward by Gautham Shenoy, and a similar approach was put forward by Srivatsa Bhat. The idea behind both of these patches is that readers check a flag before entering their read-side critical sections. If that flag indicates that there are no writers, then the readers proceed without executing any atomic instructions or memory barriers, and, in the common case, without incurring any cache misses. However, nothing comes for free, and for these approaches, the penalty is paid by the writers.

A writer must first set the flag, then wait for all the pre-existing readers, who might have loaded the flag before the writer set it. This is handled by requiring readers to check the flag within an RCU read-side critical section, and, if the flag is clear, to execute their entire read-side critical section under RCU protection. This allows the writer to use synchronize_rcu() after setting the flag to wait for pre-existing readers. Of course, the resulting grace-period latency can be problematic in many cases. This latency can be reduced by using expedited grace-period primitives such as synchronize_rcu_expedited(), but these result in high CPU overhead as well as a set of inter-processor interrupts (IPIs) to other CPUs. The high overhead can reduce throughput, and the IPIs can spell trouble for real-time applications.

Liu, Zhang, and Chen take a slightly different approach in their USENIX paper. As noted earlier, they use a special-purpose RCU implementation tailored specifically for reader-writer locking, which they call a "passive reader-writer lock" or prwlock. In a manner roughly similar to signal-based user-space RCU, writers increment a version number, and then wait for all CPUs to acknowledge the new version. Readers automatically acknowledge when acquiring or releasing the lock, but CPUs that are not using the lock will not acknowledge the new version.

One way of handling this would be to place full memory barriers in the read-side code, but this group decided to strive for ultimate read-side performance, which means that the read-to-write transition involves IPIs. The idea is that if a given CPU has not either entered or exited a read-side critical section in a timely fashion, it is sent an IPI, which causes it to respond. The group also looked at a few optimizations, including scoping the IPIs on a per-process basis for mmap_sem, optimizing the wakeup path, and, for user-space implementations, using an in-kernel helper similar in some ways to sys_membarrier(). This optimized reader-writer lock provided significant increases in performance for systems of up to 64 CPUs on a number of workloads (see pages 11-13 of the paper for details).

Like most conferences, USENIX imposes strict length limits, which means that this paper does leave some unanswered questions.

One question is whether the number of IPIs could be reduced by taking advantage of the Linux kernel's per-CPU dyntick-idle state. This seems possible, and it seems especially beneficial for NO_HZ_FULL kernels, where it would prevent IPIing time-critical user-mode realtime and HPC applications. It would be interesting to see how this approach plays out.

A second question involves Figure 17 on page 12, which shows that an RCU-based resizable hash table takes considerably longer to resize than does one protected by the prwlock. This is quite true, but it is also true that when using RCU, reads and updates can proceed concurrently with the resizing. It would be interesting to directly measure the actual effect of a resize operation on read and update throughput and latency.

A third question involves the paper's comparison of a user-mode implementation of the reader-writer lock to signal-based user-space RCU. In this case, the reader-writer lock used an in-kernel optimization. However, user-space RCU also has an in-kernel optimization in the form of sys_membarrier(). It would therefore be interesting to compare both mechanisms using their in-kernel optimizations. Similarly, it would be interesting to compare prwlock to RCU expedited grace periods as well as the slower normal grace periods.

A fourth question involves configuration of the benchmark runs. For example, the comparisons of prwlock to user-space RCU set user-space RCU's batch size to one. This is of course an attractive setting if you wish to highlight prwlock, but it would be more interesting to compare against the much larger default setting of 4096 for the batch size. It also raises the question of how and to what extent prwlock can amortize single sets of IPIs over the write acquisitions of multiple prwlocks.

A final question involves scalability, both in terms of numbers of prwlocks and in terms of system size. Exploring these areas should provide much interesting future work. All these questions aside, and in part because of these questions, this was an extremely interesting paper that might well have significant practical applicability.

Conclusions

It is very good to see increased academic interest in read-mostly techniques such as RCU. We can all look forward to much interesting (and hopefully some useful) work along the way.

Acknowledgments

We all owe thanks to Davidlohr Bueso, Austin Clements, Hagit Attiya, Maya Arbel, Eddie Kohler, and Haibo Chen for their help making this human-readable. I am grateful to Jim Wasko for his support of this effort.

Answers to Quick Quizzes

Quick Quiz 1: But what can I do if I work with production-quality code but nevertheless would like to discover something new?

Answer: There are no guarantees. That said, here are some things you can try:

1. Set up automated testing for your code, and make this testing as vicious as you possibly can. Otherwise, you will be spending all your time dealing with bugs and will therefore have no time to discover something new. (Yes, it is quite possible that one of your bugs will lead to something new, but the odds are stacked against you.)
2. Learn as much as you can about the area of your interest. However, emphasize learning by doing, and especially try doing things wrong to see what happens.
3. Study things completely unrelated to your area of interest. It is surprising how often standard practice in one area inspires something new in another area.
4. Seek out trouble, especially when the trouble involves doing something that has never been done before. (Important safety tip: It is almost as easy to overdo this particular piece of advice as it is to avoid it completely.)
5. Make heavy use of source-code control systems such as Git in order to manage the resulting chaos—or at least to keep the chaos safely removed from your production systems.

With some luck, persistence, and hard work, who knows what you might find?

Back to Quick Quiz 1.

Quick Quiz 2: Why don't more academic research projects make full use of open-source projects such as the Linux kernel?

Answer: As you can see from this article, a goodly number of researchers do use Linux. That said, the size and complexity of the Linux kernel can be problematic for some research projects—considerable courage and talent are required to take on Linux-kernel internals.

Back to Quick Quiz 2.

Quick Quiz 3: Given the need to examine per-CPU data for each and every CPU, isn't prwlock inherently non-scalable?

Answer: Not necessarily.

For example, there is no reason why multiple kernel threads could not be brought to bear in order to parallelize the scan of the per-CPU data. In fact, if examining one CPU's data required 100 ns and waking up a helper kthread required 5 us, then it might make sense to provision a helper kthread for each group of (say) 100 CPUs.

That said, this approach would get quite “interesting” from irq-disabled regions of code.

Back to Quick Quiz 3.

Comments (8 posted)

OpenWrt releases "Barrier Breaker"

Continuing its tradition of releases named after cocktails, the OpenWrt project released the latest version of its firmware for home routers and other devices, "Barrier Breaker" (14.07), on October 2. As one might guess, it comes with plenty of new features, especially in the networking area (improved IPv6 support, in particular). OpenWrt has long been used by many of the technically savvy, but it is reaching a point where it "just works" and provides such an upgrade from the factory firmware that it may well make sense for some regular users as well.

OpenWrt began in 2003 by starting with the code grudgingly released by Linksys to comply with the GPL for its WRT54G wireless home router. Support for other routers and devices soon followed and the current Table of Hardware that documents the hardware supported by the distribution is truly impressive. We last looked at OpenWrt in 2011, shortly before the "Backfire" maintenance release (10.03.1). Since then, there was also the 12.09 "Attitude Adjustment" release in April 2013.

Though it is getting a little long in the tooth now, the Netgear WNDR3800 is a nice router for testing (and running) OpenWrt. Used devices appear to be plentiful and fairly inexpensive (less than $75 on eBay) or new ones can be purchased for a bit more than double that. The WNDR3800 is certainly one of the easier routers to install OpenWrt on; it is simply a matter of grabbing the right firmware (the factory image for WNDR3800 from this directory) and uploading it to the device using the factory firmware's web interface. Upgrading from an earlier OpenWrt release uses the sysupgrade image, instead; uploading that file into the OpenWrt interface will upgrade the device while preserving the existing configuration (though not any extra packages that may have been installed).

Once installed, there are several things that OpenWrt does (or, perhaps, doesn't do) to improve the security of newly installed devices. To start with, WiFi and ssh are disabled, while telnet is enabled. Turning off ssh and telnet on might seem like poor choices, but the idea is to force the user to set a root password. Users can either telnet or browse to 192.168.1.1 (after connecting some system to one of the device's ethernet ports) to set the password. Once a password has been set (and the same root password works for logging in as root to both the ssh command line and the web interface), the telnet service is terminated and Dropbear SSH is started instead.

From then on, most of the administration or monitoring that needs to be done can either use the command-line interface (CLI) or the LuCI Lua-based web interface. It uses the Unified Configuration Interface (UCI), which provides the CLI tool for the configuration of OpenWrt devices. For example, one of the first tasks is likely to be turning on the WiFi, which is easily done from LuCI, though it can also be done using UCI or by editing /etc/config/wireless directly.

Using LuCI is quite straightforward. It has hierarchical menus that govern most of the tasks an administrator might need to do. There are, naturally, realtime traffic graphs of various sorts, along with log file output, diagnostic tools (e.g. ping, traceroute), and other troubleshooting and monitoring aids available. Configuring various services, such as DHCP or DNS, is easy to do. LuCI restarts the services or network interfaces as needed to effect whatever changes were made.

Beyond that, there are some fairly complicated configuration options that one normally would not see in a router's web interface. The firewall configuration and monitoring (listing all of the different tables) is top-notch. Port forwarding, quality of service (QoS), VLAN, and other configuration are all quite accessible.

On the negative side, though, is the documentation, which suffers from a few flaws. It is disorganized, with information scattered on the Documentation page, wiki, and in the forum. You can often find what you are looking for in one of those places, but it is easiest to consult a search engine. After finding the topic you are looking for, though, you may encounter another problem: the information may be out of date—sometimes long out of date. This is not really meant as a knock on the project, as the versatility of the distribution means that there is an incredible amount of information to maintain. But it can be frustrating to new users.

Some of that versatility and complexity may be part of the reason there are several derivative distributions based on OpenWrt. We have looked at CeroWrt and the EFF's Open Wireless Router project (which is based on CeroWrt) in the last few years. Both of those focus on a single router (the WNDR3800) to try to cut down the complexity of the multi-platform problem. Documenting the base functionality of those routers is a much simpler task.

OpenWrt has, thankfully, taken a much larger bite; its work can be picked up—and simplified—by others. Development flows in both directions, though, as OpenWrt has adopted most of the anti-bufferbloat work that CeroWrt has done.

Beyond just the multi-platform support, OpenWrt's complexity comes from the sheer number of packages it supports. Numbers are hard to come by, but Attitude Adjustment came with nearly 3,500 packages, so one would guess that Barrier Breaker has at least that many. Certainly scanning through the package list in LuCI is eye-opening. One of the features for the release was a reorganization of the package feed into a single GitHub repository. For now, the older repository is still available, but it must be manually enabled.

The opkg package management system works just as one would expect. Packages can be installed from the command line or LuCI and, as with any modern package manager, dependencies are automatically resolved. One of the features that will be coming is the ability to sign and verify packages, which is a welcome feature.

Security updates are somewhat problematic for OpenWrt, however. Kernel fixes require installing a new image. User-space vulnerabilities can be fixed by updating packages, but the distribution isn't really set up to handle the kind of advisory and update process that is normal for desktop and server distributions. Package signing seems like a step along the way toward a solution to some of these problems.

Barrier Breaker is based on the 3.10 kernel, which moves things along a good ways from the 3.3 kernel (with some backports from 3.5) that was used in Attitude Adjustment, but is still more than a year old at this point. The release notes for Barrier Breaker say that the next release (named "Chaos Calmer") will be based on 3.14 or some more recent kernel that has long-term support. Given that the project is aiming for a release of Chaos Calmer this year, it would seem that 3.14 is a good bet.

While it has been around 18 months between releases, OpenWrt certainly gives the appearance of being a highly active project. Another release before the end of the year would seem pretty aggressive, however. In any case, it is a distribution worth checking out for a wide variety of embedded Linux needs. As they say: Friends don't let friends run factory firmware. Perhaps that overstates things, but there are many things that can be done with an OpenWrt device—well beyond the manufacturer's vision.

Comments (14 posted)

Distribution quotes of the week

-- Joey Hess resigns from Debian (Thanks to Josh Triplett)

-- Sam Hartman

-- Jesse Smith (DistroWatch review)

-- Francesca Ciceri

Comments (2 posted)

Red Hat Enterprise Linux 7 Atomic Host Beta

Red Hat has announced the availability of the first public beta of Red Hat Enterprise Linux 7 Atomic Host. "Red Hat Enterprise Linux 7 Atomic Host Beta provides a streamlined host platform that is optimized to run application containers. The software components included in Red Hat Enterprise Linux 7 Atomic Host Beta, as well as the default system tunings, have been designed to enhance the performance, scalability and security of containers, giving you the optimal platform on which to deploy and run application containers."

Comments (59 posted)

Bits from the release team: Jessie Freeze

Debian 8.0 "Jessie" has been frozen. That means only release critical bugs will be fixed. There will be no more new features, just some polishing. The release team also provided some additional information from a recent sprint. During the sprint the code names for the next two versions of Debian were chosen: Debian 9 "Stretch" and Debian 10 "Buster".

Full Story (comments: none)

DEP-14: Recommended layout for Git packaging repositories in Debian

Raphaël Hertzog has posted a proposal for the standardization of Git repositories used for Debian packaging.

Comments on the proposal are requested (and many have been posted).

Full Story (comments: 5)

A proposed policy to remove unfixable packages from Ubuntu

In response to the recent ownCloud troubles, Martin Pitt has put together a proposal allowing for the removal of problematic packages from the Ubuntu repositories in the future. "In rare cases, an universe package becomes actively detrimental in stable releases: If it is unmaintained in Ubuntu and has unfixed security issues or got broken because of changing network protocols/APIs, it is better to stop offering it in Ubuntu altogether rather than continuing to encourage users to install it." Comments are requested.

Full Story (comments: 46)

Distribution newsletters

• DistroWatch Weekly, Issue 584 (November 10)
• 5 things in Fedora this week (November 5)
• Ubuntu Weekly Newsletter, Issue 391 (November 9)

Comments (none posted)

The 9 Best Linux Distros (Datamation)

Bruce Byfield takes a quick look at nine reliable distributions for everyday use. He includes Bodhi Linux, Debian, elementary, Fedora, Linux Mint, Mageia, Manjaro, openSUSE, and Ubuntu. "This list focuses on distributions for the average user. However, change what you are looking for, and the list changes, although a few distros like Debian are versatile that they tend to show up on any list."

Comments (none posted)

Meteor: A framework for web applications

Writing rich web applications can be demanding. Developers might be expected by their clients or employers to have a good grasp of HTML, JavaScript, and PHP, as well as other tools and languages. The desire to simplify and hasten web application development has led to the rise of open-source frameworks, such as Node.js and Angular.js, that provide most or all of what's needed to build and deploy a web application. Meteor, another open-source web-application framework, stands out with a number of notable features.

Thousands of developers, in dozens of cities around the world, celebrated Meteor's 1.0 release on November 6: Worldwide Meteor Day. The release announcement nicely sums up Meteor's features:

The real-time updating system, which instantly pulls in changes to the database and refreshes the client-side interface to reflect them, may be the greatest advantage that Meteor has to offer over competing, server-centric solutions like Ruby on Rails. Meteor provides a full-stack solution, while Ruby on Rails has to be configured and connected to a separately managed front-end framework for the client interface.

Live updating is provided by Meteor's various software components, particularly Blaze and the Distributed Data Protocol (DDP). With Blaze, developers write HTML templates, and the software transforms them into elements that live-update the document object model (DOM) of the page. This lets users see database changes in real-time. DDP assists the live-updating process by using JSON and WebSockets; it lets web applications retrieve server data and update on-the-fly based on that data.

There are multiple ways that live updates can be used by web applications. For example, the open source Telescope social news application, which is built in Meteor, adds new entries in real-time on users' browsers without needing to refresh the page.

Because of Blaze, DDP, and other Meteor subprojects, Meteor allows developers to write client-side and server-side code in the same file. Meteor thus removes some of the concerns that developers had with other frameworks, such as Angular.js, that require extra code be written to manage the interactions between the front-end and the back-end. This simplification can be a major advantage to small teams trying to make the most of limited time and resources; with Meteor, all a team needs is a single JavaScript developer who can write both front-end and server-side code.

Meteor's back-end is based on Node.js, but provides more features and ease-of-use for web developers. Meteor is easy to pick up: a basic understanding of JavaScript is all that's needed to get started. Those interested in playing around with it can sign up on www.meteor.com to make and deploy a test Meteor application for free. For inspiration, scroll down on that page to see videos about six different, financially successful projects relying mostly or entirely on Meteor. These include Workpop, which is an online job marketplace. Readers who are interested in the code of Meteor itself can visit the project's GitHub repository; Meteor is licensed under the permissive X11 license.

The Meteor project started in December 2011 under the name Daybreak; in April 2012, its name changed to Meteor and the startup company backing its development became the Meteor Development Group (MDG). There was considerable interest about the project, in part due to MDG's founding members who have worked on a variety of web-centric projects (e.g. Etherpad, Miro). That expertise in web development, along with Meteor's promised live-updating features and easy conversion to mobile apps, led to a lot of interest in a 2012 Hacker News discussion as well.

MDG eventually intends to develop a proprietary multi-tenant hosting environment for Meteor applications, named Galaxy, to make the endeavor profitable. That proprietary product is a long-term milestone for MDG, though; it would only be pursued once Meteor has matured and become popular.

Meteor has indeed started to become popular—something that was made apparent at a Worldwide Meteor Day celebration I attended. The event took place at Shopify's headquarters in Ottawa, Canada, where several enthusiasts praised Meteor and explained its technical features to the rest of the group. The group also interacted with other celebrations online via Google Hangouts.

Meteor has an active development community, which has generated a number of resources for programmers. These include educational ebooks like Discover Meteor, videos from Meteor training website EventedMind, and a discussion forum on Google Groups. With a focus on simplifying workflows, a company backing its development, and a strong community, Meteor looks to have a substantial impact on web development in the years to come.

Comments (none posted)

Quotes of the week

— Larry Wall

— James Bridle (Thanks to Paul Wise)

— Junio C Hamano

Comments (13 posted)

Firefox developer edition released

Mozilla has announced the first release of a version of the Firefox browser aimed at web developers. "Ten years ago, we built Firefox for early adopters and developers to give them more choice and control. Firefox integrated WebAPIs and Add-ons to enable people to get the most out of the Web. Now we’re giving developers the whole browser as a hard-hat area, allowing us to bring front and center the features most relevant to them. Having a dedicated developer browser means we can tailor the browsing experience to what developers do every day."

Comments (3 posted)

GnuPG 2.1.0 "modern" released

Version 2.1.0 of the GNU Privacy Guard has been released; this is the first release in the new "modern" branch. Changes include elliptic curve cryptography support, better keyserver pool handling, the creation of revocation certificates by default, the removal of support for PGP2 keys, and more.

Full Story (comments: 41)

New releases from KDE

KDE has announced the release of KDE Frameworks 5.4.0. KDE Frameworks are 60 addon libraries to Qt which provide a wide variety of commonly needed functionality. This release fixes many issues.

KDE Applications and Platform 4.14.3 has been released. This release also includes Plasma Workspaces 4.11.14. This a primarily a bug-fix release.

KDE Applications 14.12 beta is available for testing. "With the various applications being based on KDE Frameworks 5, the KDE Applications 14.12 releases need a thorough testing in order to maintain and improve the quality and user experience. Actual users are critical to maintaining high KDE quality, because developers simply cannot test every possible configuration. We're counting on you to help find bugs early so they can be squashed before the final release."

Comments (5 posted)

Microsoft open-sources the .NET core

Microsoft has announced that the .NET core code is now available under an open-source (MIT) license. "As a .NET developer you were able to build & run code on more than just Windows for a while now, including Linux, MacOS, iOs and Android. The challenge is that the Windows implementation has one code base while Mono has a complete separate code base. The Mono community was essentially forced to re-implement .NET because no open source implementation was available." Amusingly, the code has been placed on GitHub; the announcement notes that code located there gets far more contributions than code on Microsoft's own "CodePlex" site.

Comments (76 posted)

Development newsletters from the past week

• What's cooking in git.git (November 5)
• What's cooking in git.git (November 11)
• LLVM Weekly (November 10)
• OCaml Weekly News (November 11)
• Perl Weekly (November 10)
• PostgreSQL Weekly News (November 9)
• Python Weekly (November 6)
• Ruby Weekly (November 6)
• This Week in Rust 56 (November 10)
• Tor Weekly News (November 12)
• Wikimedia Tech News (November 10)

Comments (none posted)

Kügler: Diving into Plasma’s 2015

On his blog, Sebastian Kügler looks at what next year holds for KDE Plasma 5. He looks at high-DPI and Wayland support as well as the plans by distributions (Kubuntu 15.04 for example) to start shipping Plasma 5 as the default desktop environment. "In terms of user demographic, we’re almost certain to see one thing happening with the new Plasma 5 UI, as distros start to ship it by default, this is what these new users are going to see. Not everybody in this group of users is interested in how cool the technology stack lines up, they just want to get their work done and certainly not feel impeded in their daily workflows. This is the target group which we’ve been focusing our work on in months since summer, since the release of Plasma 5.0. Wider group of users sounds pretty abstract, so let’s take some numbers: While Plasma 5 is run by a group of people already, the number of users who get it via Linux distributions is much larger than the group of early adopters. This means by the end of next year, Plasma 5 will be in the hands of millions of users, probably around 10 million, and increasing."

Comments (none posted)

Peck: New GIMP Save/Export plug-in: Saver

At her blog, Akkana Peck has announced a new GIMP plugin called "Saver" that is intended to replace the default Save/Export functionality introduced with the GIMP 2.8 release. GIMP 2.8 famously separated "Save"and "Export" into two separate functions, with "Save" only able to write out images to GIMP's native, multi-layer XCF format. As Peck notes, that change "has been a matter of much controversy. It's been over two years now, and people are still complaining on the gimp-users list." The new plugin is an attempt to perform the "expected" action in each circumstance. "I've been using Saver for nearly all my saving for the past year. If I'm just making a quick edit of a JPEG camera image, Ctrl-S overwrites it without questioning me. If I'm editing an elaborate multi-layer GIMP project, Ctrl-S overwrites the .xcf.gz. If I'm planning to export that image for the web, I Ctrl-Shift-S to bring up the Saver As... dialog, make sure the main filename is .xcf.gz, set a name (ending in .jpg) for the exported copy; and from then on, Ctrl-S will save both the XCF and the JPG copy.

Comments (38 posted)

Meeks: OpenGL rendering for LibreOffice 4.4

Michael Meeks looks at OpenGL rendering in LibreOffice. "Image scaling is another area where we currently suffer; with several open bugs - first one complains about performance, and then when you lower rendering quality to get performance, another bug complains about rendering quality. Doing high quality image interpolation of large images takes time, even when threaded. People love to whack large, high-DPI images into their documents and presentations. By moving all of the image interpolation work to the GPU we should be able to have our cake: pretty scaled images, and also eat it quickly: with fast rendering."

Comments (19 posted)

Baker: Mozilla and the Future of the Open Internet

Mitchell Baker celebrates Firefox's 10th anniversary. "The answer is: yes, Firefox did win in the desktop era. We changed the fundamental landscape by bringing a new experience and a new view of the world to hundreds of millions of people. However, there is still essential work to do as the Web still faces real threats today — and likely will again in the future. Here are details on what’s happening as part of the 10th anniversary of Firefox."

Comments (2 posted)

FSF and Software Freedom Conservancy unveil Copyleft.org

The Free Software Foundation (FSF) and the Software Freedom Conservancy (SFC) have announced a new site called Copyleft.org that will play host to "useful information, tutorial material, and new policy ideas regarding all forms of copyleft licensing." The most prominent content at present is a comprehensive guide to the concept of copyleft and copyleft licenses. The announcement notes that the content is viable, among other things, as training material. "As the author, primary interpreter, and ultimate authority on the GPL, the FSF is in a unique position to provide insights into understanding free software licensing. While the guide as a living text will not automatically reflect official FSF positions, the FSF has already approved and published one version for use at its Seminar on GPL Enforcement and Legal Ethics in March 2014."

Comments (none posted)

GNOME gets GroupedOn

GroupOn, a sort of Internet sales discount coupon company has recently announced a point-of-sale tablet called "Gnome". The GNOME Foundation, by virtue of having used that name since the 1990's and having trademarked it in 2006, objects strongly to what it sees as a blatant infringement of its trademark. The organization is scrambling to file its opposition to GroupOn's new trademark filings, but that takes work — and money. So there is now a fund-raising effort in the works to help make this opposition happen. "Help us raise the funds to fight back and most of all call public attention to this terrible behavior by Groupon. Help us make sure that when people hear about GNOME software they learn about freedom and not proprietary software. Our counsel has advised us that we will need $80,000 to oppose the registration of the first set of 10 applications. If we are able to defend the mark without spending this amount, we will use the remaining funds to bolster and improve GNOME."

Update: according to Engadget, GroupOn says it wants to work things out, all the way to picking a new product name if necessary.

Another update: The GNOME Foundation reports that Groupon will abandon its pending trademarks and proceed with a name change.

Comments (75 posted)

FSFE Newsletter – November 2014

The Free Software Foundation Europe's newsletter for November covers free software in Munich, EU wide open standards policy, open standard compliance checks, open internship positions, and several other topics.

Full Story (comments: none)

Python @ FOSDEM 2015 - Call For Proposals

The Python Devroom at FOSDEM (January 31-February 1 in Brussels, Belguim) will be open all day on January 31. The call for proposals is open until December 1.

Full Story (comments: none)

CFP Deadlines: November 13, 2014 to January 12, 2015

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
November 30	January 13	Linux.Conf.Au 2015 Systems Administration Miniconf	Auckland, New Zealand
December 1	February 6 February 8	DevConf.cz	Brno, Czech Republic
December 1	March 11 March 12	Vault Linux Storage and Filesystems Conference	Boston, MA, USA
December 7	January 31 February 1	FOSDEM'15 Distribution Devroom/Miniconf	Brussels, Belgium
December 8	February 18 February 20	Linux Foundation Collaboration Summit	Santa Rosa, CA, USA
December 10	February 19 February 22	Southern California Linux Expo	Los Angeles, CA, USA
December 14	January 12	LCA Kernel miniconf	Auckland, New Zealand
December 17	March 25 March 27	PGConf US 2015	New York City, NY, USA
December 21	January 10 January 11	NZ2015 mini-DebConf	Auckland, New Zealand
December 21	January 12	LCA2015 Debian Miniconf	Auckland, New Zealand
December 23	March 13 March 15	FOSSASIA	Singapore
December 31	March 17 March 19	OpenPOWER Summit	San Jose, CA, USA
January 1	March 21 March 22	Kansas Linux Fest	Lawrence, Kansas, USA
January 2	May 21 May 22	ScilabTEC 2015	Paris, France
January 5	January 12	Linux.conf.au 2015 Multimedia and Music Miniconf	Auckland, New Zealand
January 5	March 23 March 25	Android Builders Summit	San Jose, CA, USA
January 5	February 11 February 12	Prague PostgreSQL Developer Days 2015	Prague, Czech Republic
January 9	March 23 March 25	Embedded Linux Conference	San Jose, CA, USA
January 10	May 16 May 17	11th Intl. Conf. on Open Source Systems	Florence, Italy
January 11	March 12 March 14	Studencki Festiwal Informatyczny / Academic IT Festival	Cracow, Poland
January 11	March 11	Nordic PostgreSQL Day 2015	Copenhagen, Denmark

If the CFP deadline for your event does not appear here, please tell us about it.

Events: November 13, 2014 to January 12, 2015

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
November 9 November 14	Large Installation System Administration	Seattle, WA, USA
November 10 November 14	21'th Annual Tcl/Tk Conference	Portland, Oregon, USA
November 13	Hackaday Munich	Munich, Germany
November 16 November 21	Supercomputing 14	New Orleans, LA, USA
November 17 November 21	ApacheCon Europe	Budapest, Hungary
November 18 November 20	Open Source Monitoring Conference	Nuremberg, Germany
November 19 November 21	CloudStack Collaboration Conference Europe	Budapest, Hungary
November 21 November 23	Debian Bug Squashing Party in Munich	Munich, Germany
November 22 November 23	AdaCamp Bangalore	Bangalore, India
November 25	New Directions in Operating Systems	London, UK
November 29 November 30	OpenPhoenux Hard and Software Workshop	Munich, Germany
December 5 December 7	SciPy India	Bombay, India
December 27 December 30	31st Chaos Communication Congress	Hamburg, Germany
January 10 January 11	NZ2015 mini-DebConf	Auckland, New Zealand

If your event does not appear here, please tell us about it.