Wikipedia reconsiders H.264 support

As a matter of policy, Wikipedia has historically opposed using the H.264 video codec, because the organization's mission dictates the use of open data formats. But the Wikipedia Foundation multimedia team has now opened a request for comments (RfC) asking the community's input as to whether or not this stance should be reconsidered—specifically, whether to make H.264 video a fallback option for users who cannot view open formats. The rationale for the proposal incorporates the difficulty of creating (in addition to viewing) open video formats, but there are still quite a few community members who oppose the idea of any H.264 support.

Pros and cons

The RfC was opened on January 15 and will remain open until February 14. At that time, the multimedia committee will consider the results of the public comments before it makes its decision, although the wording of the RfC indicates that the public comments will be merely advisory. The document leads off with a summary of Wikipedia's current video situation, which led to the consideration of a policy change. It then summarizes the basic arguments for and against adding H.264 support.

The impetus for the discussion is, essentially, that video has yet to make a significant impact on the overall Wikipedia experience. Despite the benefits that video resources could bring to a variety of topics (as well as to users who cannot read), so far only 40,500 video files have been uploaded to the project. That constitutes about 0.2% of the content at Wikimedia Commons and, the RfC points out, pales in comparison to the number of educational videos contributed to YouTube (at about 6.5 million).

One possible explanation is that Wikipedia's insistence on supporting only open video formats (Ogg Theora and WebM) hampers video contribution. Few readily-available devices can record Theora or WebM, and few applications can edit or export to either format. The situation for viewing open video formats is just as bad; one third of Wikipedia page hits come from mobile browsers (which by and large do not support Theora or WebM playback), and a significant portion of desktop browser users (namely those running Safari or Internet Explorer) cannot play back the open formats either.

As most people who follow the multimedia codec space know, the situation is almost the reverse for H.264: mobile phones and video cameras can record directly to H.264, video editing software support is widespread, and the majority of users can view it on the desktop and on mobile devices.

The problem, of course, is that H.264 is proprietary and patent-encumbered. Wikipedia's legal department has decided that the H.264 licenses required would be acceptable; essentially the project would need to implement H.264 encoding and decoding for videos uploaded to Wikimedia Commons, but Wikipedia would not have to pay license fees to publish H.264 videos, nor would readers have to pay fees to view them.

The basic plan being offered for comment would automatically transcode all video uploads, so that every video is available in both H.264 and in an open format. The open formats would be the default, but would fall back to H.264 for browsers and devices without support for Theora and WebM.

This plan, the RfC emphasizes, would also transcode H.264 uploads into an open format—not just transcode open formats to H.264. This is to ensure that users opposed to supporting proprietary video would not be locked out even if the original uploader used H.264. Nevertheless, four options are put forth for community comment: fully supporting both H.264 and open formats, supporting H.264 only for viewing, supporting H.264 only for uploads (i.e., automatically transcoding uploads to an open format), and not supporting H.264 at all.

Compromise

The various arguments for and against H.264 support are discussed in more detail in the RfC (as is true of Wikipedia "talk" pages, most of the comments are signed, but unfortunately there are not direct links to them). Essentially, though, no variation of the proposed change involves dropping support for open video formats or even for dropping them as the default. Thus, the major arguments against adding H.264 support are not about whether or not H.264 is preferable to the open formats or whether open formats should be abandoned; rather, they are arguments about whether the compromise to support H.264 is a net gain or loss. Since the answer to that question requires peering into the future, it is naturally a bit speculative.

For example, one argument against adding H.264 support is that the open formats are catching up to it. First, Google has secured support for VP9 (the successor to WebM's current codec, VP8) from a long list of hardware vendors, so the next generation of mobile devices will be able to record and edit VP9 natively. Second, the development of Xiph.org's new Daala video codec is being staged out to compete head to head with future proprietary video codecs that are still several years down the road. Many attribute H.264's dominance over VP8 to the fact that the proprietary codec had a head start. Theora was never really competitive in that fight, as it was an even older codec.

But with Daala's opportunity to leapfrog the proprietary alternatives, the argument goes, the future actually looks quite good for open formats, so Wikipedia backing down on its stance against them would be detrimental. The flip side was argued by Michael Dale, who said that OpenOffice.org's interoperability with proprietary formats was what allowed it to gain a significant foothold. The usage of free formats cannot grow on principles alone, he said, "Interoperability is the only way to grow our video platform that has free format support."

Of course, there are other arguments against compromise. One mentioned in the RfC is that Wikipedia has a commitment to open source software in addition to open content; deploying H.264 transcoders on Wikipedia's servers could mean adding a component that Wikipedia would have trouble freely distributing to downstream users—since those users would likely still be required to purchase an H.264 license. The foundation prefers not to distribute components that might place users in such a bind, the RfC notes, although, in fact, Wikipedia's Director of Platform Engineering Rob Lanphier said in an IRC discussion: "I don't think we can say for sure we'll be using free software for this."

The soap box

Anyone with a viewpoint to share on the proposed plan is invited to add a comment; the RfC is not limited to registered Wikipedia users or contributors. As of this writing (about one week after the RfC was posted), 421 signed statements have been added by the public; more than half (245) oppose any H.264 support, while 123 are in favor full upload and playback support, 44 favor supporting H.264 uploads only, four favor enabling H.264 playback only, and five describe themselves as completely neutral.

There have been a few alternate proposals as well, such as supporting playback only for the H.264 "baseline" profile (the patents for which will be the first to expire) or partnering with YouTube (so that YouTube delivers video content for Wikipedia pages). Few of these suggestions have garnered any significant support.

Of course, it is also difficult to adjust the numbers for how well the voluntary public comments represent the overall views of the Wikipedia community. Since there is a major ideological component to opposing H.264 support, one would certainly expect a more vocal crowd to weigh in—but that does not mean that the opposition viewpoint is the real majority.

But the RfC is not a ballot measure being put up for a vote; the Wikipedia Foundation will make its own decision—it is simply asking for a public discussion of the issues. One of the benefits of the public discussion should be that additional viewpoints and concerns come out that were not raised by the multimedia team itself, and that appears to be the case. Several of the comments raise important side issues. For example, user Alterego asks:

Other commenters have asked whether or not supporting H.264 could result in a tidal wave of low-quality video uploads, which would consume far more volunteer time and server resources to vet than do text or image submissions. Of course, the quality level of Wikipedia's text articles is the subject of frequent debates too, but the original point still stands: video itself may be desirable, but a large influx of it would pose practical challenges regardless of which codecs are used.

The other wrinkle to the discussion is that there is not a viable alternative plan that would substantially alleviate the initial concerns: the dearth of video content uploads, the difficulty of editing and converting Theora and VP9, and the inability of many Wikipedia users to view the open video formats. In other words, if Wikipedia rejects the idea of H.264 fallback support, it will still have to look for another solution—and so far no one has proposed one that has any traction.

In a sense, the community comments fall into camps that disagree on the most fundamental aspect of the question: those favoring H.264 support do so on practical grounds, while those opposing H.264 support do so largely on philosophical grounds. That distinction about the nature of the debate makes it difficult to imagine the two sides reaching a consensus in the middle. Thus, while the multimedia committee no doubt values the open discussion, that discussion is unlikely to resolve itself into an answer. The team will make its own decision, using whatever balance of practical and philosophical merits it chooses.

The greater free software world is no stranger to this debate; in 2012, Mozilla enabled fallback support of OS-level H.264 decoders for many of the same reasons articulated in the Wikipedia debate. If that is a valid predictor of the general temperature, the safe bet would be for Wikipedia to adopt some form of H.264 support, at least for video playback. Those with strong ideological opposition to that possibility could perhaps take comfort in the fact that Mozilla, true to its word to "fight another day," has indeed devoted considerable effort to making the next generation of free video codecs a best-in-class option. No doubt, though, that the next-generation codec battle will be just as contentious as this one.

Comments (20 posted)

US government opens access to scientific research

On January 16, the US Congress passed its annual "omnibus" spending bill for the year, which included a provision that mandates the open access publishing of a large swath of scientific research that is funded by taxpayers. Estimates say that the proportion of research projects that will be released under the new provisions account for just over half of the government's total research funding. Although the importance of open access is easy to lose among all of the various "open x" movements that have arisen since the rise of open source software, the new policy is one with ramifications for the open source community—given how much scientific research these days is conducted with software.

Officially, the omnibus bill is known as the Consolidated Appropriations Act for 2014, which is available in the form of a 1,582-page PDF for those brave enough to download it. The important section, however, is on page 1,020, which the Electronic Frontier Foundation (EFF) kindly extracted into a standalone download [PDF] for the organization's own blog entry about the new policy.

The rule applies to those agencies under the Departments of Labor, Health and Human Services, and Education which have annual research budgets greater than $100 million, and stipulates that they must release to the public the final manuscripts of all research that is fully or partially funded by the government—and that they must do so no later than twelve months after the research is published in a peer-reviewed journal. The rule also specifies that the publication must be online, free, and published in a machine-readable format.

The three departments listed may not sound like the nexus of scientific research (the Department of Energy, for example, is home to a lot of research though its National Laboratories), but those departments do include some key agencies, such as the Food and Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), National Institutes of Health (NIH), and Occupational Safety and Health Administration (OSHA). Certainly a lot of scientific research is funded through other agencies (including NASA and the National Science Foundation), but the section of the omnibus bill including the new research rules applies only to this set of departments. In its analysis of the bill, though, the Scholarly Publishing and Academic Resources Coalition (SPARC) says that the covered agencies (that is, excluding those agencies that fall below the $100 million limit) total around $31 billion out of the government's $60 billion annual research investment.

The new rule builds on two similar moves from recent years. In February 2013, the White House published an "open access memo" directing federal agencies to develop a plan for publishing publicly-funded research; the law essentially makes that plan mandatory for the specific agencies it covers. But even that 2013 memo is seen as the successor to the successful public access policy that the NIH voluntarily started in 2008. That policy is regarded as one of the cornerstones of the Human Genome Project and several subsequent NIH projects.

Research

Naturally, there is an easy case to be made for publishing this research simply because it is funded by the public's tax dollars. That seems "fair" by most definitions, and indeed the open source community has repeatedly shown that it can take taxpayer-funded public releases and transform them into something with more value than the original—the Department of Veterans Affairs' electronic health platform VistA, for example, or OpenStreetMap's incorporation of the Census Bureau's TIGER map database.

But scientific research is special in other ways that make the new policy particularly important. Restricting access to published research hampers attempts to build on previous results and makes it difficult or impossible for independent researchers to verify a study's findings. Those principles are key to the "open access" movement as it is defined by the Public Library of Science (PLOS) and similar initiatives.

In most cases, however, the main voice opposing open access publication is not the researchers themselves, but rather the publishers of the scientific journals where the research is found. The journal publishers typically claim copyright over the author's articles, and only allow access to them for a fee. Since publication in a well-known journal is typically a prerequisite for considering the research validated (not to mention ensuring job security for the researcher), the publishers can get away with it, and will take extreme measures to protect their access-fee revenue.

In December, for example, The Washington Post reported that journal publisher Elsevier sent out thousands of copyright-infringement takedown notices to researchers and universities for making their own research available on their own web sites. Most complied, although there was considerable blowback against the company.

More data

The practical effect of the new regulation, therefore, is to free a great many researchers from draconian publishing contracts that prohibit publicly sharing the researchers' work; US government funding underwrites so much research that few if any journals could risk running afoul of its rules. That is undoubtedly a good thing for the research community, but for many it is not sufficient. First, the new rule only applies to the final paper that is published; it does not mandate that the raw data, for example, also be published. Admittedly, in most cases, if researchers are amenable to freely publishing a study, they are not likely to oppose publishing the data with it, but it is still optional.

Second, the rule only requires that the published research be online, machine-readable, and free; it does not mandate any document formats. Open access advocates emphasize that publishing in open formats is important to making a study genuinely accessible. That concern is obviously more relevant for publishing a data set than a manuscript, but PLOS and other open access proponents want to see both changes.

Finally, the EFF and others advocate the passage of a separate law, the Fair Access to Science and Technology Research (FASTR) Act, which would not only broaden the scope of the policy, but would be more resistant against change than the White House memo. The memo, they note, covers agencies not addressed by the new omnibus bill, but it is only a memo—the next administration can undo it with another memo.

The new open access policy has the potential to release a considerable volume of research studies to the public in the coming months and years. Modern scientific research is a field that relies heavily on open source: data mining, statistical analysis, and simulation tools, for example. As more and more research reaches the public that previously had no access to it, one can expect the software community to continue playing a key role in the resulting conversations—validating and reproducing results, correlating studies, and so forth. Consequently, an increase in open access research may lead to more software contributions from researchers, although they are likely to be limited to specialized tools.

Just as interesting, though, will be seeing how the academic publishers react to the changes in their business model—and what effect they will have on the continuing debate about data publication and open formats. It took the open source community years to sort out community norms for critical questions like license interoperability, attribution and sign-off, trademarks, business models, and distributed workflow—in fact, many would probably argue that most of these issues have yet to be "sorted out" entirely. So the scientific research community may contribute code, but the open source community will likely be called upon to contribute back in other ways, too, as the open access community grapples with its own set of analogous problems.

The open source community has certainly had its share of false starts, blind alleys, and fiascoes over the years; perhaps open access research will fare better by learning from the software community's example. But either way, open access is poised to become a major public issue, and its overlap with open source will no doubt have long-term consequences.

Comments (9 posted)

A Debian init system update

The discussion on the debian-ctte list has been long, detailed, and only occasionally heated. Over time, the committee members have revealed their positions on the matter. Russ Allbery came out in favor of systemd just before the end of the year; since then, he has been joined in that position by committee chair Bdale Garbee, brand-new member Keith Packard, and (with weak support) Don Armstrong. Members who have identified themselves as being in the upstart camp are Ian Jackson, Colin Watson, and Andreas Barth. The only member of the committee who has not posted a summary of his position is Steve Langasek, but, given that he is the Debian upstart maintainer, there are few doubts as to where he will land.

The posted positions have led to a complaint that the OpenRC option has not been fairly considered by the committee. OpenRC, it is claimed, is the only init system that brings new features while working on all of Debian's target architectures. This appeal does not appear to have changed many minds, though; the prevailing opinion appears to be that OpenRC, while having a number of nice attributes, is too far behind and lacks a development community large enough to catch up in the near future.

A divided committee?

Perhaps the more compelling question, though, is this: what happens if the committee ends up hung on a four-to-four split? Section 6.3.2 of the Debian Constitution states that the chair of the committee has a casting (tie-breaking) vote. So, if the committee is evenly split on the issue the final decision could be made by Bdale, who has identified himself as being in the systemd camp.

That assumes, of course, that section 6.1.4 of the constitution does not apply. That section states that, in cases where the committee is voting to overrule a Debian developer in the management of their packages, a three-to-one supermajority is required. It has been suggested in the discussion that a vote to pick anything other than sysvinit amounts to overriding the sysvinit maintainers, but it is unlikely that this viewpoint would carry the day. The committee would be forcing no changes to sysvinit; instead, it would be changing which init system is deemed to be the default for Debian. So a simple majority should suffice in this case.

That said, a simple majority in the form of a 4:4 decision pushed one way or the other by the committee chair will certainly not be seen by all developers as being truly decisive or representative of the project. In such a situation, the wider debate is unlikely to come to a conclusion once the results of the vote are announced; indeed, it may become stronger and more polarized than ever. It seems almost certain that the level of discontent would be high enough that a general resolution — a referendum voted on by all Debian developers — would be certain to follow.

Indeed, Guillem Jover has already proposed such a resolution without waiting for the committee to act. He claims that referring the problem to the committee is inappropriate for a number of reasons, including a lack of sufficient effort to reach consensus, a perceived attempt by the committee to do "design work" (not allowed by the constitution), and the above-mentioned need for the 3:1 supermajority. Others disagree with his reasoning, needless to say; it is hard to imagine, for example, that the numerous lengthy discussions spread out over years do not amount to a sufficient attempt to reach consensus.

But even people who disagree with Guillem's reasons might find themselves backing a general resolution; Ian, for example, described himself as "not entirely opposed to the idea". He made the point that much of the argument around init systems is not actually technical. Members of the committee on both sides of the debate have made it clear that, in the end, they could live with a decision for either systemd or upstart, even if it wasn't the one that they wanted. One way or another, either system could be made to meet Debian's needs. But that is not all that is being debated here.

Systemd opponents, for example, go to great pains to cite the allegedly poor attitude of that system's primary developers; there have also been worries about adopting a system that is seen to be controlled by Red Hat. Upstart detractors see attitude problems of their own at Canonical — another commercial distributor, albeit a Debian-based one — and nobody likes the contributor agreement that Canonical demands of those who would submit patches. The importance of the non-Linux ports to the Debian project is also an important issue; upstart is seen as being more friendly to the non-Linux ports (even if it supports neither of them currently) than systemd is. These issues are all hard to weigh in a committee that is supposed to restrict itself to technical decisions.

In a project like Debian, the more political issues might just be better suited to a general resolution. There is an interesting constitutional quirk that comes into play, though: section 4.1.4 allows the technical committee to be overridden by a general resolution, but only with a 2:1 supermajority. So the worst-case scenario could be ugly: a hung committee releases a decision with the chair's casting vote; the community then votes for a different alternative via general resolution, but without enough votes to actually override the committee. That seems unlikely to be a happy day in the Debian community.

In response to this concern, Ian suggested that the committee could explicitly include a clause in its decision allowing that decision to be overridden by a simple majority vote. As an alternative, the committee could simply step back and send the issue to a general resolution from the outset. That outcome seems unlikely, though; the committee members seem to feel a responsibility to reach a decision on this issue if at all possible. So there will likely be an attempt to come to some sort of conclusion within the committee. And, regardless of which system is chosen, there are a number of associated policy and transition details to be dealt with. To that end, Ian has been drafting a resolution in the committee's git repository; it talks about upstart, but that would be changed if the committee were to go with systemd.

When a decision will be made is unclear, though one might expect that the committee members would, by now, be united in their desire to put this particular issue behind them. What that conclusion will be is also still hard to guess, even after the members have posted their thoughts; some of them seem to feel more strongly about their positions than others, so it is possible that some changing of minds could occur. With luck, the committee will come to a reasonably well-supported decision that will allow the project to move on with getting the "jessie" release ready. Other distribution communities that have had messy init-system discussions have mostly found peace once the decision was made; hopefully Debian will be able to follow a similar path.

Comments (280 posted)

FreeOTP multi-factor authentication

Back in 2010, Google rolled out multi-factor authentication for its services using an open source project named Google Authenticator. The system generated one-time pad (OTP) passcodes on a mobile phone; users could log in to their favorite Google service with their usual password plus the passcode. In addition to its availability on Google servers, the open source project included a PAM module that Linux users could run on their own machines. Sadly, Google Authenticator fell victim to the gradual move toward proprietary code of recent "Google Play Services" releases and the open source version was discontinued, without fanfare, in 2013. Fortunately, the community has picked up the challenge and is developing a workalike called FreeOTP.

As we saw in our look at Google Authenticator, the system consisted of two halves: a mobile application and a server-side authentication plug-in. When provisioned with a key, the mobile app generated either HMAC-based OTP (HOTP) or Time-based OTP (TOTP) passcodes. Since HOTP and TOTP are both open IETF standards, the app could be used with any compliant server and supported storing multiple sets of credentials—the only prerequisite being that the phone would needed to be provisioned with the key for each service. The server-side tools included a key-generation program, and the project was popular enough that multiple third-party projects and web services deployed it.

The turn from free software to proprietary software came after the 2.21 release. The notice for the change was posted only as an update to the Google Authenticator web page at Google Code, which was highlighted in this Reddit discussion from May 2013. The wording change noted simply that the subsequent, closed-source versions of the app "contain Google-specific workflows". FreeOTP was started in August 2013; although it includes bits of code from Google Authenticator, it is predominantly a fresh rewrite—albeit one that does its best to serve as a free-software clone of the original.

The new project is hosted at Fedorahosted.org, and is maintained by Red Hat's Nathaniel McCallum. On the mobile side of the equation, it replicates the Android and iOS client apps, but drops the Blackberry variant. As of right now, the iOS app has not yet been published on Apple's app store, but the Android version is available both through Google Play and through the free-software repository F-Droid.

On the server side, so far FreeOTP is not producing any components. Since any standards-compliant HOTP or TOTP server should work with FreeOTP, there is no absolute need for the project to produce its own server-side code. One of the nice things about Google Authenticator and FreeOTP, of course, is that the server and the client do not talk directly to each other: the client app generates the passcode, then the user types it in. Still, this does mean that the HOTP/TOTP Linux PAM module is now unmaintained. Although it is likely to work, that likelihood certainly comes with no guarantees—particularly since system security is at stake.

The FreeOTP project site points users to FreeIPA as a recommended solution. FreeIPA is an identity-management suite offered by Red Hat that glues together several existing open source components to provide LDAP, Kerberos, RADIUS, and other authentication or authorization services. McCallum, unsurprisingly, works on FreeIPA, and has added HOTP/TOTP support to MIT Kerberos. Work is also underway for a standalone HOTP/TOTP implementation in FreeIPA, though it is still under development.

Two-factorization

FreeOTP replicates the same general workflow used in Google Authenticator. The first step is to provision the device with the key necessary to generate the HOTP or TOTP passcodes. How this step is done varies depending on the service one is trying to connect to, but the simplest option (if available) is to photograph a QR code that is generated by the server; these QR codes encode the key as well as the metadata one needs to keep track of the account (e.g., the server and account username).

FreeOTP is not as visually polished, but it does score some points over Google Authenticator by including the ZXing QR code–reading library within the app; Google requires downloading and installing a ZXing app separately. For services that do not generate QR codes for key provisioning, FreeOTP allows the user to manually enter the key, the key's options (e.g., the hash algorithm used) and information for the issuing service.

Once provisioned, opening the app shows a list of the saved accounts, and opening each account reveals the current one-time pad passcode. TOTP, the time-based passcode variant, is designed so that the codes are only valid for a brief interval of time (say, 30 seconds). The FreeOTP interface displays a little clock animation next to each code showing how much time remains, and automatically updates to the next passcode after every elapsed interval.

In practice, TOTP seems to be considerably more widespread than HOTP, presumably because HOTP seeds each passcode-generation run with a counter, rather than the current time, which in turn makes the resulting passcode valid for an indefinite length of time. This is of concern for Google Authenticator and FreeOTP because passcodes generated on phones can be compromised—a passcode with an indefinite lifetime allows more attack vectors than one that expires (for example, by jotting down the code and leaving, instead of stealing the phone).

But that is not the whole story; using one's Android phone as if it is a hardware crypto-token is a bit of a fudge regardless of the passcode-generation technique. In reality, since a phone can easily be lost or stolen, the only protection against the attacker using it to take over one's HOTP/TOTP online accounts is securing the phone's contents itself: password-protection, encrypted storage, and so on. What percentage of phone users actually take advantage of these options is anyone's guess.

For those who are concerned about the security of the device itself, it is worth noting that neither the Google Authenticator nor FreeOTP apps are password-protected. In an email, McCallum did mention that encrypted key storage is on the to-do list, but that other items (such as getting the iOS app released) have taken precedence for now.

On the other hand, many (or perhaps most) security schemes can lead into a "the perfect is the enemy of the good" cycle of disagreement that has no simple answer. Usage of two-factor authentication, even if the second "factor" is a software-generated passcode on a phone that could get stolen, is an improvement over user passwords alone. Similarly, while FreeOTP does not offer an expanded feature set over Google Authenticator, the fact that it is open source and thus auditable makes it a better option in absolute terms.

The shift of Google Authenticator to a proprietary app was certainly a step backward for the security-conscious, so it is good to see an open-source alternative arrive with feature parity. Of course, the mobile apps only cover one half of the problem; there is more work that remains to be done implementing multi-factor authentication on applications and servers.

Comments (21 posted)

Security quotes of the week

Comments (1 posted)

almanah: does not encrypt its database

Comments (none posted)

augeas: world writable config files

Comments (none posted)

cups: information disclosure

Comments (none posted)

cups: cups.socket is listening on 0.0.0.0

Comments (none posted)

drupal7: multiple vulnerabilities

Comments (none posted)

drupal7-entity: multiple vulnerabilities

Comments (none posted)

ejabberd: information disclosure

Comments (none posted)

hplip: file overwrites

Comments (none posted)

java-1.7.0: multiple unspecified vulnerabilities

Comments (none posted)

libpng16: denial of service

Comments (none posted)

libvirt: denial of service

Comments (none posted)

libxslt: denial of service

Comments (none posted)

memcached: multiple vulnerabilities

Comments (none posted)

mysql: two unspecified vulnerabilities

Comments (none posted)

mysql: multiple unspecified vulnerabilities

Comments (none posted)

nagios: denial of service

Comments (none posted)

nss: information disclosure

Comments (none posted)

ntp: denial of service

Comments (none posted)

rubygem-will_paginate: cross-site scripting

Comments (none posted)

virtualbox: multiple unspecified vulnerabilities

Comments (none posted)

zabbix: SQL injection

Comments (none posted)

Kernel release status

Stable updates: there have been no stable updates in the last week, and none are in the review process as of this writing.

Comments (none posted)

Quotes of the week

Comments (none posted)

Kroah-Hartman: Kdbus Details

Comments (37 posted)

3.14 Merge window part 1

• The user-space lockdep patch set has been merged. This feature makes the kernel's locking debugging capabilities available to user-space applications.

• After years of development, the deadline scheduling class has finally been merged. This class allows processes to declare an amount of work needing to be done and a deadline by which it must happen; with care, it can guarantee that all processes will meet their deadlines. See this in-progress document for more information about the current status of the deadline scheduler.

• The m68k architecture has gained support for the kexec() system call.

• The kexec() system call now works on EFI BIOS systems.

• Xen is no longer supported on the ia64 architecture.

• The arm64 architecture has gained support for jump labels and the CMA memory allocator.

• The perf events subsystem has gained support for Intel "RAPL" energy consumption counters. The user-space perf tool has also gained a long list of new features and enhancements; see Ingo Molnar's pull request for a detailed list.

• The kernel address space layout randomization patches have been merged. Depending on who you believe, this feature may make the kernel more resistant to certain types of attacks. Note that, as of this writing, enabling this feature breaks hibernation and perf events.

• New hardware support includes:
  • Processors and systems: Intel Clovertrail and Merrifield MID platforms.

  • Audio: Tegra boards with MAX98090 codecs, Broadcom BCM2835 SoC I2S/PCM modules, Freescale SoC digital audio interfaces, Freescale enhanced serial audio interface controllers, and Analog Devices AXI-I2S and AXI-SPDIF softcore peripherals.

  • GPIO / pinmux: MOXA ART GPIO controllers, Xtensa GPIO32 GPIO controllers, SMSC SCH311x SuperI/O GPIO controllers, Freescale i.MX25 pin controllers, Qualcomm TLMM pin controllers, Qualcomm 8x74 pin controllers, NVIDIA Tegra124 pin controllers, Broadcom Capri pin controllers, and TI/National Semiconductor LP3943 GPIO expanders.

  • Miscellaneous: IBM Generic Work Queue Engine accelerators, Realtek RTS5208/5288 PCI-E card readers, Freescale MPL3115A2 pressure sensors, Freescale i.MX6 HDMI transmitters, DHT11 and DHT22 temperature/humidity sensors, HID 3D inclinometers, Capella Microsystem CM32181 ambient light sensors, Humusoft MF634 and MF624 data acquisition cards, Renesas RIIC I2C interfaces, TI/National Semiconductor LP3943 PWM controllers, ams AS3722 power-off units, and Maxim MAX14577 MUIC battery chargers.

  • USB: Maxim MAX14577 micro USB interface controllers, OMAP USB OTG controllers, Tahvo USB transceivers, Ingenic JZ4740 USB device controllers, Broadcom Kona USB2 PHYs, Aeroflex Gaisler GRUSBDC USB peripheral controllers, MOXA UPort serial hubs, and RobotFuzz Open Source InterFace USB to I2C controllers.

Changes visible to kernel developers include:

• The new smp_load_acquire() and smp_store_release() memory barrier operations have been added; see this article for information on when they are needed and how they can be used.

• The kernel can be built with the -fstack-protector-strong compiler option, available in GCC starting with version 4.9. This option allows more functions within the kernel to have stack overrun protection applied while still keeping the overhead to (hopefully) reasonable levels.

• preempt_enable_no_resched() is no longer available to modules which, according to the scheduler developers, should not "be creative with preemption".

• The internals of the sysfs virtual filesystem are being massively reworked to create a new "kernfs" that can serve as the base for a number of such filesystems. The first target is the cgroup control filesystem, but others may follow. This work is incomplete in 3.14, but has still resulted in a lot of internal code changes.

• The new documentation file Documentation/driver-model/design-patterns.txt tries to formalize some of the design patterns seen in driver code. It is just getting started; contributions would undoubtedly be welcome.

• There is a new "componentized subsystems" infrastructure for management of complex devices made up of a number of smaller, interacting devices; see this commit for details.

• The Android ION memory allocator has been merged into the staging tree. A long list of improvements, including changes to make ION use the dma-buf interface and the CMA allocator, has been merged as well.

If the normal rules apply, the 3.14 merge window can be expected to remain open until around February 2. At this point, a number of large trees — networking, for example — have not yet been pulled, so one can expect quite a bit more in the way of changes between now and when the window closes. As always, next week's Kernel Page will continue to follow what gets into the mainline for this development cycle.

Comments (1 posted)

Btrfs: Send/receive and ioctl()

Send and receive

The subvolumes and snapshots article described a rudimentary scheme for making incremental backups to a Btrfs filesystem. It was simple enough: use rsync to make the backup filesystem look like the original, then take a snapshot of the backup to preserve the state of affairs at that particular time. This approach is relatively efficient; rsync will only copy changes to the filesystem (passing over unchanged files), and each snapshot will preserve those changes without copying unchanged data on the backup volume. In this way, quite a bit of filesystem history can be kept in a readily accessible form.

There is another way to do incremental backups, though, if both the original and backup filesystems are Btrfs filesystems. In that case, the Btrfs send/receive mechanism can be used to optimize the process. One starts by taking a snapshot of the original filesystem:

    btrfs subvolume snapshot -r source-subvolume snapshot-name

The snapshot-name should probably include a timestamp, since the whole mechanism depends on the existence of regular snapshots at each incremental backup time. The initial snapshot is then copied to the backup drive with a command like:

    cd backup-filesystem
    btrfs send path-to-snapshot | btrfs receive .

This operation, which will copy the entire snapshot, can take quite a while, especially if the source filesystem is large. It is, indeed, significantly slower than just populating the destination filesystem with rsync or a pair of tar commands. It might work to use one of the latter methods to populate the backup filesystem initially, but using the send/receive chain ensures that things are set up the way those commands expect them to be.

Note that if the source snapshot is not read-only, btrfs send will refuse to work with it. There appears to be no btrfs command to set the read-only flag on an existing snapshot that was created as writable, but it is, of course, a simple matter to create a new read-only snapshot of an existing read/write snapshot, should the need arise.

Once the initial copy of the filesystem is in place, incremental backups can be done by taking a new snapshot on the source filesystem, then running a command like:

    cd backup-filesystem
    btrfs send -p path-to-previous-snapshot path-to-new-snapshot | btrfs receive .

With the -p flag, btrfs send will only send files (or portions thereof) that have changed since the previous-snapshot was taken; note that the previous snapshot needs to exist on the backup filesystem as well. Unlike the initial copy, an incremental send operation is quite fast — much faster than using a command like rsync to find and send changed files. It can thus be used as a low-impact incremental backup mechanism, possibly running many times each day.

Full use of this feature, naturally, is likely to require some scripting work. For example, it may not be desirable to keep every snapshot on the original filesystem, especially if space is tight there. But it is necessary to keep each snapshot long enough to use it for the next incremental send operation; using the starting snapshot would result in the unnecessary copying of a lot of data. Over time, one assumes, reasonably user-friendly tools will be developed to automate these tasks.

Btrfs ioctl() commands

Like most of the relatively complex Linux filesystems, Btrfs supports a number of filesystem-specific ioctl() commands. These commands are, as a rule, entirely undocumented; one must go to the (nearly comment-free) source to discover them and understand what they do. This article will not take the place of a proper document, but it will try to point out a few of the more interesting commands.

Most of the Btrfs-specific commands carry out operations that are available via the btrfs command-line tool. Thus, there are commands for the management of subvolumes and snapshots, devices, etc. For the most part, the btrfs tool is the best way to access this type of functionality, so those commands will not be covered here. It is amusing to note that several of these commands already come in multiple versions; the first version lacked a field (usually flags to modify the operation) that was added in the second version.

The structures and constants for all Btrfs ioctl() commands should be found in <linux/btrfs.h>; some distributions may require the installation of a development package to get that header.

• Cloning files. The Btrfs copy-on-write (COW) mechanism can be used to make copies of files that share the underlying storage, but which still behave like separate files. A file that has been "cloned" in this way behaves like a hard link as long as neither the original file nor the copy is modified; once a change is made, the COW mechanism copies the modified blocks, causing the two files to diverge. Cloning an entire file is a simple matter of calling:

      status = ioctl(dest, BTRFS_IOC_CLONE, src);
  

  Where dest and src are open file descriptors indicating the two files to operate on; dest must be opened for write access. Both files must be in the same Btrfs filesystem.

  To clone a portion of a file's contents, one starts with one of these structures:

      struct btrfs_ioctl_clone_range_args {
    	__s64 src_fd;
    	__u64 src_offset, src_length;
    	__u64 dest_offset;
      };
  

  The structure is then passed as the argument to the BTRFS_IOC_CLONE_RANGE ioctl() command:

      status = ioctl(dest, BTRFS_IOC_CLONE_RANGE, &args);
  

  As with BTRFS_IOC_CLONE, the destination file descriptor is passed as the first parameter to ioctl().

  Note that the clone functionality is also available in reasonably modern Linux systems using the --reflink option to the cp command.

• Explicit flushing. As with any other filesystem, Btrfs will flush dirty data to permanent storage in response to the fsync() or fdatasync() system calls. It is also possible to start a synchronization operation explicitly with:

      u64 transid;
  
      status = ioctl(fd, BTRFS_IOC_START_SYNC, &transid);
  

  This call will start flushing data on the filesystem containing fd, but will not wait for that operation to complete. The optional transid argument will be set to an internal transaction ID corresponding to the requested flush operation. Should the need arise to wait until the flush is complete, that can be done with:

      status = ioctl(fd, BTRFS_IOC_WAIT_SYNC, &transid);
  

  The transid should be the value returned from the BTRFS_IOC_START_SYNC call. If transid is a null pointer, the call will block until the current transaction, whatever it is, completes.

• Transaction control. The flush operations can be used by an application that wants to ensure that one transaction completes before starting something new. Programmers who want to live dangerously, though, can use the BTRFS_IOC_TRANS_START and BTRFS_IOC_TRANS_END commands (which take no arguments) to explicitly begin and end transactions within the filesystem. All filesystem operations made between the two calls will become visible to other processes in an atomic manner; partially completed transactions will not be seen.

  The transaction feature seems useful, but one should heed well this comment from fs/btrfs/ioctl.c:

      /*
       * there are many ways the trans_start and trans_end ioctls can lead
       * to deadlocks.  They should only be used by applications that
       * basically own the machine, and have a very in depth understanding
       * of all the possible deadlocks and enospc problems.
       */
  

  Most application developers, one might imagine, lack this "very in depth understanding" of how things can go wrong within Btrfs. Additionally, there seems to be no way to abort a transaction; so, for example, if an application crashes in the middle of a transaction, the transaction will be ended by the kernel and the work done up to the crash will become visible in the filesystem. So, for most developers considering using this functionality, the right answer at this point is almost certainly "don't do that." Anybody who wants to try anyway will need the CAP_SYS_ADMIN capability to do so.

There are quite a few more ioctl() commands supported by Btrfs, but, as mentioned above, most of them are probably more conveniently accessed by way of the btrfs tool. For the curious, the available commands can be found at the bottom of fs/btrfs/ioctl.c in the kernel source tree.

Series conclusion

At this point, LWN's series on the Btrfs filesystem concludes. The major functionality offered by this filesystem, including device management, subvolumes, snapshots, send/receive and more, has been covered in the five articles that make up this set. While several developers have ideas for other interesting features to add to the filesystem, chances are that most of that feature work will not go into the mainline kernel anytime soon; the focus, at this point, is on the creation of a stable and high-performance filesystem.

There are few knowledgeable developers who would claim that Btrfs is fully ready for production work at this time, so that stabilization and performance work is likely to go on for a while. That said, increasing numbers of users are putting Btrfs to work on at least a trial basis, and things are getting more solid. Predictions of this type are always hard to make successfully, but it seems that, within a year or two, Btrfs will be accepted as a production-quality filesystem for an increasingly wide range of use cases.

Comments (4 posted)

Namespaces in operation, part 7: Network namespaces

It's been a while since last we looked at Linux namespaces. Our series has been missing a piece that we are finally filling in: network namespaces. As the name would imply, network namespaces partition the use of the network—devices, addresses, ports, routes, firewall rules, etc.—into separate boxes, essentially virtualizing the network within a single running kernel instance. Network namespaces entered the kernel in 2.6.24, almost exactly five years ago; it took something approaching a year before they were ready for prime time. Since then, they seem to have been largely ignored by many developers.

Basic network namespace management

As with the others, network namespaces are created by passing a flag to the clone() system call: CLONE_NEWNET. From the command line, though, it is convenient to use the ip networking configuration tool to set up and work with network namespaces. For example:

    # ip netns add netns1

This command creates a new network namespace called netns1. When the ip tool creates a network namespace, it will create a bind mount for it under /var/run/netns; that allows the namespace to persist even when no processes are running within it and facilitates the manipulation of the namespace itself. Since network namespaces typically require a fair amount of configuration before they are ready for use, this feature will be appreciated by system administrators.

The "ip netns exec" command can be used to run network management commands within the namespace:

    # ip netns exec netns1 ip link list
    1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT 
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

This command lists the interfaces visible inside the namespace. A network namespace can be removed with:

    # ip netns delete netns1

This command removes the bind mount referring to the given network namespace. The namespace itself, however, will persist for as long as any processes are running within it.

Network namespace configuration

New network namespaces will have a loopback device but no other network devices. Aside from the loopback device, each network device (physical or virtual interfaces, bridges, etc.) can only be present in a single network namespace. In addition, physical devices (those connected to real hardware) cannot be assigned to namespaces other than the root. Instead, virtual network devices (e.g. virtual ethernet or veth) can be created and assigned to a namespace. These virtual devices allow processes inside the namespace to communicate over the network; it is the configuration, routing, and so on that determine who they can communicate with.

When first created, the lo loopback device in the new namespace is down, so even a loopback ping will fail:

    # ip netns exec netns1 ping 127.0.0.1
    connect: Network is unreachable

    # ip netns exec netns1 ip link set dev lo up
    # ip netns exec netns1 ping 127.0.0.1
    PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
    64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.051 ms
    ...

    # ip link add veth0 type veth peer name veth1
    # ip link set veth1 netns netns1

    # ip netns exec netns1 ifconfig veth1 10.1.1.1/24 up
    # ifconfig veth0 10.1.1.2/24 up

    # ping 10.1.1.1
    PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
    64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.087 ms
    ...
    
    # ip netns exec netns1 ping 10.1.1.2
    PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data.
    64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=0.054 ms
    ...

As mentioned, though, namespaces do not share routing tables or firewall rules, as running route and iptables -L in netns1 will attest.

    # ip netns exec netns1 route
    # ip netns exec netns1 iptables -L

Non-root processes that are assigned to a namespace (via clone(), unshare(), or setns()) only have access to the networking devices and configuration that have been set up in that namespace—root can add new devices and configure them, of course. Using the ip netns sub-command, there are two ways to address a network namespace: by its name, like netns1, or by the process ID of a process in that namespace. Since init generally lives in the root namespace, one could use a command like:

    # ip link set vethX netns 1

Uses for network namespaces

As we have seen, a namespace's networking can range from none at all (or just loopback) to full access to the system's networking capabilities. That leads to a number of different use cases for network namespaces.

By essentially turning off the network inside a namespace, administrators can ensure that processes running there will be unable to make connections outside of the namespace. Even if a process is compromised through some kind of security vulnerability, it will be unable to perform actions like joining a botnet or sending spam.

Even processes that handle network traffic (a web server worker process or web browser rendering process for example) can be placed into a restricted namespace. Once a connection is established by or to the remote endpoint, the file descriptor for that connection could be handled by a child process that is placed in a new network namespace created by a clone() call. The child would inherit its parent's file descriptors, thus have access to the connected descriptor. Another possibility would be for the parent to send the connected file descriptor to a process in a restricted network namespace via a Unix socket. In either case, the lack of suitable network devices in the namespace would make it impossible for the child or worker process to make additional network connections.

Namespaces could also be used to test complicated or intricate networking configurations all on a single box. Running sensitive services in more locked-down, firewall-restricted namespace is another. Obviously, container implementations also use network namespaces to give each container its own view of the network, untrammeled by processes outside of the container. And so on.

Namespaces in general provide a way to partition system resources and to isolate groups of processes from each other's resources. Network namespaces are more of the same, but since networking is a sensitive area for security flaws, providing network isolation of various sorts is particularly valuable. Of course, using multiple namespace types together can provide even more isolation for both security and other needs.

Comments (31 posted)

OSTree for Fedora

Linux distributions are typically delivered either in piecemeal fashion via packages or as a single bundle like Android or Chrome OS. Colin Walters's OSTree project is taking something of a middle course. It is delivering a kind of bundle, but with a twist: the tool makes it easy to switch between multiple independent operating systems that are all simultaneously installed on the system.

We first looked at OSTree as part of our coverage of GUADEC 2012. Since that talk, though, Walters has not been idle. His gnome-continuous project has been building OSTree versions of GNOME using continuous integration for some time. Beyond that, he recently announced an OSTree repository for Fedora: fedostree:

Rather than break things up into packages, OSTree builds fully assembled, versioned filesystem trees that hold what would have gone into a set of package (e.g. RPM) files. All of the binaries, libraries, documentation files, and so on are gathered together into the OSTree repository and can be retrieved using the ostree command. The fedostree installation instructions make it clear that the model Walters used was Git; many of the commands are clearly patterned after Git's. In fact, the project's home page calls it "git for operating system binaries".

For Fedora, one can install multiple different kinds of trees, based on either Fedora 20 or Rawhide, and switch between them at boot time. Each of the trees will have a focus on a particular project's output, such as GNOME, Docker, FreeIPA, or others. Those can be combined with a tree that provides the minimal F20 or Rawhide base to produce a system that can boot and run the project's output, and can do so repeatedly as each new version gets built from newer upstream code. Developers on those projects will then have a tool to quickly and easily switch between multiple versions of the project. That can make bisecting problems much easier, for example. Beyond the project documentation and Walters's announcement, he also describes some of the details of the tree switching in a blog post.

Each of the OSTree-installed systems will share the /etc directory; changes will be merged with those made by the other OSTrees. Also, /var will be shared between the instances. Beyond that, OSTree does quite a bit of filesystem rearranging to enable it to switch between the different trees. In the blog post, Walters gives some details:

# ostree admin switch fedostree/20/x86_64/server/docker-io

# ostree admin switch fedostree/20/workstation/gnome/core

In addition to bisecting problems in broken umbrella projects (like GNOME or Mesa) that may not be easy to find otherwise, there are a number of other possible uses that Walters lists on the fedostree page. Installing RHEL and Fedora in parallel, or Fedora and Rawhide, would be one possibility. That would make switching between them far easier, while still keeping each up to date. It would also make testing Rawhide much safer since the only real risk is a filesystem-corrupting kernel bug. Making an Android-like recovery partition with a minimal install is another idea.

As he notes in the announcement (and elsewhere), fedostree (and OSTree itself) are rather "bleeding edge" at this point. He is looking for brave souls to evaluate the ideas, tools, and repositories to help him find problems. There is also a list of "development areas" that need to be addressed in the near future.

OSTree/fedostree is clearly a project that is just getting off the ground. OSTree has proven itself useful for GNOME continuous integration, so Walters plans to bring that to fedostree; trees would get rebuilt (and booted as a smoketest) immediately after a change is made that affects any package relevant to that tree. Before much of that can happen, though, some testing and shaking things out is required. It should be interesting to see where things go from here.

Comments (9 posted)

Distribution quotes of the week

Comments (none posted)

FreeBSD 10.0

Comments (16 posted)

GNU Linux-libre 3.13-gnu

Full Story (comments: none)

bits from the DPL -- December 2013

Full Story (comments: none)

Fedora Workstation proposal: ease installation of non-free software

Full Story (comments: 51)

Distribution newsletters

• DistroWatch Weekly, Issue 542 (January 20)
• Ubuntu Weekly Newsletter, Issue 351 (January 19)

Comments (none posted)

CentOS Project Leader Karanbir Singh Opens Up on Red Hat Deal (Linux.com)

Comments (100 posted)

Of bytes and encoded strings

String handling is quite different between Python 2 and Python 3, largely because the latter is designed to handle Unicode natively. But those differences make it more difficult for Python-2-based projects to make the move to Python 3. A proposal to add a feature to the language that would make that transition easier has led to a series of massive threads on the python-dev mailing list. A seemingly simple idea has interesting ramifications, both technical and philosophical.

A bit of an introduction to Python string types is probably in order. Both Python 2 and Python 3 have two types that hold string-like data, but they are quite different. Python 2 has str and unicode. Its str contains 8-bit data that could be binary data or encoded text data in some unknown encoding. Typically, it is hoped that the unknown encoding will be ASCII compatible (i.e. shares the same mappings as 7-bit ASCII), but that is not guaranteed. On the other hand, unicode strings (e.g. u'foo') contain a series of Unicode code points.

For Python 3, though, the str type contains a sequence of Unicode code points, while the bytes type (e.g. b'foo') contains a sequence of integer values in the range 0–255, without any further interpretation of those values. The language removes the ambiguous string type in favor of clear boundaries between Unicode strings and binary data (which might, in fact, be encoded text). Also, unlike Python 2, the two string types are not interoperable. unicode + bytes is not defined and will raise an exception. So, in order to get external data (from a file or socket, say) into a Python 3 str, it must be decoded from a known encoding (e.g. "ascii", "utf-8", etc.), which is quite different from the Python 2 behavior. This is all covered in much more detail in various places including Nick Coghlan's Python 3 Q&A document, this section in particular.

Various projects that are looking into moving to Python 3 have stumbled over a particular problem: formatting (or interpolating) values into bytes arrays. The following is illegal in Python 3 today:

    b'%d' % 42

    b'%s' % 'foo'
    b'%s' % b'foo'

PEP 460

Both the Mercurial and Twisted projects have requested support for certain kinds of bytes formatting to help in their Python 3 porting efforts. That is what led to an RFC post from Victor Stinner that contained a draft of Python Enhancement Proposal (PEP) 460. The proposal would add string-like interpolation to the Python 3 bytes type. Both the "%" operator and the format() method would be supported for a wide range of formatting characters. In Stinner's original proposal, "%s" would only be supported for numeric or bytes arguments. It was targeted as a change for Python 3.5, which is due sometime late in 2015.

There were multiple folks commenting on the idea in the thread, both for and against the idea. For example, Mark Shannon thinks that adding formatting for bytes strings, particularly when converting things like numbers to their ASCII representation, violates "the separation of str and bytes in the first place". On the other hand, Marc-Andre Lemburg agrees with Stinner's proposal as it makes things much easier in "situations where you have to work on data which is encoded text using multiple (sometimes even unknown) encodings in a single data chunk. Think MIME messages, mbox files, diffs, etc."

With Stinner's approval, Antoine Pitrou reworked PEP 460 to restrict the feature to a small subset of the formatting codes ("%s" for bytes and "%c" for single integers in the 0–255 range). The rework also added support for the bytearray type (mutable bytes). It got an immediate +1 from Coghlan, who was "initially dubious about the idea". Others were not happy to see the removal of numeric formatting. As Ethan Furman pointed out, allowing ASCII (in the form of numeric formatting) to be interpolated into the bytes type does not mean that the type is ASCII compatible:

At some level, it comes down to a question of "language purity" and how far to push the separation between str and bytes. In a fairly long message, Stephen Hansen described what currently needs to be done for HTTP's Content-Length header:

    headers.append((b"Content-Length": ("%d" % (len(content))).encode("ascii")))

But Coghlan is tired of seeing repeated arguments that ultimately lead to what he sees as a dangerous blurring of the line between str and bytes. One possible solution that he sees is in the nascent asciicompat project, which would add a new type (first as an extension, then perhaps to the language core) that had some of the properties being advocated in the thread.

PEP 461

The discussion in that thread (and others) got more heated than most in python-dev. Eventually, Furman forked the PEP into PEP 461, adding back the numeric formatting. Brett Cannon characterized the split as one that is between people intent on "explicit is better than implicit" (PEP 460) versus those who are focused on "practicality beats purity" (PEP 461). Both of those statements come from The Zen of Python, so it comes down to a question of the priority each person puts on them.

As it turns out, though, Python benevolent dictator for life (BDFL) Guido van Rossum is clearly in the "practicality beats purity" camp. He noted that there already is some inherent ASCII bias in the bytes type. Others had earlier pointed out some similar examples, but Van Rossum succinctly summarized the case:

That summary seemed to cut through a lot of the argument (Van Rossum's BDFL role plays a part in that, of course). The only real sticking point seems to be what should be returned for constructs like:

    b'%s' % 'x'

Van Rossum thinks it should work as follows:

Not everyone is happy with that, but it is in keeping with an existing behavior, he said:

While that symmetry had some appeal, it was seen as unnecessary and potentially a source of hard-to-track-down bugs. So, after another lengthy discussion, most seemed to settle for a restricted version of the "%s" specifier: it will only accept arguments that provide the buffer protocol or has a __bytes__() method, otherwise a TypeError will be raised. In practice, that will restrict %s to some internal CPython array types and to bytes; str and numeric types do not have __bytes__() methods.

In several followup threads, Furman revised PEP 461 into something that is getting close to a final version. Along the way, it lost support for the format() method, so only support for the __mod__() method ("%" operator) would be added. The current version of PEP 461 is still missing a "Rationale" section, and there have been some minor quibbles with the wording and such, but overall it is nearly complete. That is not to say that everyone agrees with the PEP, but it would appear that Van Rossum does, which makes it seem likely that we will see it in Python 3 before too long.

Comments (78 posted)

Quotes of the week

Comments (3 posted)

Rivendell 2.7.0 released

Version 2.7.0 of the Rivendell radio automation system has been released. This version adds support for AIFF audio import, optimized CD ripping, and the ability to customize log message formatting through templates, among other changes.

Full Story (comments: none)

Libraw 0.16 available

Version 0.16 of the Libraw library for raw photo file manipulation has been released. This version adds support for many more camera models, as well as two new demosaicing algorithms and improved Exif tag parsing.

Comments (none posted)

LabPlot 2.0.0 released

Version 2.0.0 of the LabPlot data analysis and visualization application for KDE has been released. Denoted the "first stable version", the new release is a complete rewrite of the LabPlot 1.x codebase, and supports generating 2D plots in a wide array of output formats, including LaTeX.

Comments (none posted)

Development newsletters from the past week

• What's cooking in git.git (January 16)
• What's cooking in git.git (January 22)
• LLVM Weekly (January 20)
• OCaml Weekly News (January 21)
• OpenStack Community Weekly Newsletter (January 17)
• Perl Weekly (January 20)
• PostgreSQL Weekly News (January 20)
• Ruby Weekly (January 16)
• This Week in Rust (January 18)
• Tor Weekly News (January 22)

Comments (none posted)

Pocock: Debian.org enabled for SIP federation, who will be next?

At his blog, Daniel Pocock notes Debian's new SIP federation service, which provides voice-over-IP (VoIP) service to approximately 1000 Debian Developers. "This means packages providing SIP (and soon XMPP/Jabber) will hopefully be under much more scrutiny from developers in the lead up to the next major Debian release", he observes, although the broader benefit might be that other free software community members will start to take federated SIP support more seriously.

Comments (none posted)

Happy 25th Tcl! (TkDocs)

Comments (1 posted)

LinuxGizmos: Ultra-secure “Blackphone” to run on Android-based “PrivatOS”

LinuxGizmos is reporting on the Blackphone, a just-announced "ultra-secure" mobile phone to be jointly produced by Geeksphone (who released the Firefox OS developer phones) and security-conscious email vendor Silent Circle. The OS is advertised as "PrivatOS," a security-tuned derivative of Android. The phone is expected to be available in late February, in time to debut at Barcelona's Mobile World Congress. The Blackphone site calls it "the world's first smartphone which prioritizes the user's privacy and control, without any hooks to carriers or vendors", although as LinuxGizmos points out, few other details—including hardware details—are available yet.

Comments (24 posted)

CFP Deadlines: January 23, 2014 to March 24, 2014

If the CFP deadline for your event does not appear here, please tell us about it.

SCALE 12X: Event update

Full Story (comments: none)

The Linux Foundation's 2014 event schedule

Comments (none posted)

Events: January 23, 2014 to March 24, 2014

If your event does not appear here, please tell us about it.