LWN.net Weekly Edition for March 13, 2014

The Linux Foundation invests in MOOCs

In addition to hosting events, shepherding software projects, and supporting kernel developers, the Linux Foundation (LF) has long offered training courses of various kinds—for developers, system administrators, and even managers. These courses are generally paid, "professional-development"–style training, but the LF recently announced that it would open up its introductory Linux course as a free online offering, built in conjunction with one of the leading sources for massive open online courses (MOOCs).

The announcement was made on March 6, noting the LF's partnership with edX, an online-courseware organization whose site hosts MOOCs for several major universities. As we saw in September 2013, edX has released the source code to its hosting platform and now maintains it as an active open-source project. Many of the universities who offer classes through edX contribute developer time (and even components), as does Google, who re-tasked the team from its own MOOC platform Course Builder, to edX development.

Although the edX platform can be self hosted, the LF course in question, Introduction to Linux, will be offered through the main edX site. The course is already available for registration, though it will not come online until the third quarter of 2014. As for the content of the course, it comes from the systems administration section of LF's course catalog, and the current page about the paid, in-person variant of the course describes it as teaching "a good working knowledge of Linux, from both a graphical and command line perspective, allowing you to easily navigate through any of the three major Linux distributions." As LF's Jerry Cooperstein explained in a comment to our earlier blurb about the announcement, the "three major distributions" refer to the Fedora/CentOS/RHEL family, the Debian/Ubuntu/Mint family, and the SUSE/openSUSE family.

An outline offers a few more specific topics, but it is quite clearly a course meant for those with zero prior experience working with a Linux distribution. The paid version of the course costs $2400 and takes four days. When the edX version launches, it will be a work-at-your-own-pace endeavor.

For those people already familiar with Linux, there is probably little to see in Introduction to Linux. The more important aspect of the announcement is the fact that the LF has joined edX as a Member Institution. That suggests that LF will likely make more and more of its training courses available in MOOC form, which will open them up to far more participants. As it stands now, LF training courses are generally offered as on-site corporate offerings, co-located at LF events, or as periodic training events held in big cities. There may multiple opportunities for an interested individual to get to one of the in-person classes, but the availability and self-pacing of a MOOC allow considerably more flexibility.

In a March 11 blog post, LF's Amanda McPherson linked the decision to offer the course for free at edX to the current Linux talent shortage, and noted that more than 38,000 people had already registered for the course in advance. Considering that level of interest, reaching the wider pool of potential students seems to be the major benefit of the MOOC approach.

Furthermore, while the introductory course is understandably light in its subject matter, the other offerings already offered as LF training do include some pretty in-depth topics, like Optimizing Linux Device Drivers for Power Efficiency, and cover some specific software projects like Yocto, Tizen, and OpenStack that are distinct enough to warrant special attention.

The other wrinkle, of course, is the training classes are one of the ways that the LF raises funds. The edX site itself offers only free classes, so if the LF intends to offer more of its own content in MOOC form, it will probably do so at its own site. Some of its existing courses are offered in "virtual" form, but this means a combination of whiteboard software and live audio conferencing to connect to a classroom where an instructor is lecturing.

The MOOC format has proven itself to be a far more popular way to work through course materials, with the likes of Stanford, MIT, and Harvard (among many others) offering MOOC classes. So it is a wise move for the LF to embrace it for technical training; by partnering with the leading open-source MOOC project, it can do so while also supporting the development of an important software framework.

Comments (2 posted)

What's new with HTTP2

Changing any widely used network protocol is a risky affair—more so with a protocol as ubiquitous as HTTP. But the venerable web transport protocol is expected to see its first update of the century later this year when HTTP2 is slated for release. The new revision can be hard to characterize, since it changes the way HTTP is sent over the wire, but it does not alter the existing HTTP semantics. In essence, the revisions are designed to speed up how HTTP traffic is relayed between clients, servers, proxies, and switches, but to keep intact the format and meaning of HTTP requests and responses. The upshot will be faster throughput, even if few people outside of network operators will see the actual changes to the protocol.

HTTP has not been updated since 1999, when the Internet Engineering Task Force (IETF) published version 1.1 in RFC 2616. Version 1.1, notably, added support for persistent, pipelined TCP connections between clients and servers. Thus, it was no longer necessary to open and close a separate TCP connection for each HTTP request, and clients could send multiple requests without waiting for each request's response before sending the next.

Those changes did a lot to reduce latency in HTTP, and the concepts implemented arose from the real-world problems experienced with HTTP 1.0. The HTTP2 situation is similar; the vendors that make web software and networking hardware—as well as large-scale web service providers—identified a new set of pain points as the World Wide Web continued to evolve. Several players (including Google and Microsoft) started their own research projects to develop improvements, and by 2007 there was enough interest for the IETF to charter a new working group to address a new revision.

The working group is called the HTTP Bis group—where Bis is a perhaps uncommon designation meaning "twice" that is intended to communicate that the group's charter is not to invent new methods or headers, but just to improve on existing HTTP functionality. The notion that HTTP2 is semantically compatible with HTTP 1.1 is clearly important to the project. The group has recently taken to calling revised protocol HTTP2 or HTTP/2, rather than "HTTP 2.0," just to make such a distinction more prominent, although there is not yet universal agreement on the point.

Drafting

The most recent HTTP2 draft is version 10, which was published on February 13. The first draft was based directly on Google's SPDY (which we looked at in 2009), although as one would expect, it has evolved in a number of ways since then. The basic approach, however, remains the same.

HTTP2 is backward compatible with all of HTTP 1.1's request methods, responses, and headers. But it attempts to decrease the overall latency of serving a page to a client by multiplexing requests, compressing HTTP headers, and allowing the server to prioritize some requests over others.

Multiplexing requests is in many ways an extension to the pipelining feature of HTTP 1.1. Pipelining allows the client to send request #2 before request #1 has been fulfilled by the server, but it still has a weakness: request #2 is still processed by the server after request #1. So if request #1 takes a long time, it still introduces latency. This is called "head-of-line blocking." HTTP 1.1 is also hampered by the requirement that clients not open more than two simultaneous connections to the same server. HTTP2 does away with that limit, and averts head-of-line blocking by multiplexing several HTTP request connections over the same underlying TCP connection (testing of current implementations seems to indicate somewhere between four to eight streams as the best number on average). Since these multiplexed HTTP streams are independent of each other, it is far less likely for slow requests to block the delivery of the entire page.

In addition to making more efficient use of bandwidth, HTTP2's multiplexing will alleviate the need to do several other popular workarounds, such as resource sharding (i.e., splitting one image into several pieces each served from a different server) and the inlining of CSS and JavaScript. Both of these techniques have their costs; sharding introduces multiple DNS lookups for the various servers (and then opening lots of separate HTTP connections to them), and inline CSS and JavaScript cannot be cached by the browser. So the full benefits of HTTP2 multiplexing should exceed the raw gains seen just from reducing head-of-line blocking.

HTTP2 also compresses the HTTP headers, which typically contain a lot of redundancy—particularly between related requests for the same page. The earliest implementations used gzip for compression, and built a shared Huffman code dictionary for all of the requests on a single connection. But that scheme was found to be vulnerable to the CRIME (Compression Ratio Info-leak Made Easy) attack, in which an attacker can discover the bytes of a session cookie by injecting test bytes into HTTP requests—if the compressed result is the same length as the unmodified request, then the attacker knows the test byte is indeed part of the cookie. The HTTP Bis group has subsequently moved to a new compression scheme called HPACK.

There is little opposition to header compression as an idea, but the same cannot be said of another HTTP2 optimization: binary encoding. As befits a compressed format, HTTP2's frame format is binary, both for the header and the payload. This decision has its critics, such as John Graham-Cumming, who said that he wondered "whether it was really necessary to add a complex, binary protocol to the HTTP suite". But, in response, Mozilla's Patrick McManus commented that the orderly message boundaries of the binary format make it considerably more straightforward—and, therefore, faster—to parse than HTTP 1.1's ASCII. Of course, it could also be argued that the binary format is not human-readable, but the counter-argument is that it is merely the binary encoding of existing ASCII request/response semantics, rather than something new.

HTTP2 adds several other mechanisms that can speed up the process of fulfilling requests, but which (unlike compression and multiplexing) require some deployment-time tweaks to produce optimized results. For instance, servers can prioritize one request stream over another. The assumption is that some content (such as page text) is more critical to deliver to the client than others (such as CSS or decorative images), but, naturally, the decision of which resources to prioritize and which to leave untouched cannot be solved in the abstract. Similarly, HTTP2 allows servers to effectively "push" some resources, initiating the transfer before the client makes the actual request. The most effective implementation might begin sending linked-in CSS or image resources after the main page is requested, but the protocol itself does not dictate how servers decide what to push.

100 CONTINUE

Work continues on HTTP2, and the final draft is not expected until mid-2014. Most recently, discussion has included whether (and how) the new protocol should deal with encryption. At one time or another, dropping support for unencrypted http:// URIs was considered, as was dropping HTTPS as a separate protocol—in favor of requiring that HTTP2 run over TLS. For now, no such large steps have been taken. But the group has introduced HTTP Alternate Services (ALTSVC), a new HTTP header through which a server can move a URI request to a different host, port, or even protocol, without tearing down the connection and starting a new one.

This technique could be used to "upgrade" an http:// URI request from HTTP 1.1 to HTTP2, and thus move the connection to TLS. ALTSVC is a comparatively recent draft, which is still subject to change. Among other things, upgrading a connection to TLS relies on the presence of some specific TLS extensions, such as Server Name Indication (SNI), so while the idea of "opportunistic encryption" sounds appealing to many, there may be a fair amount of tire-kicking left before it is finalized.

The working group maintains a list of issues on GitHub and a publicly archived mailing list at the W3C site, which makes it easy to follow the development of the protocol. Just as importantly, the group tracks working implementations of HTTP2. Firefox and Chromium already support HTTP2, and Google, Twitter, and a few other large-scale application services support it on the server side. Since many of the benefits of the new protocol rely on implementations testing and getting the optimization details right, this is welcome news, and keeping in line with the IETF mantra of "rough consensus and working code." End users, of course, will notice only reduced page load times if they notice anything at all—but perhaps that is a good goal for a protocol update to set.

Comments (23 posted)

A false midnight

Interpreted, "duck typing" languages often have some idiosyncrasies in their definitions of "truth" and Python is no exception. But Python goes a bit further than some other languages in interpreting the True or False status of non-Boolean values. Even so, it often comes as a big surprise for programmers to find (sometimes by way of a hard-to-reproduce bug) that, unlike any other time value, midnight (i.e. datetime.time(0,0,0)) is False. A long discussion on the python-ideas mailing list shows that, while surprising, that behavior is desirable—at least in some quarters.

Python's notion of truth values defines False as various kinds of numeric zero (0, 0L, 0.0, 0j), "empty" objects ('', (), [], {}), the special value None, and, of course, False. User-defined classes can also have a False value, but many will not, which means that all objects of those classes are True. However, buried in the datetime module documentation for a time object is the following somewhat complicated explanation:

It all boils down to the idea that a time of 00:00 UTC (which could be something else entirely in the local time zone, of course) is False. If one were testing a variable meant to hold a time value as follows:

    if current:
        do_something(current)  # we have a valid current time

    if current is not None:
        do_something(current)  # we have a valid current time

The datetime module in the standard library is a fairly simple way to represent dates, times, and timestamps (using the datetime class). The date class does not have a time associated with it, while the time class (with resolution in seconds) is undated. Oddly, a time can have a time zone associated with it (even though time zones are often date-related). A datetime has both time and date (and can include a time zone as well).

A Python bug was opened in 2012 about midnight as False, but it was closed soon thereafter as "invalid". The time class was documented to work that way and someone had created a __nonzero__() (now __bool__()) method for time to implement that behavior. But, the behavior still didn't sit well with some. Nine months after the bug was closed, Danilo Bargen reported that the False interpretation caused problems with filters in Django templates. He asked that the bug be reopened, but nothing happened—and that's where things stood until recently.

That's when Shai Berger asked again for the bug to be reopened. That request was seconded by Andreas Pelme before moving to python-ideas at the request of Alexander Belopolsky. Berger posted a summary of the issue and asked for a reconsideration. That led to a large thread, even by python-ideas standards.

Berger's argument was twofold. First, paraphrasing Bargen and Pelme, that the current behavior is "surprising and useless" since users generally don't want special behavior at midnight. Second, his personal argument is that midnight "is not a 'zero value', it is just a value". He went on to say that midnight as False made as much sense as datetime.date(1,1,1) (the minimum date value) evaluating to False (and it doesn't).

While Paul Moore was unhappy with the practice of using Boolean tests for date validity (as it should be "is not None" in his mind), he did think the current behavior was wrong. Oscar Benjamin went further, noting that if this was new code being added, returning False for midnight in a Boolean context would never pass muster. "The question is surely whether the issue is worth a backwards compatibility break not whether the current behaviour is a good idea (it clearly isn't)."

On the other hand, Skip Montanaro is convinced that the problem is all in the application code and that the bug should remain closed. There weren't too many voices joining in with that sentiment, though Tim Peters and Belopolsky, who were both involved in the creation of the datetime module, were opposed to making any change. But Nick Coghlan posted an analysis of the current behavior that made it clear to many that it was completely inconsistent and not really tenable:

Coghlan had already reopened the bug, suggesting that anyone interested in seeing it fixed could post a patch to do so. The only question was how to handle deprecating the existing behavior. Python 3.4 is feature-frozen, so it is out of the picture; any change would come in 3.5 and subsequent releases. Coghlan's suggestion was to issue a DeprecationWarning whenever a time returns False in a Boolean context for 3.5, then to change the behavior so that all time values are True for 3.6. That means the change won't actually be released for roughly three years, which is too long for some.

One of those who might be in that "change it already" camp is BDFL Guido van Rossum. He posted an unequivocal statement that False time values were a mistake made a long time ago:

Furthermore, Van Rossum said, he would be inclined to fix the problem "immediately" (for Python 3.5) but would, perhaps, like to see a search for legitimate uses of the feature. Both Peters and Belopolsky noted that they had used the False midnight in programs, but nothing that was widely distributed; no other examples were mentioned in the thread. Meanwhile, multiple reports of problems stemming from the feature did come up.

While there is a danger that some legitimate programs might run afoul of the change, Van Rossum's intuition is probably right: there are likely few instances of real False midnight users, while the unexpected False problem is likely to be fairly common, with many if var: instances going undetected (or at least undiagnosed) for years. Backward compatibility is always a balancing act, but in this case it would seem that making a clean break from the existing behavior—and relatively soon—is probably the right approach.

Comments (82 posted)