LWN.net Weekly Edition for February 13, 2014

Debian decides on systemd—for now

Technical Committee (TC) chair Bdale Garbee has made his casting vote in favor of systemd as the default init system for Debian "jessie" (8.0). That would seem to put an end to a contentious whipsaw of multiple calls for votes (CFVs) on competing proposals, CFVs ending because the wording was sub-optimal, and other bits of drama, at least for the TC. It is hard to imagine that we have seen the end of this controversy for Debian as a whole, but it is a concrete step toward resolution. There are a number of developments to report since we last looked at the tussle back at the end of January.

Another CFV

On February 5, Ian Jackson called for a vote on his ballot that included language about "loose" vs. "tight" coupling for whatever default init system was chosen. Essentially, "loose coupling" means that no other package can depend on a particular init system running as PID 1, while "tight coupling" would allow such dependencies. Jackson is concerned about GNOME and other packages that depend on systemd for "proper" functioning and would like to see Debian reject that approach. His ballot would also choose the default init system; the various combinations made for ten separate options (each of the four init systems with each coupling possibility, plus the general resolution and further discussion options).

Jackson's CFV came about after several revisions on the tech-ctte mailing list, as well as an IRC meeting to discuss wording. But he eventually withdrew support for the proposal by voting for further discussion (FD) first, after complaints from Steve Langasek and Debian project secretary Kurt Roeckx. Both of them were concerned about wording issues, so Jackson rolled up his sleeves and went back to work.

Another IRC meeting was scheduled and Jackson posted a timetable for how he thought the voting process should proceed. He also made it abundantly clear that any CFV posted without allowing him to amend it to tie it to the coupling question would be, in his eyes, "a clear breach of process".

YA CFV

But posting a CFV is exactly what Garbee did on February 8. From that message, it is clear that Garbee saw no value in complicating the ballot by tying it to the coupling question:

While it was confrontational to call for a vote (and thus preclude amendments) in the face of Jackson's stated plans and warning, it's not clear what else Garbee could have done. There was obviously no consensus on the committee about the contents of the ballot and Jackson made it quite clear that he would offer amendments to a minimal ballot if offered the chance, so Garbee had to circumvent that to put a minimal question before the TC. As TC member Russ Allbery put it:

Garbee acknowledged that the members could override his choice by voting FD, and that Jackson almost certainly would, "but I sincerely hope the rest of you will not do that and instead vote this ballot to a useful conclusion". The question was what the default init system for Debian Linux (and not the Debian kFreeBSD or Hurd ports) should be for the jessie release. It also included language that would allow the project to override the TC's choice via a general resolution (GR) with a simple majority vote (rather than the usual 2:1 majority required for overriding the TC).

Decision

The vote came down much as expected, with a 4:4 split between systemd and Upstart proponents. Anthony Towns analyzed the votes and declared a tie between systemd and Upstart, which left it up to the chairman to decide by using the casting vote. Garbee did just that, voting for systemd, which makes it the Debian Linux default init for jessie. At least for now, since the prospect of a GR to decide is being bandied about.

There is another concern that Garbee expressed in his CFV: does the TC have the "jurisdiction" to decide on the coupling question at all. Opinions vary, but coupling was certainly not part of the original question that was asked of the TC.

The coupling question is a bit murky from a Debian Constitution standpoint, as Roeckx is concerned that a binding decision from the TC (rather than just an advisory one) would run counter to the committee's mandate. In section 6.3.5 of the Constitution, the committee is restricted from "engag[ing] in design of new proposals and policies" and should be "choosing from or adopting compromises between solutions and decisions which have been proposed and reasonably thoroughly discussed elsewhere". Deciding on loose vs. tight coupling might violate either or both, though Roeckx has not offered a formal opinion on that question. Jackson took strong exception to Roeckx's concerns and claimed that the Project Secretary should not be able to overrule the TC:

Roeckx did not disagree about the Secretary's powers, but he did see the possibility that a dispute might arise. He would have to make a decision at that point, which would presumably be handled by a GR if other Debian Developers (DDs) agreed. Mostly, he would like to avoid the problem altogether:

Aftermath

Garbee's CFV clearly made Jackson extremely angry. As Garbee predicted, Jackson voted FD above all other choices (and ranked the init choices in the following order: sysvinit, OpenRC, Upstart, systemd), but he also let his anger get away from him. He quickly called for three separate votes on various earlier proposals, as well as a call for the TC to ask Garbee to resign as its chair. That was quickly followed up with a note that he would be stepping away from the keyboard and taking a break from TC business for a few days.

As Allbery noted in the message linked above and another, both of which were in response to a call for Jackson to be removed from the TC, it is not surprising that Jackson got so angry at what he sees as a "betrayal". Allbery argued that Jackson retreated into the details of the process because the decision was so divisive:

He goes on to show how "real world" legislative bodies suffer from some of the same troubles. The US Senate has been fighting about its procedures for some years, and a unilateral change to those traditions led to some of the same kind of rancor we see here. But Allbery brought up another important point: the tradition of consensus ballot-building (and, in general, largely consensus-based decisions) in the TC meant that the corner cases when things got messy had never been tested. This particular incident has shown a number of those corner cases, any or all of which might lead to GRs to fix them.

One of those corner cases was alluded to by Garbee in his latest CFV: that the Condorcet voting method, coupled with Debian's vote resolution procedure could produce results that were, at best, unexpected. As Allbery said, the procedure fails the "later no harm" criterion. In a nutshell, that means that a participant can cause their preferred outcome to fail by ranking less-preferred outcomes (e.g. an Upstart proponent could cause systemd to win by ranking it second to Upstart). Sébastien Villemot posted an analysis showing one possible surprising outcome: "In particular, if the ballot had not included the options about coupling, then systemd would have won because of the casting vote of the chairman."

It is not exactly clear why Jackson is so insistent that the two questions be tied together. He said that the coupling question was the most important in his mind and that voting for the default init first would make it difficult for him to decide where to rank systemd as he is trying to avoid the systemd-tight-coupling pairing. But there is nothing stopping him from assuming that tight coupling wins and voting down systemd accordingly. In the end analysis, there never seemed to be enough votes for loose coupling to win, either on its own or in a ballot with the default init question, which is part of why some saw Jackson's actions as largely just delaying the inevitable.

There are ongoing discussions between Langasek and Allbery to reach some kind of accord on the coupling question (or at least a ballot that may be agreeable). TC member Keith Packard is optimistic that enough progress is being made in that discussion that it may be resolvable without the TC having to make a decision at all:

Next steps

Jackson reappeared on the mailing list on February 12. He posted a draft of the coupling question (along with the wording to allow a majority override via GR) that he plans to put up for a vote on February 14 (with amendments as suggested by other TC members in the interim). He also issued an ultimatum: he will propose and/or sponsor a GR on the init question (presumably the default init system and the coupling question) under a few different scenarios. If his proposal results in a vote of FD or if anyone calls an immediate CFV on a related issue without giving him the opportunity to amend it, he will immediately propose a GR. Beyond that: "If the TC's conclusion on the coupling question is IMO not sufficiently robust I will probably canvass opinions before deciding whether to propose a GR."

The GR question is, of course, the big one moving forward. It's hard to imagine that six DDs—the number required to put a GR to a vote—won't come together to try to push the decision in one direction or the other, so the only real question may be what form it will take. Will there be push to switch the default to Upstart? Or, as some think more likely, a move back to the status quo of sysvinit? Or will Jackson sponsor a GR? The possibility of multiple conflicting GRs also exists. There are, undoubtedly, wagers being made about various possible outcomes, but for now the default for Debian in the jessie release for Linux will be systemd. Stay tuned for further updates ...

Comments (58 posted)

New features in Python 3.4

Python major releases come at roughly 18-month intervals, so the announcement of the first release candidate for Python 3.4 is pretty much right on schedule—Python 3.3 was released in September 2012. There are no changes to the core language in this release, but there are changes to the implementation and libraries; we will look at some of that here.

Python development proceeds in a generally orderly fashion. Python Enhancement Proposals (PEPs) are proposed, discussed, and pronounced upon by BDFL Guido van Rossum (or his designee for that PEP, the BDFL-Delegate). If they are accepted, they get added into the next release of the language. So, for Python 3.4, there is a list of PEPs that were incorporated into the release.

There are, for example, new modules that have been added to the standard library. Unlike a number of other "scripting" languages, Python comes "batteries included" with a large and diverse set of utility modules. A new enum type has been added to bind symbolic names to constant values. We looked at the discussions around the new type back in May 2013. It will allow code like the following:

    from enum import Enum

    class Color(Enum):
        red = 1
        blue = 2
        green = 3

Another addition is the statistics module. It provides a basic set of statistical functions, such as mean, median, mode, variance, standard deviation, and so on. It is not meant to be an advanced statistical function library; for that, users should turn to NumPy or some other tool.

Van Rossum's latest project (beyond just shepherding Python development, of course) is the asyncio module for supporting asynchronous I/O in the standard library. There have long been solutions outside of the standard library for doing asynchronous I/O (e.g. Twisted, Tornado, gevents, ...). Van Rossum looked at the existing frameworks and came up with his own. It adopts some features from Twisted (Transports and Protocols), but has a way for those who don't like callbacks (Van Rossum doesn't) to use a combination of Futures and yield from to create coroutines. It is complicated to understand (he called it "brain-exploding" at PyCon 2013), but is eagerly anticipated in some circles.

Developers are likely to appreciate the addition of a tracemalloc module. Valgrind and other tools can report on memory allocation problems, but those reports are based on the C code in the interpreter rather than the Python code that triggered the problems. Using tracemalloc will allow developers to see a Python traceback from the location where objects were allocated. For ease of use, it can be enabled via an environment variable or command-line flag.

The final new module added for 3.4 is an object-oriented path-handling module, which is based on the existing Python Package Index (PyPI) pathlib module and uses the same name. It provides a higher-level abstraction than the os.path module does and will make a lot of path-handling code easier to write. It comes with a rich set of operations to handle matching, globbing, various kinds of path decomposition, and so on.

The standard functools module gets some additional functionality to support "single dispatch" functions. These are functions that have different implementations based on the type of a single argument (thus "single" dispatch). That will allow separate functions to implement the functionality for each different type:

    from functools import singledispatch

    @singledispatch
    def fun(arg, optarg=False):
        print(arg)

    @fun.register(int)
    def _(arg, optarg=False):
        print(arg + 10)

    @fun.register(list)
    def _(arg, optarg=False):
        for i in arg:
            print(i)

    >>> fun(10)
    20
    >>> fun([ 'a', 'list' ])
    a
    list
    >>> fun(30.9)
    30.9

One of the headline features of 3.4 will surely be the inclusion of the pip installer bundled with the language. Python has long lacked a standard installer for external modules, so the addition of pip will be welcomed by many. Nick Coghlan, one of the PEP's authors, spoke about Python packaging at linux.conf.au earlier this year.

The Python import system got an upgrade as well. The ModuleSpec type has been added to hold a number of attributes about modules (name, location, loader, etc.) that can be used when extending the import system.

The hash algorithm used for strings (and bytes) has been changed to use SipHash on most systems to eliminate a problem with hash collisions. Those hashes are used for dictionary keys; a large number of collisions can result in a major performance decrease and, eventually, a denial of service. We looked at the switch back in November.

A new pickle protocol has been added (protocol number 4). It will be more space-efficient, support large objects requiring 64-bit lengths, handle sets and frozensets, and more.

Another smaller enhancement is to make subprocess file-descriptors not be inheritable by children. It will use the O_CLOEXEC flag to the open() system call on POSIX systems to ensure that children don't inherit the parent's descriptors, which can cause a number of race conditions and security holes. "We are aware of the code breakage this is likely to cause, and doing it anyway for the good of mankind."

There are also a number of new features in the CPython code (which is what implements the standard Python interpreter). An improvement to the garbage collector will allow objects with finalizers (e.g. a __del__() method) to be reclaimed even if they are part of reference cycle. There is also a new API for memory allocators. Under some circumstances, like embedding the interpreter or running it on low-memory devices, a different memory allocator is desirable. This PEP adds a stable API to do so.

In addition, Python 3.4 release manager Larry Hastings has added the Argument Clinic, which is a domain-specific language (DSL) to simplify the argument processing of "built-ins" (language functions that are implemented in CPython). It is a pre-processor that is run on suitably annotated C files that adds C code right after the annotation (which lives in a comment). An Argument Clinic Derby was held recently to add the annotation to much of C source of the interpreter.

There are also lots of bug fixes that went into the release, of course. Those are detailed in the Changelog. If the current release schedule holds, we can expect the final Python 3.4 release on March 16.

Comments (none posted)