New features in Python 3.4

Python major releases come at roughly 18-month intervals, so the announcement of the first release candidate for Python 3.4 is pretty much right on schedule—Python 3.3 was released in September 2012. There are no changes to the core language in this release, but there are changes to the implementation and libraries; we will look at some of that here.

Python development proceeds in a generally orderly fashion. Python Enhancement Proposals (PEPs) are proposed, discussed, and pronounced upon by BDFL Guido van Rossum (or his designee for that PEP, the BDFL-Delegate). If they are accepted, they get added into the next release of the language. So, for Python 3.4, there is a list of PEPs that were incorporated into the release.

There are, for example, new modules that have been added to the standard library. Unlike a number of other "scripting" languages, Python comes "batteries included" with a large and diverse set of utility modules. A new enum type has been added to bind symbolic names to constant values. We looked at the discussions around the new type back in May 2013. It will allow code like the following:

    from enum import Enum

    class Color(Enum):
        red = 1
        blue = 2
        green = 3

Another addition is the statistics module. It provides a basic set of statistical functions, such as mean, median, mode, variance, standard deviation, and so on. It is not meant to be an advanced statistical function library; for that, users should turn to NumPy or some other tool.

Van Rossum's latest project (beyond just shepherding Python development, of course) is the asyncio module for supporting asynchronous I/O in the standard library. There have long been solutions outside of the standard library for doing asynchronous I/O (e.g. Twisted, Tornado, gevents, ...). Van Rossum looked at the existing frameworks and came up with his own. It adopts some features from Twisted (Transports and Protocols), but has a way for those who don't like callbacks (Van Rossum doesn't) to use a combination of Futures and yield from to create coroutines. It is complicated to understand (he called it "brain-exploding" at PyCon 2013), but is eagerly anticipated in some circles.

Developers are likely to appreciate the addition of a tracemalloc module. Valgrind and other tools can report on memory allocation problems, but those reports are based on the C code in the interpreter rather than the Python code that triggered the problems. Using tracemalloc will allow developers to see a Python traceback from the location where objects were allocated. For ease of use, it can be enabled via an environment variable or command-line flag.

The final new module added for 3.4 is an object-oriented path-handling module, which is based on the existing Python Package Index (PyPI) pathlib module and uses the same name. It provides a higher-level abstraction than the os.path module does and will make a lot of path-handling code easier to write. It comes with a rich set of operations to handle matching, globbing, various kinds of path decomposition, and so on.

The standard functools module gets some additional functionality to support "single dispatch" functions. These are functions that have different implementations based on the type of a single argument (thus "single" dispatch). That will allow separate functions to implement the functionality for each different type:

    from functools import singledispatch

    @singledispatch
    def fun(arg, optarg=False):
        print(arg)

    @fun.register(int)
    def _(arg, optarg=False):
        print(arg + 10)

    @fun.register(list)
    def _(arg, optarg=False):
        for i in arg:
            print(i)

The singledispatch decorator indicates that fun() will have different implementations based on the type of arg. The register() calls then set up each function, so calling fun() with different argument types would look something like this:

    >>> fun(10)
    20
    >>> fun([ 'a', 'list' ])
    a
    list
    >>> fun(30.9)
    30.9

One of the headline features of 3.4 will surely be the inclusion of the pip installer bundled with the language. Python has long lacked a standard installer for external modules, so the addition of pip will be welcomed by many. Nick Coghlan, one of the PEP's authors, spoke about Python packaging at linux.conf.au earlier this year.

The Python import system got an upgrade as well. The ModuleSpec type has been added to hold a number of attributes about modules (name, location, loader, etc.) that can be used when extending the import system.

The hash algorithm used for strings (and bytes) has been changed to use SipHash on most systems to eliminate a problem with hash collisions. Those hashes are used for dictionary keys; a large number of collisions can result in a major performance decrease and, eventually, a denial of service. We looked at the switch back in November.

A new pickle protocol has been added (protocol number 4). It will be more space-efficient, support large objects requiring 64-bit lengths, handle sets and frozensets, and more.

Another smaller enhancement is to make subprocess file-descriptors not be inheritable by children. It will use the O_CLOEXEC flag to the open() system call on POSIX systems to ensure that children don't inherit the parent's descriptors, which can cause a number of race conditions and security holes. "We are aware of the code breakage this is likely to cause, and doing it anyway for the good of mankind."

There are also a number of new features in the CPython code (which is what implements the standard Python interpreter). An improvement to the garbage collector will allow objects with finalizers (e.g. a __del__() method) to be reclaimed even if they are part of reference cycle. There is also a new API for memory allocators. Under some circumstances, like embedding the interpreter or running it on low-memory devices, a different memory allocator is desirable. This PEP adds a stable API to do so.

In addition, Python 3.4 release manager Larry Hastings has added the Argument Clinic, which is a domain-specific language (DSL) to simplify the argument processing of "built-ins" (language functions that are implemented in CPython). It is a pre-processor that is run on suitably annotated C files that adds C code right after the annotation (which lives in a comment). An Argument Clinic Derby was held recently to add the annotation to much of C source of the interpreter.

There are also lots of bug fixes that went into the release, of course. Those are detailed in the Changelog. If the current release schedule holds, we can expect the final Python 3.4 release on March 16.

---

to post comments