Development

New features in Python 3.6

The Python 3.6 release occurred on December 23, only one week later than planned all the way back in October 2015. Python 3.6 adds a number of new features, including more support for asynchronous operations (generators and comprehensions), a filesystem path protocol, a new literal string formatting option, two random-number-related features, a frame evaluation API for debuggers and just-in-time (JIT) compilation, and more. Some of these features have been described in LWN articles along the way, but many haven't, so an overview of the highlights of the new release is in order.

As always, the release schedule (PEP 494) and, especially, the "What's New In Python 3.6" web pages provide an excellent overview of the release as well.

String formatting

Two string-formatting changes made their way into 3.6. The first adds yet another way to format text strings in Python using f-strings (which are described in PEP 498). This creates a new literal string type in the language (f'string') that can be used to directly interpolate variables (and full expressions) into strings:

    >>> num = 42
    >>> f'November 20{num}'
    November 2042
    >>> f'{num} * {num} = {num*num}'
    42 * 42 = 1764

In addition, PEP 515 was adopted, which extends the language's syntax to allow underscores into numeric literals for readability purposes:

    a = 1_000_000
    b = 0x_abcd_ef09

A path protocol

The addition of a filesystem path protocol (PEP 519) will allow easier handling of path names in the language and standard library. The pathlib module was meant to help solve the problem of different representations for path names when it was added (on a provisional basis) for Python 3.4. But it suffered from a lack of widespread adoption, even by other parts of the language (e.g. open()), at least partly because it was provisional. Some discussions in the early part of this year led to the development of the PEP and the removal of pathlib's provisional status.

All of that paved the way for open() and the standard library to start accepting pathlib.Path objects. The protocol allows objects to declare their path-like nature by implementing the __fspath__() method. The os.PathLike() function can be consulted to determine if something is path-like and there are explicit operations in the os module to get either str or bytes representations of paths. The "What's New" page fills in some details as well:

Frame evaluation

JITs were a hot topic at the 2016 Python Language Summit; one of the presentations on the Pyjion project introduced a plan to add an API to allow alternative ways to evaluate Python code (i.e. a frame-evaluation API). That idea has come to fruition with the adoption of PEP 523. It adds hooks to the CPython interpreter (the reference Python implementation) that will allow JIT engines as well as debugging tools to intercept the evaluation of frames. That will allow other programs to provide alternate ways to evaluate and execute the code in the frame objects.

Asynchronicity

Two PEPs that extend Python's asynchronous feature set have been adopted for 3.6. They both extend the async/await concept into more contexts. The addition of the async and await keywords in Python 3.5 furthered the move toward better support for asynchronous programming into the language itself. That largely started with the provisional adoption of the asyncio module back in 3.4 (see our coverage of Guido van Rossum's 2013 PyCon keynote for some background).

The asynchronous generators feature (as described in PEP 525) is effectively just a shortcut, but also provides improved performance—roughly twice as fast. Currently, in order to have an asynchronous generator that can be used in async for statements, a class must be defined that implements the __aiter__() and __anext__() asynchronous protocol. An alternative might have been to allow yield statements inside functions defined with async def, but when support for async/await was added, the yield statement was not allowed for async def coroutines. Changing the language to accept that made asynchronous generators easier to write and roughly twice as fast. The PEP gives an example of how that might be used (and compares it to how it would currently need to be written):

    async def ticker(delay, to):
	"""Yield numbers from 0 to `to` every `delay` seconds."""
	for i in range(to):
	    yield i
	    await asyncio.sleep(delay)

PEP 530 allows async and await to be available to comprehensions (i.e. list, dict, and set comprehensions as well as generator expressions) so that async can be used in coroutines defined with async def. The PEP gives an example of an asynchronous list comprehension that uses an asynchronous generator (agen()):

    [ i async for i in agen() ]

In addition, await can be used in comprehensions in coroutines to wait for the completion of a asyncio.Future object. One of the dict examples given in the PEP is shown below, but more intricate comprehensions are possible:

    result = { fun : await fun() for fun in funcs }

Random numbers

As was described in an article back in July, Python will return to having a blocking os.urandom() call. That change was added to Python 3.5, but had an unexpected side effect: Python initialization could block waiting for enough entropy to initialize its seed for dictionary randomization (to avoid hash collision attacks). In 3.5.2, Python stopped using the blocking call for its own initialization, but also partly reverted the change to os.urandom() to give users some time to adjust. For Python 3.6 and beyond, though, the call will block until there is sufficient entropy (i.e. will call getrandom() without the GRND_NONBLOCK flag) in the kernel's entropy pool.

Another change that came about as the project looked at how Python generates random numbers is the advent of the secrets module, which has been specified to provide crypto-strength random numbers. The default random number generator provided by the random module is documented to be unsuited for cryptographic purposes, but was seen as an "attractive nuisance" (as PEP 506 called it) that would draw in new programmers based on code snippets on various web sites. It is hoped that adding secrets will provide users with an obvious standard library alternative that does not suffer from the shortcomings of random.

Variable annotations

Type annotations (or type hints) have been fast-tracked into Python over the last two years or so. The first support for the optional annotations was added to Python 3.5 (PEP 484), but that did not address annotations for variables (just function arguments and return values). PEP 484 suggested using comments as a stopgap, but it was clear that variable annotation would eventually come (see, for instance, Guido van Rossum's 2015 Python Language Summit talk on type hints).

As Van Rossum predicted in that talk, variable annotations will make an appearance in Python 3.6. The optional annotation will be allowed in variable initializations or just as a type declaration. From PEP 526:

    some_number: int           # variable without initial value
    some_list: List[int] = []  # variable with initial value

As with the earlier annotations, there are no language semantics associated with the annotations, but they will be placed into the __annotations__ ordered dictionary for the class, module, or function where they occur. They can also be retrieved using the typing.get_type_hints() API. It is important to note (and the PEP emphasizes) that: "Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention."

Compact dictionaries

The internal representation of the dict type has been changed to use far less memory (20-25% less compared to Python 3.5). This is a feature that came from the PyPy project. As a side effect of the new representation, dict now preserves the order in which items were added. This side effect should not be relied upon going forward, but choosing collections.OrderedDict when preserving the order is needed will benefit from the new dict implementation.

And more

There are more features listed for the release, including the removal of the provisional status for the asyncio module, updating the SSL modules to support OpenSSL 1.1.0, adding a PYTHONMALLOC environment variable for debugging memory allocation issues, and more. All in all, it is a solid release that brings some useful facilities, as well as rounding out some features that have appeared in earlier releases. It is far too early to guess what might appear in Python 3.7, but it is currently scheduled for a mid-2018 release.

Comments (9 posted)

Development quote of the week

Comments (none posted)

darktable 2.2.0 released

Comments (2 posted)

Inkscape 0.92 released

Comments (3 posted)

LedgerSMB 1.5.0 released

Full Story (comments: none)

Python 3.6.0 released

Full Story (comments: none)

sed-4.3 released [stable]

Full Story (comments: 2)

Development newsletters

• Emacs News (December 26)
• Emacs News (January 2)
• What's cooking in git.git (December 21)
• What's cooking in git.git (December 22)
• What's cooking in git.git (December 27)
• Koha Community Newsletter (December)
• OCaml Weekly News (December 27)
• OCaml Weekly News (January 3)
• Perl Weekly (December 25)
• Perl Weekly (January 2)
• PostgreSQL Weekly News (December 25)
• PostgreSQL Weekly News (January 1)
• Project Gutenberg Newsletter (October-December)
• Python Weekly (December 22)
• Ruby Weekly (December 22)
• This Week in Rust (December 27)
• This Week in Rust (January 3)

Comments (none posted)

Grumpy: Go running Python

Comments (47 posted)

Ringing in 2017 with 90 hacker-friendly single board computers (HackerBoards)

Comments (25 posted)

Larsson: A stable base for Flatpak: 0.8