Development

Python coroutines with async and await

It is already possible to create coroutines for asynchronous processing in Python. But a recent proposal would elevate coroutines to a full-fledged language construct, rather than treat them as a type of generator as they are currently. Two new keywords, async and await, would be added to the language to support coroutines as first-class Python features.

A coroutine is a kind of function that can suspend and resume its execution at various pre-defined locations in its code. Subroutines are a special case of coroutines that have just a single entry point and complete their execution by returning to their caller. Python's coroutines (both the existing generator-based and the newly proposed variety) are not fully general, either, since they can only transfer control back to their caller when suspending their execution, as opposed to switching to some other coroutine as they can in the general case. When coupled with an event loop, coroutines can be used to do asynchronous processing, I/O in particular.

Python's current coroutine support is based on the enhanced generators from PEP 342, which was adopted into Python 2.5. That PEP changed the yield statement to be an expression, added several new methods for generators (send(), throw(), and close()), and ensured that close() would be called when generators get garbage-collected. That functionality was further enhanced in Python 3.3 with PEP 380, which added the yield from expression to allow a generator to delegate some of its functionality to another generator (i.e. a sub-generator).

But all of that ties coroutines to generators, which can be confusing and also limits where in the code it is legal to make an asynchronous call. In particular, the with and for statements could conceptually use an asynchronous call to a coroutine, but cannot because the language syntax does not allow yield expressions in those locations. In addition, if a refactoring of the coroutine moves the yield or yield from out of the function (into a called function, for example), it no longer is treated as a coroutine, which can lead to non-obvious errors; the asyncio module works around this deficiency by using a @asyncio.coroutine decorator.

PEP 492 is meant to address all of those issues. The ideas behind it were first raised by Yury Selivanov on the python-ideas mailing list in mid-April, it was enthusiastically embraced by many in that thread, and by May 5 it had been accepted for Python 3.5 by Guido van Rossum. Not only that, but the implementation was merged on May 12. It all moved rather quickly, though it was discussed at length in multiple threads on both python-ideas and python-dev.

The changes are fairly straightforward from a syntax point of view:

    async def read_data(db):
        data = await db.fetch('SELECT ...')
	...

That example (which comes from the PEP) would create a read_data() coroutine using the new async def construct. The await expression would suspend execution of read_data() until the db.fetch() awaitable completes and returns its result. await is similar to yield from, but it validates that its argument is an awaitable.

There are several different types of awaitable. A native coroutine object, as returned by calling a native coroutine (i.e. one defined with async def) is an awaitable, as is a generator-based coroutine that has been decorated with @types.coroutine. Future objects, which represent some processing that will complete in the future, are also awaitable. The __await__() magic method is present for objects that are awaitable.

There is a problem that occurs when adding new keywords to a language, however. Any variables that are named the same as the keyword suddenly turn into syntax errors. To avoid that problem, Python 3.5 and 3.6 will "softly deprecate" async and await as variable names, but not have them be a syntax error. The parser will keep track of async def blocks and treat the keywords differently within those blocks, which will allow existing uses to continue to function.

There are two other uses of async that will come with the new feature: asynchronous context managers (i.e. with) and iterators (i.e. for). Inside a coroutine, these two constructs can be used as shown in these examples from the PEP:

    async def commit(session, data):
	...

	async with session.transaction():
	    ...
	    await session.update(data)
	    ...
        ...
        async for row in Cursor():
            print(row)

Asynchronous context managers must implement two magic async methods, __aenter__() and __aexit__(), both of which return awaitables, while an asynchronous iterator would implement __aiter__() and __anext__(). Those are effectively the asynchronous versions of the magic methods used by the existing synchronous context manager and iterator.

The main question early on was whether the deferred "cofunction" feature (PEP 3152) might be a better starting point. The author of that PEP, Greg Ewing, raised the issue, but there was a lot of agreement that the syntax proposed by Selivanov was preferable to the codef, cocall, and the like from Ewing's proposal. There was a fair amount of back and forth, but the cofunction syntax for handling certain cases got rather complex and non-Pythonic in the eyes of some. Van Rossum summarized the problems with cofunctions while rejecting that approach.

There were also several suggestions of additional asynchronous features that could be added, but nothing that seemed too urgent. There was some bikeshedding on the keywords (and their order, some liked def async, for example). The precedence of await was also debated at some length, with the result being that, unlike yield and yield from that have the lowest precedence, await has a high precedence: between exponentiation and subscripting, calls, and attribute references.

Mark Shannon complained that there was no need to add new syntax to do what Selivanov was proposing. Others had made similar observations and it was not disputed by Selivanov or other proponents. The idea is to make it easier to program with coroutines. Beyond that, Van Rossum wants the places where a coroutine can be suspended to be obvious from reading the code:

Over a two to three week period, multiple versions of the PEP were posted and debated, with Selivanov patiently explaining his ideas or modifying them based on the feedback. For a feature that seems likely to be quite important in Python's future, the whole process went remarkably quickly—and smoothly. It will probably take a fair amount more time for those ideas to sink in more widely with Python developers.

Comments (7 posted)

Quotes of the week

— Nick Coghlan

— Jason Scott, announcing the debut of the Internet Archive's in-browser DOS simulator.

Comments (1 posted)

Firefox 38.0

Mozilla has released Firefox 38.0. This version features new tab-based preferences and Ruby annotation support. Also, it will be the base for the next ESR release. The release notes contain more information.

Comments (22 posted)

Update on Digital Rights Management and Firefox

At the Mozilla Blog, Denelle Dixon-Thayer announced that the first Firefox builds to include a "digital rights management" (DRM) module have been released. The module comes from Adobe and, in acknowledgment of its controversial nature, Mozilla will still be offering DRM-free builds for download, instructions on how to remove the module, and a "teaching kit" that details the various side of the DRM debate.

Comments (5 posted)

Choosing a license for Mailpile 1.0

The Mailpile project is soliciting input on its eventual license. The leading contenders are the Affero GPLv3 and the Apache 2.0 license. The project's post on the question outlines the usual (perceived) advantages and disadvantages of each. "For some, this is an idealogical question, a matter of right or wrong. Is it morally acceptable to allow people to benefit from Mailpile's work without contributing back? Is it morally acceptable for Mailpile's authors to tie the hands of other businesses? For others, this is merely a matter of strategy and tactics." In this case, "voting" appears to mean commenting on the topic in various public discussion forums. Mailpile is a highly anticipated project; the results of these discussions will, at the very least, be interesting to watch.

Comments (1 posted)

Glyphr Studio 1.0 released

Open-source browser-based font editor Glyphr Studio has made its 1.0 release. We noted the development of Glyphr Studio in out recent look at the FontForge's development talk from Libre Graphics Meeting 2015. The 1.0 release supports import and export of both OpenType and TrueType fonts, as well as a feature initially slated for post-1.0 release: the ability to construct glyphs out of reusable components.

Comments (none posted)

GNU inetutils 1.9.3 released

GNU inetutils 1.9.3 is now available. New in this release are the ability to specify a pattern as the payload for ping, several updates to ftp, and a --local-time switch for syslogd that "makes the service ignore a time stamp passed on by the remote host, recording instead the local time at the moment the message was received."

Full Story (comments: none)

Development newsletters from the past week

• What's cooking in git.git (May 8)
• What's cooking in git.git (May 11)
• Git Rev News (May 13)
• LLVM Weekly (May 11)
• OCaml Weekly News (May 12)
• OpenStack Community Weekly Newsletter (May 8)
• Perl Weekly (May 11)
• PostgreSQL Weekly News (May 10)
• Python Weekly (May 7)
• Ruby Weekly (May 7)
• TUGboat (April 2014)
• Wikimedia Tech News (May 11)

Comments (none posted)

How OpenStack gets translated (Opensource.com)

Over at Opensource.com, one of the translators for OpenStack, Łukasz Jernaś, is interviewed about the process of translating a large project like OpenStack. "How does OpenStack's release cycle play into the translation process? Is it manageable to get translations done on a six-month release cycle? Most of the work gets done after the string freeze period, which happens around a month before the release, with a lot of it being completed after getting the first release candidate out of the window. Documentation is translated during the entire cycle, as many parts are common between releases and can be deployed independently to the releases. So we don't have to focus that much about deadlines, as it's available online all the time and not prepackaged and pushed out to users and distributions. Of course, having a month to do the translations can be cumbersome, depending on the team doing the translation (some do that part time, some people in their spare time), and how many developers push out new strings during the string freeze."

Comments (none posted)

MathML Accessibility

At his blog, Frédéric Wang has posted a comparison between the Orca screen reader and several proprietary alternatives when they are given the task of reading MathML expressions in Firefox. The audio snippets provided indicate that Orca has some catching up to do. Fortunately, it appears that Mozilla and Orca developers are intent on closing the gap.

Comments (none posted)

Testable Examples in Go

At the Go Blog, Andrew Gerrand provides a look at the language's approach to combining example code and documentation. "Godoc examples are snippets of Go code that are displayed as package documentation and that are verified by running them as tests. They can also be run by a user visiting the godoc web page for the package and clicking the associated "Run" button. Having executable documentation for a package guarantees that the information will not go out of date as the API changes." Each package's examples are compiled as part of the package test suite; examples can also (optionally) be executed in order to capture failures with the testing framework.

Comments (7 posted)