Development [LWN.net]

Improving performance in Python 2.7

By Jake Edge
June 3, 2015

Backporting a major performance improvement from Python 3 to Python 2 might seem to be in the "no-brainer" category, but things are not quite that simple. Python 2 (in the form of Python 2.7.x) is in a "no new features" mode that would normally preclude large changes, but 2.7 will be around for a lot longer than was previously envisioned. That makes the core development team more willing to consider this kind of patch, especially since it seems to come with a promise of more contributions. As with any topic that touches on the 2 vs. 3 question, though, it led to a long discussion and some dissent.

Vamsi Parasa of the Server Scripting Languages Optimization team at Intel posted a patch to change the switch statement that executes Python bytecode in the CPython interpreter to use computed gotos instead. GCC has support for the feature, which has already been used as an optimization in Python 3. As Eli Bendersky explained in a 2012 blog post, the enormous (2000+ line) switch statement for interpreting Python bytecodes can be made 15-20% faster by changing it to use computed gotos. There are two reasons for that: computed gotos avoid a bounds check that is required by the C99 standard for switch and, perhaps more significantly, the CPU can do better branch prediction, which reduces expensive pipeline flushes.

Several core developers spoke up in favor of the patch, but Berker Peksağ was not so sure. He complained that "performance improvements are not bug fixes" and that it would be better to spend time making Python 3 better. But there is more to this patch than meets the eye, Nick Coghlan explained:

Internal performance improvements, by contrast, don't hurt end users at all beyond the stability risks, and in this case, the request to make the change is being accompanied by the offer to assist with ongoing maintenance (including engaging an experienced core developer to help coach Intel contributors through the contribution process).

So when folks ask "What changed?" in relation to this request, what changed is the fact that it isn't expected to be a one off contribution, but rather part of a broader effort focused on improving the performance of both Python 2 and Python 3, including contributions to ongoing maintenance activities.

As was noted in an email exchange tacked onto Parasa's post, Intel has hired Python core developer David Murray's company to help out navigating the Python development process with contributions it wants to make. But those contributions "came with one string attached: that the Python 2.7 branch be opened up for performance improvements in addition to bug fixes", Coghlan said. Because the proposal is "backed by a credible offer of ongoing contributions to CPython maintenance and support", that makes it different than other performance enhancements (or features) offered up for Python 2.7 along the way, he continued.

But accepting the offer does signal something of a shift in the way Python 2.7 will be maintained going forward. Encouraging more developers who are being paid to do the "boring" parts of Python maintenance (bug fixes and performance enhancements for 2.7, mostly) will allow volunteers to concentrate on the more interesting bits. Coghlan again:

Giving the nod to an increased corporate developer presence in Python 2 maintenance should eventually let volunteers stop worrying about even Python 2.7 bug fix changes with a clear conscience, confident that as volunteer efforts drop away redistributors and other folks with an institutional interest will pick up the slack with paid development time. "Do the fun stuff for free, figure out a way to get paid for the boring-but-necessary stuff (or leave those tasks to someone else that's getting paid to handle them)" is a good sustainable approach to open source development, while trying to do it *all* for free is a fast path to burnout.

Python's benevolent dictator for life (BDFL) Guido van Rossum was "strongly in favor" of the patch noting that it could save companies like his employer, Dropbox, "a lot of money". Dropbox has been slow to move to Python 3 but regularly updates to the latest Python 2.7 release, he said. But Victor Stinner wondered if it made more sense to put that effort into Python 3 in the hopes of getting more migration to that version of Python.

Van Rossum was clearly not happy with the idea of crippling Python 2 to somehow promote Python 3:

However this talk of "wasting our time with Python 2" needs to stop, and if you think that making Python 2 less attractive will encourage people to migrate to Python 3, think again. Companies like Intel are *contributing* by offering this backport up publicly.

As Larry Hastings pointed out, though, the question had already become moot. Python 2.7 release manager Benjamin Peterson had merged the patch for 2.7.11, which is slated for release in December.

The rate of Python 3 adoption and the best way to encourage it are often somewhat contentious subjects in Python circles. Though many have bemoaned the lack of new features in Python 2.7, a large part of the decision to do that has been driven by a lack of core developers interested in continuing to work on that branch. Adding more paid developers should help alleviate some of those concerns. It seems unlikely to lead to any wholesale changes to 2.7, but there may be enhancements that can be made in the next few years. Meanwhile, new features like async/await, type hints, and others will continue to be added to Python 3 to hopefully provide a carrot to draw more and more to that version of the language. Or at least that is the plan ...

Comments (14 posted)

Brief items

Quotes of the week

And yes, if you learn Python 3 first, subsequently encountering Python 2's quirks and cruft is likely to encourage folks that know both versions of the language to start advocating for a version upgrade :)

— Nick Coghlan

libinput originally only supported single-tap and double-tap. With version 0.15 we now support multi-tap, so you can tap repeatedly to get a triple, quadruple, etc. click. This is quite useful in text editors where a triple click highlights a line, four clicks highlight a paragraph, and 28 clicks order a new touchpad from ebay.

— Peter Hutterer (thanks to Josh Triplett)

Comments (none posted)

Firefox 38.0.5

Firefox 38.0.5 has been released. This version introduces Pocket, which helps you keep track of articles and videos. Clean formatting for articles and blog posts with Reader View is also a new feature. See the release notes for more information.

Comments (76 posted)

SfePy 2015.2 released

Version 2015.2 of the SfePy library has been released. SfePy is a Python package "for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (preliminary support)." The new version marks the release of some major code-simplification work and reorganization of the installation files and examples. Also included are several new features, such as the ability to use time stepping solvers in interactive mode.

Full Story (comments: none)

Buildroot 2015.05 available

Version 2015.05 of Buildroot has been released. The announcement notes that this was the busiest development cycle yet; support was added for the steamroller architecture as well as the corei7-avx and core-avx2 x86 variants, libatomic is now supported for both internal and external toolchains, and close to 100 new packages have been added.

Full Story (comments: none)

Virtuozzo source code available

The OpenVZ team has announced the open source code release of several Virtuozzo userspace utilities. The utilities include prlctl, a unified command line tool to manage virtual machines and containers; libprlsdk, Virtuozzo API C++ and python libraries, used for local and remote communications with a dispatcher management service; prl-disp-service, a primary Containers and Virtual machines management service; libvzctl, a low-level library for Containers management; libvzevent, a low-level library for Containers life-cycle notifications from the kernel; vzctl, a utility to control a Containers; and vztt, a utility for Containers templates management.

Comments (7 posted)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

Announcing GitTorrent: A Decentralized GitHub

At his blog, Chris Ball announces "GitTorrent," his new project designed to let developers host Git repositories on BitTorrent. The system takes advantage of Git's ability to run over arbitrary network protocols. "We ask for the commit we want and connect to a node with BitTorrent, but once connected we conduct this Smart Protocol negotiation in an overlay connection on top of the BitTorrent wire protocol, in what’s called a BitTorrent Extension. Then the remote node makes us a packfile and tells us the hash of that packfile, and then we start downloading that packfile from it and any other nodes who are seeding it using Standard BitTorrent. We can authenticate the packfile we receive, because after we uncompress it we know which Git commit our graph is supposed to end up at; if we don’t end up there, the other node lied to us, and we should try talking to someone else instead." The project is, obviously, a new one that still has important ground to cover—such as dealing with comments or pull requests—but there are interesting ideas to consider already.

Comments (18 posted)

First Open Automotive Grade Linux Spec Released (Linux.com)

Linux.com talks with Dan Cauchy, general manager of automotive at the Linux Foundation, about the release of the AGL Requirements Specification. "In July 2014, AGL released its first AGL reference platform built on the Tizen IVI platform running HTML5 apps. The new release instead details precise specifications and requirements for any AGL-compliant IVI stack. For the first time, automakers, automotive suppliers, and open source developers can collaborate on refining the spec -- the first draft of a common, Linux-based software stack for the connected car."

Comments (none posted)

Page editor: Nathan Willis
Next page: Announcements>>