Python core development infrastructure

The core development infrastructure for Python, which includes things like version-control systems and repository hosting, is the subject of two current PEPs. The PEPs offer competing views for how to move forward, Nick Coghlan said. He noted that Brett Cannon made a comment at one point that Python changes the way its development processes and systems work periodically, then leaves them alone for a few years. That is the enterprise approach, he said with a chuckle, which "sucks as much for us as it does for them".

Two PEPs

Coghlan has one proposed plan involving Kallithea (PEP 474), while Donald Stufft (who was not present at the summit) has a proposal involving GitHub and Phabricator (PEP 481). Coghlan's PEP is not focused on CPython development directly (his deferred PEP 462 is, however) but is instead looking at the infrastructure for many of the parts and pieces that surround CPython (e.g. the PEPs and the developer guide). We looked at some of the early discussion of the issue back in December.

Coghlan's interest in the issue stems from the fact that he already works on process and infrastructure for development as part of his day job at Red Hat. His job entails figuring out how to get code from desktops into the continuous-integration (CI) system and then into production. It is a job that people don't want to work on for free, he said. Instead they will find "something more entertaining to do".

He said that one idea behind his proposal is to try to respect the time people are putting into the patches they are contributing to Python. He would also like to minimize the time between when a developer makes a contribution and when they see it land in the Python mainline. The changes he wants to see would still allow people to use their current workflow, he said. It would create a new workflow that existing developers could cherry pick pieces from if they liked them. New projects could default to the new workflow.

One of the complaints about his proposal, which is based on free-software solutions hosted on Python Software Foundation (PSF) infrastructure, is that there would be no commercial support available, unlike with GitHub. But Red Hat allows him to spend 20% of his time working on the Python infrastructure, which will provide some of the support.

The pull-request model of GitHub is a good one, Coghlan said. The pull request becomes the main workflow element. For relatively simple projects with fairly small development teams, it works well. For those kinds of projects, you can live without an issue tracker, wiki, and mailing lists as what is needed is a way to propose changes and to discuss them, which pull requests do. That distills the development problem down to its minimal core.

Both proposals will accept GitHub pull requests as part of the workflow, though only Stufft's uses GitHub itself directly (and keeps a read-only copy of any repositories in Phabricator). Someone asked, facetiously, why not move fully to GitHub. Coghlan said it is partly a matter of risk management. Outsourcing the infrastructure to a proprietary tool is too risky in the long run.

In addition, GitHub only provides "all or nothing" access to its repositories. That makes it harder to build a good on-ramp for new developers. If Python uses its own service, it can use or create a fine-grained access control mechanism to allow new developers some access without giving them access to everything. That is currently done, to some extent, in the bug tracker, which is based on Roundup.

His intent is for the Python infrastructure (what he is calling "forge.python.org") to interface with a number of different services and tools, such as GitHub, Gerrit, and GitLab, in order to try to mesh with the workflow of contributors while still supporting the existing process for current core developers.

GitHub is well-suited for a small, central development team, whereas other tools are a better fit for how Python development is done. he said. "'Just use GitHub' is the answer for an awful lot of projects, but I don't think it's right for Python." OpenStack uses Gerrit for its code review because it is better suited to a large, distributed development team. There are some good ideas in the OpenStack workflows that might be applied to CPython development.

Brett Cannon said that Coghlan has "strong opinions" on the matter, which is part of why Cannon has been put in charge of making the decision between the two PEPs. He hopes to make a decision by the beginning of May, so those with an opinion should be trying to convince him of the right approach before then. With a grin, Coghlan agreed, "I am thoroughly biased, which is why I don't decide" the issue. Cannon said that he doesn't particularly care that GitHub is closed-source, so long as Python can get its data out if it ever needs to.

It is important to be able to accept GitHub pull requests, Coghlan said, as well as those from Bitbucket, GitLab, and potentially others. Mozilla has also adopted this practice. Mozilla agrees that open source needs open infrastructure, but projects have to go where their developers are.

Jacob Kaplan-Moss said he would try to "channel Donald [Stufft]" a bit. He noted that Django switched to GitHub and has tried to optimize it for the Django workflow as well as community contributions. GitHub is better for the contributors, though, it is just "OK" for the core developers. It can be made to work, but is not optimal. There is an existential question, he said, about whether a project focuses on the workflow for its core developers or for its contributors. Both are perfectly valid choices, but a choice needs to be made.

Patch backlog

Barry Warsaw is concerned that turning more toward contributor workflows will cause a loss of core developers. Coghlan noted that there are 2000 unmerged patches on bugs.python.org. The bottleneck is not on contributions, he said, but on review. It doesn't make sense to make it easier for someone to send a patch if the project is going to essentially ignore it, Thomas Wouters said.

The review process is being held back by the current workflow options, though, Coghlan said. You can't just take five minutes to review a patch, see that it passes the CI tests, and then say go ahead and merge it. Stufft's original proposal was GitHub-only, but he has added Phabricator into the mix since then, which addresses Coghlan's concerns about PEP 481. Coghlan would prefer his option, but can live with Stufft's.

The choice of Phabricator is a good one from a workflow perspective; if you are looking for good workflow design, Facebook (which created Phabricator) is a good place to look, Coghlan said. He personally doesn't want to work in PHP, which is what Phabricator is written in, but he can work on other parts of the problem if that is the direction chosen.

Cannon asked Kaplan-Moss about the bug backlog in Django after the switch to GitHub. At first, he said that Django had a huge backlog before the switch and still does today. After looking a little deeper, though, he noted that the bug backlog had been cut by a third since the switch, but he is "not sure if they are related".

The huge patch backlog indicates that the workflow for core developers needs to be fixed before the contributor workflow, Cannon pointed out. Contributors may not like it, but they seem to be willing to deal with the existing Python workflow, which is completely different than that of any other project. Once code can get reviewed and merged easily, other changes can follow. "As of now, no one is happy", he said. One important piece is to not lose the "things that we like" about the current workflow, Warsaw said, though he didn't go into any detail.

Reusing Python's choice

Jython developer Jim Baker asked about other projects that might want to piggyback on the choice made. It would be great if Jython could simply use the infrastructure and workflow that CPython decides on, he said. Cannon said that he is "just trying to be the guy that makes the decision", but that any choice will be one that other projects can pick up if they wish.

Coghlan expanded on that noting that the containerization choices that have become available relatively recently will make that all a lot easier. There is a lot of hype and "marketing rubbish" around Docker, but there is some good technology too. Docker has put a nice user experience on top of a bunch of Linux tools, which provides a packaging solution for Linux that application developers don't hate. It will make it easier for anyone that wants to run their own version of the infrastructure that is adopted. His goal is help make it so that "open source projects don't have to live with crappy infrastructure anymore". Cannon pointed out that members of the PSF board have told him that there would be money available to help make some of these things happen once a decision is made.

One of the advantages of a pull request is that it identifies unambiguously which version of the code a patch will apply to, an attendee said. Is it possible to automate turning the existing patch backlog into pull requests or to at least give hints on the version a patch is targeting, he wondered. Perhaps a Google Summer of Code (GSoC) project could be aimed at this problem.

Another problem is the lack of a test farm, which is at least partly due to the fragility of the tests themselves. If there were a build and test farm, the systems in the farm would never agree on the test results, or at least not reliably.

Coghlan and another attendee said that there are some efforts to get GSoC students involved in solving some of these problems. One project is to put a REST API on Roundup, which may help doing some of the automated processing of the patch backlog.

An OpenStack developer said that he was in favor of fixing the core developer workflow as a way to make things better for the whole community. While it is important to consider GitHub because everyone uses it, the most important thing is to try to ensure that patches land in the mainline within a reasonable time frame. Both Kallithea and Phabricator are good tools, but neither existed when OpenStack was looking for something, so it chose Gerrit. The project is making headway on making its review and CI systems more reusable by others, as well.

The final comment was that whatever happens, some people will complain about it. But that shouldn't make the project afraid to make a change.