Automated testing of CPython patches

Kushal Das is concerned about the number of patches in the CPython review queue. He has done some work to automate the testing of those patches, which he presented at the 2016 Python Language Summit.

There are too many patches and too few contributors to review them, he said. His first patch took a year and a half to get reviewed and merged—then it broke the next day. So he started thinking about how the patches could be tested automatically, which would help reviewers by giving them some assurance that the patches will build and run.

He started looking for systems to run the tests on. The Python Software Foundation infrastructure team was not particularly thrilled with the idea of running random code from the internet. Beyond that, building from source and running all the tests took longer than fifteen minutes, so the Travis CI system he was testing on would time out and kill the process.

He started asking around and soon heard about ci.centos.org, which is a Jenkins-based system with fairly beefy bare-metal servers. It takes less than five minutes to build and test on those systems. The CentOS project is "really happy" to have people using its continuous integration servers, Das said.

He has written an IRC bot that listens for announcements of bug-fix patches that are posted (which come from a different bot). It then builds and tests the main branch with the patch applied and could post a comment in the bug report. When the GitHub migration is complete, he will update his code to use that bug tracker.

Barry Warsaw noted that patches often come in without tests or without correct tests. He suggested adding something like diff-cover into the mix to determine whether the tests are covering the changed lines, which Das said could be done. It would be nice to be able to test Windows and OS X as well, an attendee said, but even without that, an automated system like this is an improvement over what is done now.