Fallout from the Python certificate verification change

Up until fairly recently, the Python standard library did not do any checking of HTTPS certificates, which allows man-in-the-middle attacks, but that all changed with PEP 476. The ssl module changed to check those certificates by default in both Python 2.7.x and 3.4.x, but that has led to compatibility problems—especially for distributors of enterprise software. Smoothing the transition for enterprise customers (and others who are change-averse) is the subject of PEP 493. But the decision on whether to accept the PEP or not seems to have gotten lost in the shuffle a bit—which proponents are now trying to rectify.

The basic problem is that some Python users are relying on the earlier behavior—knowingly or not—so changing it can come as a great unpleasant surprise. If an organization is using self-signed certificates or those signed by an entity that is not in all of the certificate root stores, it will work just "fine" for Python versions that don't implement PEP 476. It is, of course, rather insecure, as man-in-the-middle attacks can readily be performed on the connections, but there will be no errors.

Python distributors are either releasing Python 2.7.9, which has the change, or backporting that feature into earlier Python versions that they distribute. That sets their customers up for errors if they are using "bad" certificates. Since part of what those customers pay for is protection against unpleasant surprises, it makes sense that two developers from Red Hat would be joined by Marc-André Lemburg of eGenix (a Python-focused company) in creating the PEP. Their customers will need ways to override the default behavior in some cases, and the entire community will benefit if the solutions across multiple distributions are the same.

There are two separate mechanisms proposed in the version of the PEP posted by Nick Coghlan on November 11. For those wishing to provide a per-application opt out of the certificate checking, the recommendation is to add a PYTHONHTTPSVERIFY environment variable; when set to zero, it would disable certificate checking in the ssl module. For distributions that want to provide a system-wide setting, the PEP recommends using a configuration file (/etc/python/cert-verification.cfg on Unix-like systems) that can enable or disable certificate checking; it also has a setting that explicitly defers the decision to the distributor ("platform_default").

PEP 493 was originally proposed as an informational PEP that just made recommendations for distributors, but that has evolved in the last few weeks. In posting the PEP, Coghlan was asking for a pronouncement (decision) on it, but it quickly became clear that finding someone to make that decision could be troublesome. Guido van Rossum did not feel he was the right one to review it, so he asked that a benevolent dictator for life (BDFL) delegate be found. That turned out to be tricky, as some of the more likely candidates also work for Red Hat, which might make the process seem to be just a rubber-stamping of the company's decision, as Van Rossum pointed out:

The accident Van Rossum refers to is that Coghlan had admittedly dropped the ball on getting the PEP accepted or rejected and that Red Hat was close to releasing RHEL 7.2, which adopted parts of the PEP. In fact, as Coghlan noted, 7.2 was released on November 19 and it implemented the file-based configuration mechanism from the PEP.

Finding a BDFL-delegate is hard since there are not that many qualified developers and several were not able to do it for various reasons, so that part has not been resolved. But the discussion soon turned to other distributions and it started to become clear that an informational PEP, with patches that must be carried by each distribution, is not really what was wanted. Barry Warsaw, who works on Python for Debian and Ubuntu, thought that the suggestions in PEP 493 should instead be added to the standard library as distributed by python.org:

Robert Collins agreed: "a PEP telling distributors to patch the standard library is really distasteful to me". While Coghlan is sympathetic to that view, he pointed out that the distributors have other ideas:

But Warsaw noted that the presence or absence of code in the standard library really doesn't change anything with respect to what the distributors do:

Those arguments seems to have won the day, as Coghlan changed the PEP from "informational" to "standards track" and posted a summary of the changes in his new draft of the PEP. It targets 2.7.12 for implementing the changes; that release will likely come sometime in mid-2016.

At the time that PEP 476 changed the ssl module to check certificates by default, there was talk of adding a way to opt out, but it was controversial. That PEP eventually went out with a suggestion that those who needed to globally opt out perform a "monkeypatch" on the ssl module. That is not a particularly popular option and is what led to the need, at least in the eyes of some, for PEP 493. Security purists tend to believe that certificate validation should always be done, but more pragmatic developers recognize there are environments where that simply may not be possible. Whether the purists or pragmatists prevail on this topic for the language as a whole is still up in the air.