Progress in wrangling the Python C API

There has been a lot of action for the Python C API in the last month or so—much of it organizational in nature. As predicted in our late September article on using the "limited" C API in the standard library, the core developer sprint in October was the scene of some discussions about the API and the plans for it. Out of those discussions have come two PEPs, one of which describes the API, its purposes, strengths, and weaknesses, while the other would establish a C API working group to coordinate and oversee the development and maintenance of it.

Working group

In mid-October, Guido van Rossum announced PEP 731 ("C API Working Group Charter") as the first visible outcome of the meetings at the sprint. If approved by the steering council, it would establish a working group of the five PEP authors (Van Rossum, Petr Viktorin, Victor Stinner, Steve Dower, and Irit Katriel) to oversee the C API, and to steer it in ways that are analogous to what the council does for Python. There are multiple contentious issues surrounding the API, the PEP states, so there is a need for a dedicated group of core developers to work through them: "The general feeling is that there are too many stakeholders, proposals, requirements, constraints, and conventions, to make progress without having a small trusted group of deciders."

Some presentations on the C API at the 2023 Python language summit led to a site for gathering problems with the API. It has collected more than 60 different accounts of problems that people are experiencing using, maintaining, and extending the API. The PEP gives a rough summary of the kinds of problems that would be under the purview of the working group:

Despite many discussions and in-person meetings at core developer sprints and Language Summits, and a thorough inventory of the problems and stakeholders of the C API, no consensus has been reached about many contentious issues, including, but not limited to:

• Conventions for designing new API functions;
• How to deal with compatibility;
• What's the best strategy for handling errors;
• The future of the Stable ABI and the Limited API;
• Whether to switch to a handle-based API convention (and how).

Beyond just gathering problems, though, the effort has expanded to gather potential solutions in two other repositories: API evolution for relatively non-controversial, common-sense changes, and API revolution "for radical or controversial API changes" that will definitely need to go through the PEP process.

The reaction to the announcement was generally positive, though there were suggestions that the working group should include some other stakeholders, such as developers of C extensions. The PEP states that the working group will be made up of at least three core developers, but that "members should consider the needs of the various stakeholders carefully". The PEP notes that the group serves at the pleasure of the steering council; thus the council provides a check on the actions of the group. As Dower noted, the stakeholders are not being left out:

Since the WG would be proposing changes that are only directly binding on the core development team, I'm okay with the core developer requirement.

If the WG doesn't appear to be soliciting contributions from prominent users of the API and factoring them in, that's a reason to go to the steering council with a complaint.

Being directly on the WG isn't a prerequisite to contribute. It's just a burden of having to take responsibility for the decisions and how they impact other competing interests.

Analysis

On November 1, Katriel announced an outcome from this year's language summit: PEP 733 ("An Evaluation of Python's Public C API"). It has nearly 30 authors that reads like a "who's who" of Python core developers—perhaps reflecting the attendees of the summit—but was coordinated by Katriel. Effectively, it is a summary of the problems that were collected, categorized into nine separate problem areas, along with a look at the various stakeholders and their requirements. Beyond that, it also describes some of the history of the API, its purposes, and its strengths ("to make sure that they are preserved"):

As mentioned in the introduction, the C API enabled the development and growth of the Python ecosystem over the last three decades, while evolving to support use cases that it was not originally designed for. This track record in itself is indication of how effective and valuable it has been.

For one thing, the stakeholders have diversified over the years. The C API first came about "as the internal interface between CPython's interpreter and the Python layer", but it was later exposed for third-party developers to use for extending CPython and to embed the interpreter in their own applications. Since then, new Python implementations have arisen that also need to use the C API; in addition, multiple projects seek to provide bindings or a better API for Python extensions in C (e.g. Cython), Rust (e.g. PyO3), and other languages and frameworks. Those projects use the C API in various ways, as well.

The overarching problem that has been identified with the C API is the difficulty of maintaining and evolving it, in part due to all of the different stakeholders and their competing needs. A process for incremental evolution, with deprecations and eventual removals of some parts of the API, could be a possible way forward; another option is periodic upheaval of the API via redesigns "each of which learns from the mistakes of the past and is not shackled by backwards compatibility requirements (in the meantime, new API elements may be added, but nothing can ever be removed)". Between those two extremes is a compromise approach that fixes "issues which are easy or important enough to tackle incrementally, and leaving others alone".

But the CPython core developers have different opinions on how to change the API, which is "an ongoing source of disagreements". So a fundamental framework for changes needs to come about:

Any new C API needs to come with a clear decision about the model that its maintenance will follow, as well as the technical and organizational processes by which this will work.

If the model does include provisions for incremental evolution of the API, it will include processes for managing the impact of the change on users [Issue 60], perhaps through introducing an external backwards compatibility module [Issue 62], or a new API tier of "blessed" functions [Issue 55].

Another problem area is the specification of the API, or, in truth, the lack thereof. Currently it is defined as "whatever CPython does" in a particular version; the documentation provides some amount of specification, but it is insufficient to verify any of the different API levels. That leads to unexpected changes to the API between versions, for one thing. The API also exposes more of the internals of CPython than is intended—or desired—and is C-specific, so other languages need to parse and handle C language constructs of various sorts.

The different API levels (or tiers), "which provide different tradeoffs of stability vs API evolution, and sometimes performance", are also a source of problems. The stable ABI, which is used by binary extensions that are built using the limited version of the C API, is "incomplete and not widely adopted"; there are different opinions of whether it is worth keeping at all, but, if it is kept, it needs to support multiple ABI versions in a single binary. The limited API needs some changes as well. Meanwhile, there are inconsistencies in the way that CPython private functions are named and there may be a need to add an "unsafe" tier that provides functions that remove error checking for performance purposes.

Most of the other categories listed in the PEP cover choices that were made in parts of the API, in particular, object reference management, object creation, type definition, and error handling, that are now seen as sub-optimal. Evolving (or replacing) those is desired, but that will have to be worked out once the overall maintenance scheme is determined. There are a handful of implementation flaws and some missing features that need attention as well.

As can be seen, there is a lot for the working group (and steering council) to address with regard to the C API. First up, is whether the council is ready to approve the working group charter; given that the council has already indicated that it is in favor of the idea, it seems likely that approval will come fairly quickly.

Current C API changes

Meanwhile, though, some work has been going on for the Python 3.13 release that is due next year. In fact, at the end of October, Stefan Behnel raised some complaints about the large amount of changes that appeared in the first 3.13 alpha release:

Hundreds of functions were removed, renamed, replaced with different functions. Header files were removed. Names were removed from header files. Macros were changed, even a few publicly documented ones. And the default answer to "how do I replace this usage" seems to be "we'll try to design a blessed replacement in time".

The changes were extensive and disruptive enough that he (provocatively) wondered if the release should be called "Python 4". But, as Jean Abou Samra pointed out, this was all part of Stinner's longstanding plan to clarify the public versus private C API. It is, however, clearly disruptive, so others joined Behnel in thinking that things were moving too quickly.

Stinner maintains that the situation is manageable and that he is planning to devote much of his time over the next few months toward getting extensions working. He has marked many functions as private, which caused the breakage that Behnel and others encountered, but plans to "public-ize" various API elements as needed to support the existing C API users so that all of them are working by the time the first 3.13 beta is released in May.

Stinner opened a GitHub issue to track the problems. As can be seen there, opinions differ on how to address the problems that arise; Stinner would like to fix them one-by-one, but others think reverting the changes makes more sense. One complicating factor, as steering-council member Gregory P. Smith noted, is that there is another big development going on in the 3.13 development tree: removing the global interpreter lock (GIL) from the interpreter. Those changes are described in PEP 703 ("Making the Global Interpreter Lock Optional in CPython") and it is important not to impede progress in that area:

We need to treat 3.13 as a more special than usual release and aim to minimize compatibility headaches for existing project code. That way more things that build and run on 3.12 build can run on 3.13 as is or with minimal work.

This will enable ecosystem code owners to focus on the bigger picture task of enabling existing code to be built and tested on an experimental pep703 free-threading build rather than having a pile of unrelated cleanup trivia blocking that.

One senses that a directional shift in Stinner's current C API work may be in the offing; he did point out that the week-old Cython 3.0.5 release has preliminary support for 3.13-alpha1, which is an indication that his plan is generally working. One thing that will not be shifting, however, is the version to Python 4—ever, according to Smith. Should the major version of Python need to change at some point, it seems that four "shalt thou not count", as with The Holy Hand Grenade of Antioch. Five, of course, should be "right out", at least according to Monty Python—for Python, the language, though, we will just have to wait and see.