Namespaces for the Python Package Index

The Python packaging picture is generally a bit murky; there are lots of different stakeholders, with disparate wishes and needs, which all adds up to a fairly large set of multi-faceted problems. Back in the first three months of the year, we looked at various discussions around packaging, some of which are still ongoing. A packaging summit was held at PyCon 2023 to bring some of the participants of those discussions together in one room. One of its sessions was on adding a namespaces feature to the Python Package Index (PyPI). It provides a look into some of the difficulties that can arise, especially when trying to accommodate a long legacy of existing practices, which is often a millstone around the neck of those trying to make packaging improvements.

PyPI namespaces

PyPI has long been the go-to source for various kinds of Python modules, libraries, and applications. One of the PyPI maintainers, Dustin Ingram, wanted to discuss some ideas for providing namespaces for PyPI packages. The basic problem is that everyone is sharing a single global namespace on PyPI, he said. There are around 250,000 packages on PyPI currently, but many of them are old packages that may not really be in use any longer; overall, there is a lot of contention for package names.

The PEP 541 ("Package Index Name Retention") process for handling name conflicts "does not really scale", he said. It provides a means to acquire the package name from an abandoned or unmaintained project, but it is an involved process that requires a fair amount of manual review. The PyPI administrators have not prioritized that work, so there is a big backlog. Meanwhile, the task also requires "extremely high trust permissions" in order to transfer the ownership of a package, so it cannot just be handed off to new volunteers.

Typosquatting on package names is another problem that PyPI faces. There is code in place to prevent people from registering names that are confusingly similar to existing package names, but it is restrictive, of necessity. So it stops people who are trying to do legitimate things, which is not desirable.

Meanwhile, there are companies and others that want to be able to restrict the prefixes of package names so that packages coming from an "official source" can be distinguished from those coming from elsewhere. An easy way to handle that would be to hand out a namespace that only the organization can publish packages to. During PyCon, the PyPI project announced the addition of organizations for the index. The idea is that companies, projects, and, well, organizations can register with PyPI in order to be able to maintain a collection of projects as part of a single PyPI page, such as the one for the Python Cryptographic Authority (PyCA). But support for namespaces is not part of the new feature.

When Ingram mentioned the new organizations feature, it was met with a round of applause from the two dozen or so summit attendees. Organizations was the number one feature request for PyPI, previously, but now that it has been implemented, PyPI namespaces moves up into that slot, he said. It is not a surprise that organizations see that as a means to ensure that it is clear when code is coming from them.

"This has been done before", he said. One way would be to have GitHub-style namespaces, so that users and organizations could have the same project names that do not collide. Also, the npm ecosystem for JavaScript implemented a namespace feature in 2015 when it was at a similar size to what PyPI is today. There may be some lessons to be learned there. He is not close enough to that community to say whether it has been a success, but it does seem to be a widely used feature at this point.

One important requirement for the PyPI namespaces feature is that it not cause any breaking changes. Current installers should continue to function as they do today even if the target package has moved into a namespace. While he was speaking for himself, he believes that the other PyPI administrators are generally of the same mind. He wondered what attendees thought about the feature, whether it should be pursued, and, if so, how to make it happen.

Discussion

Former PyPI project manager (PM) Sumana Harihareswara (who wrote about PyPI for LWN back in 2018) wondered if Shamika Mohanan, the current PM, had gathered information on use cases and user-experience expectations for the namespaces feature as Mohanan had done for the organizations feature. Mohanan was not at the summit, but Ingram said that all of the work done so far has been targeting the organizations feature, not namespaces, at least yet.

Harihareswara said that, during the pip overhaul in 2020, the team did a lot of user-experience research, including user testing and interviews. She suggested doing that kind of research for the namespaces feature because understanding the mental models of the users, developers, and companies will result in a better feature. Ingram agreed and noted that he did not expect that the feature would be completed quickly; like organizations, it is a complicated feature that will require quite a bit of work. He expects that it will require funding and thinks that the funding should include money for research of that sort.

There are already namespaces, of a sort, in use; for example, pytest has a convention that the names of its plugins start with pytest-, as pytest developer Phebe Polk pointed out. Having namespace support in PyPI would be nice, she said, but it is unclear how the feature would work with existing conventions like that; there may also be conflicts with names that are being used in private PyPI mirrors used internally. One attendee suggested that registering empty namespaces, in order to reserve them for internal use, should be supported.

Bernát Gábor said that he works at Bloomberg, which has a lot of internal packages, but he also maintains public packages as well. He wondered about adding existing packages into a new namespace once it gets created. Companies will probably want to own everything under their namespaces, but open-source projects may want to allow other projects to publish packages under their namespaces. Ingram thought that the owner of the namespace would be able to make those kinds of decisions; he said that following the GitHub model makes sense.

The large companies already have problems with package names that refer to their names or products, but are not official packages, such as tools for NVIDIA or Amazon Web Services (AWS), Peter Wang said. "Names are a hard problem, we all know that". Organizations are going to expect that they completely control their namespace, but he wondered about packages from others that add some functionality on top of the official package.

Many companies and other organizations have their own internal repositories and mirrors that they use, so there is a question of how that interacts with the PyPI namespaces, he continued. How the resolution order will be determined when searching for a package, for example. To his eye it seems like a federated name system of some sort, perhaps modeled on the Domain Name System (DNS) or public-key infrastructure (PKI), may be needed. That may also mesh well with future plans for package signing in order to address supply-chain-security concerns.

Ingram agreed that package names that refer to another organization (e.g. a google-* package from a FOSS developer) are going to be problematic. He thinks it will be important to make a clear distinction between packages that are in a specific namespace versus those that have a prefix or other part of the name that refers to an organization. Currently, there are packages that have prefixes that do accurately identify the organization behind them, others that are unaffiliated with the prefixed organization, while still others that have no real naming convention at all. It is likely that new syntax will be needed to make the distinction clear and that a mapping layer will be required to map between names outside and inside of namespaces.

An attendee asked how many of those present were familiar with how npm had added its namespaces. Only about five people raised their hands, so it probably makes sense to put together a report on how that was done, he said. That will help inform any decisions based on the npm experience.

Clashes do not only exist at the PyPI package-name level, Toshio Kuratomi said, there is also the question of what name will be used in the Python import statement. The problem already exists today, but he thinks it will get worse with namespaces. For example, Fedora had a mock package at one time, but it was not on PyPI, so a mock package that was added to the index effectively pushed Fedora's package aside. Ingram agreed that mismatches between the package name and the imported module name are a real problem, but that problem already exists, so it is a bit out of scope for a discussion of the namespaces feature.

Organizations may wish to have aliases so that they can, essentially, typosquat themselves, an attendee said. For example, Meta may want to have aliases for its name, Instagram, Facebook, and other variants of those names. Ingram asked if being able to reserve multiple empty repositories would suffice. That would take care of the security problem, the attendee said, but not the usability problem; users may want to get the same package under the canonical name and its variants. Ingram agreed that some kind of aliasing would need to be incorporated into the feature.

Another attendee asked about whether there are efforts to target heavy users to get them to help finance PyPI and its infrastructure. Ingram said that while there are some heavy users, they are not really causing problems for PyPI right now. There are some users, for example those providing large GPU binaries for TensorFlow models, who may be heading in that direction, but the PyPI administrators are working with those projects to ensure that the problem does not get out of control. There are also already some protections in place to prevent overconsumption of PyPI's resources.

Harihareswara suggested that the financial tradeoffs in various potential implementations for namespaces be clearly aired in the upcoming discussions. If there are, for example, somewhat suboptimal designs that would end up saving an enormous amount of time (and money), that should be factored into the decision-making. Knowing what budget is available for the project will also help guide the community in making the appropriate choices. Ingram agreed, noting that the person who fills the newly announced position for a PyPI Safety and Security Engineer will likely have a role to play in the design of the feature as well. PyPI namespaces are pretty clearly part of the safety and security story for the package index and the language as a whole.

[I would like to thank LWN subscribers for supporting my travel to Salt Lake City for PyCon.]