LWN: Comments on "Convergence in the pip and conda worlds?"

Convergence in the pip and conda worlds?

fung1 — Thu, 09 Feb 2023 15:29:32 +0000

The "Authority" in "Python Packaging Authority" (PyPA) was originally meant as a joke, highlighting the lack of control that group is effectively able to exert. It's gotten increasing recognition from the core CPython developers and SC since then, but at the end of the day it's still only a self-identifying collective of people developing interrelated packaging solutions for a large segment of the Python packaging ecosystem, and the opinions expressed by folks who count themselves as a part of that collective don't necessarily the reflect those of the CPython project, even if they're probably some of the most visible and easiest opinions to find sometimes referred to by official CPython documentation. They have, for example, fairly recently (in Python lifetime terms) gotten Pip included directly in builds of the standard library shipped as part of CPython, which has increased the perception of official status for it.

A big part of why Pip is not viewed as "the official solution" to package installation is that PyPA has been striving to reinvent packaging tools so that they're based on published standards and specifications rather than a de facto "whatever this tool does is the standard" approach. This means even something as seemingly ubiquitous as Pip is supposed to just be one possible implementation of those standards, in order to allow for fair competition from anyone else who wants to develop an interoperable replacement. Officially blessing one solution is viewed by many as favoritism, making it very hard if not nearly impossible for any alternative to gain sufficient mind-share. The initial standards are being derived from what these tools do in order to not cause them to suddenly be non-compliant, but with the idea that as people want to implement sweeping new features or scope changes in the tools, they need to get them reflected in reviewed and agreed-upon published standards first.

Pip's maintainers see it as being responsible for meeting many of these use cases, but that doesn't mean the responsibility is placed on it by the CPython project. Rather, it's a scope the maintainers have chosen to give it, mostly in order to maintain backwards compatibility for users of earlier versions and predecessors like easy_install. They technically also have the "freedom to specialize" but they prefer not to exercise it, as that would leave many current users of its more general approach in the lurch.

Convergence in the pip and conda worlds?

mboisson — Thu, 09 Feb 2023 14:56:47 +0000

I think that the things conda "blames" pip for, are the same things that HPC cluster administrators blame conda for, if you replace "conda" by "system", i.e.

* don’t vendor things available as <conda=>system> packages
* do include additional dependencies for those things
* link against the import libraries/headers/options used for the matching <conda=>system> builds of dependencies

On cluster environments, we actually ask our users to *not* use conda, in large part due to these reasons.
https://docs.alliancecan.ca/wiki/Anaconda/en

conda is more like yum/apt than it is like pip, and that does not play well on a cluster, but our users keep coming to us wanting to use conda instead of pip (and then we show them that pip works better on clusters).

Convergence in the pip and conda worlds?

qwertyface — Wed, 08 Feb 2023 15:50:27 +0000

Just to say, that pypackaging-native site is a fantastic exploration of the issues. None of this is simple!

Before reading it, I'd drafted a comment saying that conda absolutely shouldn't become a standard solution to distributing Python packages. It has most of the scope of a Linux distribution, but without the helpful restriction of being Linux only. I still mostly believe that, but now think something like it might always be necessary in some circumstances.

Convergence in the pip and conda worlds?

NYKevin — Mon, 06 Feb 2023 00:56:27 +0000

With some Googling, I dug up these links, which I think might be relevant:

https://chriswarrick.com/blog/2018/07/17/pipenv-promises-...
https://old.reddit.com/r/Python/comments/8jd6aq/why_is_pi...

I cannot speak to the accuracy of anything said on either of those pages, because I have never used pipenv myself and did not follow this issue at the time. Note also that both links are several years old, and the situation may have changed since 2018.

Convergence in the pip and conda worlds?

intelfx — Fri, 03 Feb 2023 20:15:30 +0000

> I'm now becoming more and more inclined to disregard any recommendations from PyPA, They burned all my goodwill with the Pipenv fiasco <...>

What was the "Pipenv fiasco"?

Convergence in the pip and conda worlds?

cortana — Fri, 03 Feb 2023 10:22:25 +0000

The belief that pip is official is part of the problem

I'm just in despair at this point. The official Install Python Modules documentation says quite clearly:

pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers.

I'm now becoming more and more inclined to disregard any recommendations from PyPA, They burned all my goodwill with the Pipenv fiasco, and now this. These days I've settled on a combination of Poetry and micropipenv as the last annoying Python packaging tools, but I expect before too long, something else will force me to move on to newer tools with names generated by picking a random combination of terms from the list [py, dist, package, setup, env, utils, virt, build, virtual, wheel, v, pack, file, ...].

Convergence in the pip and conda worlds?

smitty_one_each — Fri, 03 Feb 2023 02:05:38 +0000

Hopefully 3.13 can be the version where python make a point of making packaging more pythonic.

Convergence in the pip and conda worlds?

gerdesj — Thu, 02 Feb 2023 10:45:12 +0000

"I'm Brian, and so's my wife" ...

Convergence in the pip and conda worlds?

dottedmag — Thu, 02 Feb 2023 10:38:54 +0000

infrastructures -> ecosystems

Convergence in the pip and conda worlds?

dottedmag — Thu, 02 Feb 2023 10:38:27 +0000

The tricky part will be the specification.

Conda, Nix, Debian and other closed infrastructures can do it by controlling the whole set of packages.

However Python packages are exposed to the wild world, where even "I need a C compiler" won't do, as there are many C compilers with different supported C versions, bugs, incompatible extensions, CLI interfaces, target platforms, ideas about ABI etc.

Convergence in the pip and conda worlds?

kleptog — Thu, 02 Feb 2023 10:05:20 +0000

On the one hand we have :

> pip is one of the "batteries included" with Python so it is often seen as the "official" packaging solution, even though the PyPA does not necessarily see it that way.

Which surprised me. If pip isn't official then no Python packaging solution is official. Which is just bizarre. It certainly surprised everyone I suggested it to.

On the other hand we have:

> because pip is responsible for supporting all python environments – venv, distro, whatever – while conda is free to specialize

If pip is responsible to for supporting all environments, then that would imply it's official. Otherwise this responsibility would not exist.

ISTM that the quickest win is to standardise a way for packages to declare external dependencies in normal python packages. Even if pip doesn't use it, it opens the way for other tools to use it.