PSF: New pip resolver to roll out this year

[Posted March 24, 2020 by ris]

The Python Software Foundation blog looks at some changes to pip, the Python Package installer, in the process of developing a new resolver. The new resolver will reduce inconsistency and be stricter, refusing to install two packages with incompatible requirements.

Also, this is a major change to a key part of pip - it's quite possible there will initially be bugs. We would like to make sure that those get caught before people start using the new version in production. [...]

We recognize that everyone's work is being disrupted by the COVID-19 pandemic, and that many data scientists and medical researchers use Python and pip in their work. We want to make the upgrade process as smooth and bug-free as possible for our users; if you can help us, you'll be helping each other.

PSF: New pip resolver to roll out this year

Posted Mar 25, 2020 3:29 UTC (Wed) by david.a.wheeler (subscriber, #72896) [Link] (7 responses)

In almost all other ecosystems, you don't post a breaking change until you have posted a version that gives a warning first and giving people time to see the warning.

Why can't there be a version of pip that warns about this, let people see the warning in production, and then much later have another version that finally enforces this breaking change?

I'm not opposing their final goal, it's the way they're getting there. I just don't understand why they can't handle a breaking change like everyone else.

Maybe I'm misunderstanding something? I hope so. It wouldn't be the first time I misunderstand something!

PSF: New pip resolver to roll out this year

Posted Mar 25, 2020 5:39 UTC (Wed) by viiru (subscriber, #53129) [Link]

> Why can't there be a version of pip that warns about this, let people see the warning in production, and then
> much later have another version that finally enforces this breaking change?

In my understanding pip has always warned about this. I don't believe I've seen a version of pip that doesn't.

PSF: New pip resolver to roll out this year

Posted Mar 25, 2020 9:06 UTC (Wed) by vstinner (subscriber, #42675) [Link] (3 responses)

In my experience, even if there is a warning for 10 years, users ignore it and then complain that nobody warned them when the incompatible change is finally made :-) Users only start to pay attention when their code stops working...

PSF: New pip resolver to roll out this year

Posted Mar 25, 2020 21:06 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (2 responses)

The other point is that their code very likely wasn't working to begin with, because they were installing incompatible packages. But now it'll fail loudly, at install time, instead of quietly corrupting your data and/or crashing at runtime.

PSF: New pip resolver to roll out this year

Posted Mar 26, 2020 15:08 UTC (Thu) by Bluehorn (subscriber, #17484) [Link] (1 responses)

I think we still actually rely on this bug in pip to compensate for bugs in library packages.

Because currently a library package can required a specific version 42.1 of some other library libflyingpigs which means you can't install 42.2 with a critical bugfix into your application without getting rid of libflyingpigs first.

Currently the toplevel requirements seem to win, triggering a warning the libflyingpigs would like to have 42.1 but is getting 42.2. But our application is working.

If this is not allowed anymore we will have to fork the problematic library and loosen the dependency.

PSF: New pip resolver to roll out this year

Posted Apr 1, 2020 12:25 UTC (Wed) by mgedmin (subscriber, #34497) [Link]

> Because currently a library package can required a specific version 42.1 of some other library libflyingpigs which means you can't install 42.2 with a critical bugfix into your application without getting rid of libflyingpigs first.

But you can install a 42.1+but.really.42.2 without touching libflyingpigs, while you file a bug for libflyingpigs and ask it nicely not to pin specific versions of dependencies.

See https://www.python.org/dev/peps/pep-0440/#local-version-i....

PSF: New pip resolver to roll out this year

Posted Mar 25, 2020 11:02 UTC (Wed) by thumperward (guest, #34368) [Link]

It's not a breaking change. It's just a large one which is likely to have bugs. The status quo is that pip will quite frequently trash one's package configuration as it happily installs broken combinations of new packages, and this is not something that anyone wants to continue to put up with.

Quite frankly I'm actually surprised they're going ahead with just making the change, though. With pretty much anything else in current Python development, the result would be a huge flame war, two competing implementations, and a final solution which was disabled by default and hidden behind an option.

PSF: New pip resolver to roll out this year

Posted Mar 27, 2020 17:17 UTC (Fri) by pradyunsg (guest, #116784) [Link]

Hi there! I'm a maintainer of pip, and one of the developers working on this. I've personally been working on improving dependency resolution in the Python ecosystem for 3-ish years now.

Users have been getting warnings regarding this problem for a while now: I'd added a warning to pip 10.0 (i.e. early 2018) to warn users about broken dependencies in the environment, and I know that a lot of users have seen these warnings.

We will provide alpha and beta releases as the work progresses, where users can opt-in to the new resolver that's under development and provide feedback on how it all works for them. Our goal is to minimize disruption for end users when we switch to the new resolver, so that (hopefully) most users don't have to deal with ant breakage; and we minimize the "pain" for users that do have breakages.

We have user experience experts (i.e. people who are much better than me at conducting user research, user testing and more) and a project manager on our team (i.e. a person much better than me at communication, coordinating work and more) whose skills are *very* valuable for managing a major change to a foundational project in the ecosystem.

The full post (that this excerpt is taken from) goes into further details of how users can help us *right now* as well as how to sign up for the beta testing when we are ready for that. Hopefully this answers your questions, addresses your concerns and provides useful context. :)