PyTorch and the PyPI supply chain

Posted Jan 14, 2023 9:29 UTC (Sat) by cyperpunks (subscriber, #39406)
Parent article: PyTorch and the PyPI supply chain

The amount of work needed and the review done for a module to be part of Python Standard Library seems to very high, however to publish something to PyPI the requirements is more or less trivial.

Maybe it's a path forward is to split PyPI in curated/blessed part and a free for all section? The blessed part will move somewhat slower, but much faster than PSL.

PyTorch and the PyPI supply chain

Posted Jan 14, 2023 11:11 UTC (Sat) by amacater (subscriber, #790) [Link]

Split PyPI into two - a curated part and a non-curated part - and you might as well have distributions doing a competent job of curation on a subset of packages. There's a reason that some of us have used distributions for >25 years and look on non-curated software with amusement and horror.

And yes - I'm frankly amazed how many language / package distribution mechanisms for various operating systems have effectively reimplemented apt poorly.

PyTorch and the PyPI supply chain

Posted Jan 16, 2023 0:24 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (3 responses)

Maybe this is unfair, but the impression I've gotten from the discussions on here is that anything less than full speed ahead will upset a lot of developers. At the very least, each developer has their own idea about how much delay for quality control is acceptable, and any delay at all will upset some people. Whatever choice you make will make some people unhappy there's too much delay and others unhappy there isn't enough QC.

PyTorch and the PyPI supply chain

Posted Jan 16, 2023 0:58 UTC (Mon) by Wol (subscriber, #4433) [Link]

This is where you NEED your Benevolent Dictator For Life.

Sod full speed ahead. Sod quality control. Just take a step back. Think about what you're doing. INVEST TIME IN DESIGN. Then you *won't* *need* so much quality control. Then "full speed ahead" will feel like a tortoise (and won't do a Torrey Canyon). Then you'll end up with twice the quality in half the time.

The problem is that, without someone who has the power to knock heads together, having a sensible design discussion can be incredibly difficult. It just takes a couple of people who think their needs are the greatest, and are determined make their voice heard over everyone else, and things will implode.

Cheers,
Wol

PyTorch and the PyPI supply chain

Posted Jan 16, 2023 10:05 UTC (Mon) by kleptog (subscriber, #1183) [Link] (1 responses)

> Maybe this is unfair, but the impression I've gotten from the discussions on here is that anything less than full speed ahead will upset a lot of developers.

I read that here a lot too, but I've yet to meet such a developer in real life. Sure, you have junior developers that wonder what the point is. When they've spent a week trying to untangle dependencies to get the buildbot to pass again they suddenly appreciate the virtue of pinning versions.

Untangling package dependencies to find a working combination is one of the least interesting jobs there is.

PyTorch and the PyPI supply chain

Posted Jan 16, 2023 11:37 UTC (Mon) by farnz (subscriber, #17727) [Link]

The thing that appears as "full speed ahead" is not that all developers want to be on the latest version of everything, but that the combined effect of all developers wanting their pet dependency to be on the latest version (which adds a feature they need, or a bugfix that affects their product's security) is "full speed ahead".

Basically, anything other than "we only accept dependencies in the oldest distribution release in extended support" (RHEL6, for example) ends up looking like "full speed ahead" in discussions, because no matter how carefully you consider your update plans, there will be someone who perceives your decision to update a minimum supported dependency version as "moving too fast".