PyTorch and the PyPI supply chain [LWN.net]

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 2:47 UTC (Thu) by mathstuf (subscriber, #69389)
In reply to: PyTorch and the PyPI supply chain by koh
Parent article: PyTorch and the PyPI supply chain

> I keep coming back to the question why every language needs their own package manager with the usual set of problems to (a) discover and (b) solve in incompatible manners...

Because the alternative is waiting for distros to repackage a hundred thousand random project repos? Add in Nix, vcpkg, chocolatey, HomeBrew, etc.

Let's say I'm working on a project. I discover that I can split a new library out of it. What do I do? I make a repo (or directory; many language package managers don't care that much) and publish it. Users can upgrade to this version just fine today. If I need to wait for…something to happen elsewhere, my tool is stuck in out-of-date versions until someone picks up the ball and adds this new package.

Sure, you could say "just use what is in your distro", but that ignores reality. People want new compilers, new development tools, etc. These end up pulling in the same things the distro wants to provide, but in newer, incompatible versions. What are you to do? Uproot your distro when Debian turns out to be too slow?

I'm all for splitting out dependencies and using system copies when possible, but I can't link my development processes with Debian (or Arch, Fedora, etc.) release cycles. I've got work to do, you know? Far better to let developers pick their own distro sandbox they like playing in and letting them do development on top of it in a convenient manner.

to post comments

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 3:09 UTC (Thu) by bferrell (subscriber, #624) [Link] (2 responses)

No, devs want NIH shiny toys and don't want to take five minutes to discover "we been doing that 'this' way for years"

This kewl new app will ONLY use the cutting edge version of the language... But the dev has no clue to document this.

It's beginning to make RPM dependency hell look simple

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 11:44 UTC (Thu) by kleptog (subscriber, #1183) [Link] (1 responses)

The problem is not new features, it's the bugs. Sure, there are big commonly used projects like Django which are well tested, but there's a long tail of smaller packages for things that only a much smaller group of developers use. If you're a developer using one of these packages in a slightly unusual setting you find bugs.

No problem, create a patch, send it to the developer, they merge it, push a new minor release to PyPI and you can get on with it. In my experience, a month or two is the usual turnaround time. This fits in the release cycle, we just pause the ticket till the upstream release. Telling us to "use the version shipped by the distribution" is equivalent to saying "work around this bug for the next year or two". And it's not just one bug, it's several over several different packages. Eventually tracking which workarounds you're waiting for an upstream release becomes a significant amount of work.

Besides, workarounds are annoying, this is open source, we should be fixing the upstream packages, not working around the issues elsewhere.

I know projects with the strict rule that all packages must be installed from Debian. And it *almost* works. If the packages are missing features you can simply tell the product owner it's not possible yet. But there are always a few packages for which the Debian release is simply buggy, but such a corner case that only affects basically you it's not going to be updated there (because upstream has fixed it in a new version, and Debian isn't going to bump the version). So you end up making an exception for just these handful of packages (basically using py2deb). And hope it doesn't get too many.

The step from there to "just pull everything from PyPI with version/hash pinning" is very, very small.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 13:53 UTC (Thu) by farnz (subscriber, #17727) [Link]

And don't forget that predicting what a distro will have when you release your new version is hard. If you're going to release in April 2023, and something you depend upon has a necessary update released in December 2022, is that update going to be in the current stable Debian release when you make your release?

If you guess wrong, you end up in one of two sub-optimal situations:

If you assume the update won't be submitted by the Debian Developers in time to make it into Debian 12 (Bookworm), or that the release will slip into May, and then the update makes it in anyway, you're carrying sub-optimal code to handle the pre-update version of your dependency.
If you assume it will make it in, and then something causes Debian to have to delay the release, or if the update can't be put into Bookworm before the freeze kicks in without breaking something critical, your project now doesn't run on Debian stable on the day of release, because you're waiting for an update.

Bundling from a vendor source neatly sidesteps this - if Bookworm has the dependency version you need, then unbundling can be done then, while if it doesn't, no problem, you've got the vendored code. And then language repositories like PyPI make it simpler, because they're already working in terms of a dependency tree, not copied code, so you can look and go "aha, when I build the Debian package, I can unbundle libfoo, since Debian has the right version of libfoo already".

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 15:37 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (4 responses)

I don't know if waiting for the distributions to package everything is the right solution. I'm inclined to believe developers who say it's unlikely to work for them, if only because of the problem you highlight of there being too many libraries to package. I also believe, though, that the current solution of grabbing whatever is out there, trusting it's fine, and then acting surprised by supply chain attacks is also not working. It's just resulting in occasional spectacular failures rather than regular, boring unavailability of bug fixes and product enhancements.

What is needed is a system that provides some kind of real quality control, so developers can have a confidence the libraries they're using are what it says on the tin. This has the unfortunate side effect of slowing everything down for the QC step, but the alternative is occasionally getting pwned when attackers finally decide your system is worth attacking. Pretending everything is fine in an attempt to go as fast as possible is demonstrably not working.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 17:22 UTC (Thu) by kleptog (subscriber, #1183) [Link] (1 responses)

We need to be able to have quality control somewhere, the question is if PyPI is the organisation to do it. What I'm thinking is a kind of meta-index where different organisations can give a "trust rating" or something, and that I can filter on that in pip.

For example, it would be possible for someone to write a bot that checked if the contents of the wheels distributed by PyPI match the source in the indicated repository. The problem is there is nowhere to place this information in a way that is of any use. Or, it would be nice to straight up reject any package that has existed for less than 3 months. Or being able to namespace dependencies to ensure they come from the right repository.

Python is here paying the decision early on that no packaging/repository standard would be made and to let the community create one organically. It's biting back hard now. More recent languages did not repeat that mistake.

PS. Don't talk to me about solutions like Nexus which try to solve the problem on the client-side but don't really have any extra information to work with and so just end up adding an extra layer of frustration. Until the necessary information is available in machine readable form no client-side tooling can help.

PyTorch and the PyPI supply chain

Posted Jan 13, 2023 4:01 UTC (Fri) by pabs (subscriber, #43278) [Link]

I feel like distributed code review ala crev, along with reproducible and bootstrappable builds is the right model here.

https://github.com/crev-dev/
https://reproducible-builds.org/
https://bootstrappable.org/

PyTorch and the PyPI supply chain

Posted Jan 18, 2023 1:15 UTC (Wed) by hazmat (subscriber, #668) [Link] (1 responses)

re distros, too much plurality and too much complexity to get across the swathe when viewed in aggregate for a developer/publisher of a python dependency, ie language tools exist for a reason. i'd be willing to settle for simple signature upload and verification (sans gpg, say cosign) on the language repos.. but there has been a slow moving effort to try and make things better albeit glacial speeds.. https://blogs.vmware.com/opensource/2020/09/17/tuf-pypi-i... updates are scattered across internet/github issues / tulip chat logs. i do wonder if some of the ossf projects will help on the broader ecosystem. its unclear if we're not just creating another checkbox for enterprises sometimes vs actually moving the needle.

PyTorch and the PyPI supply chain

Posted Jan 18, 2023 15:34 UTC (Wed) by hazmat (subscriber, #668) [Link]

found the best link for current state at
https://github.com/pypi/warehouse/issues/10672