PyTorch and the PyPI supply chain [LWN.net]

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 1:13 UTC (Thu) by koh (subscriber, #101482)
Parent article: PyTorch and the PyPI supply chain

To me the solution is pretty simple and nothing that hasn't been implemented elsewhere: identify the repository in the dependency specification.

Granted, I'm using Gentoo since quite a while, so to me it feels natural to say, e.g., '>=sys-libs/glibc-2.32::gentoo' in order to give the constraints
- package 'sys-libs/glibc'
- version larger or equal to 2.32
- repository called 'gentoo' (locally)

In a non-centralized setting with "‑‑extra‑index‑url" there is no 'local' name/reference to a repository, but that shouldn't be a problem - at least on the technical level. The URLs are still managed centrally (for most of the internet for most of the time - that's another can of worms, though).

I keep coming back to the question why every language needs their own package manager with the usual set of problems to (a) discover and (b) solve in incompatible manners...

to post comments

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 2:47 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (8 responses)

> I keep coming back to the question why every language needs their own package manager with the usual set of problems to (a) discover and (b) solve in incompatible manners...

Because the alternative is waiting for distros to repackage a hundred thousand random project repos? Add in Nix, vcpkg, chocolatey, HomeBrew, etc.

Let's say I'm working on a project. I discover that I can split a new library out of it. What do I do? I make a repo (or directory; many language package managers don't care that much) and publish it. Users can upgrade to this version just fine today. If I need to wait for…something to happen elsewhere, my tool is stuck in out-of-date versions until someone picks up the ball and adds this new package.

Sure, you could say "just use what is in your distro", but that ignores reality. People want new compilers, new development tools, etc. These end up pulling in the same things the distro wants to provide, but in newer, incompatible versions. What are you to do? Uproot your distro when Debian turns out to be too slow?

I'm all for splitting out dependencies and using system copies when possible, but I can't link my development processes with Debian (or Arch, Fedora, etc.) release cycles. I've got work to do, you know? Far better to let developers pick their own distro sandbox they like playing in and letting them do development on top of it in a convenient manner.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 3:09 UTC (Thu) by bferrell (subscriber, #624) [Link] (2 responses)

No, devs want NIH shiny toys and don't want to take five minutes to discover "we been doing that 'this' way for years"

This kewl new app will ONLY use the cutting edge version of the language... But the dev has no clue to document this.

It's beginning to make RPM dependency hell look simple

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 11:44 UTC (Thu) by kleptog (subscriber, #1183) [Link] (1 responses)

The problem is not new features, it's the bugs. Sure, there are big commonly used projects like Django which are well tested, but there's a long tail of smaller packages for things that only a much smaller group of developers use. If you're a developer using one of these packages in a slightly unusual setting you find bugs.

No problem, create a patch, send it to the developer, they merge it, push a new minor release to PyPI and you can get on with it. In my experience, a month or two is the usual turnaround time. This fits in the release cycle, we just pause the ticket till the upstream release. Telling us to "use the version shipped by the distribution" is equivalent to saying "work around this bug for the next year or two". And it's not just one bug, it's several over several different packages. Eventually tracking which workarounds you're waiting for an upstream release becomes a significant amount of work.

Besides, workarounds are annoying, this is open source, we should be fixing the upstream packages, not working around the issues elsewhere.

I know projects with the strict rule that all packages must be installed from Debian. And it *almost* works. If the packages are missing features you can simply tell the product owner it's not possible yet. But there are always a few packages for which the Debian release is simply buggy, but such a corner case that only affects basically you it's not going to be updated there (because upstream has fixed it in a new version, and Debian isn't going to bump the version). So you end up making an exception for just these handful of packages (basically using py2deb). And hope it doesn't get too many.

The step from there to "just pull everything from PyPI with version/hash pinning" is very, very small.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 13:53 UTC (Thu) by farnz (subscriber, #17727) [Link]

And don't forget that predicting what a distro will have when you release your new version is hard. If you're going to release in April 2023, and something you depend upon has a necessary update released in December 2022, is that update going to be in the current stable Debian release when you make your release?

If you guess wrong, you end up in one of two sub-optimal situations:

If you assume the update won't be submitted by the Debian Developers in time to make it into Debian 12 (Bookworm), or that the release will slip into May, and then the update makes it in anyway, you're carrying sub-optimal code to handle the pre-update version of your dependency.
If you assume it will make it in, and then something causes Debian to have to delay the release, or if the update can't be put into Bookworm before the freeze kicks in without breaking something critical, your project now doesn't run on Debian stable on the day of release, because you're waiting for an update.

Bundling from a vendor source neatly sidesteps this - if Bookworm has the dependency version you need, then unbundling can be done then, while if it doesn't, no problem, you've got the vendored code. And then language repositories like PyPI make it simpler, because they're already working in terms of a dependency tree, not copied code, so you can look and go "aha, when I build the Debian package, I can unbundle libfoo, since Debian has the right version of libfoo already".

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 15:37 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (4 responses)

I don't know if waiting for the distributions to package everything is the right solution. I'm inclined to believe developers who say it's unlikely to work for them, if only because of the problem you highlight of there being too many libraries to package. I also believe, though, that the current solution of grabbing whatever is out there, trusting it's fine, and then acting surprised by supply chain attacks is also not working. It's just resulting in occasional spectacular failures rather than regular, boring unavailability of bug fixes and product enhancements.

What is needed is a system that provides some kind of real quality control, so developers can have a confidence the libraries they're using are what it says on the tin. This has the unfortunate side effect of slowing everything down for the QC step, but the alternative is occasionally getting pwned when attackers finally decide your system is worth attacking. Pretending everything is fine in an attempt to go as fast as possible is demonstrably not working.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 17:22 UTC (Thu) by kleptog (subscriber, #1183) [Link] (1 responses)

We need to be able to have quality control somewhere, the question is if PyPI is the organisation to do it. What I'm thinking is a kind of meta-index where different organisations can give a "trust rating" or something, and that I can filter on that in pip.

For example, it would be possible for someone to write a bot that checked if the contents of the wheels distributed by PyPI match the source in the indicated repository. The problem is there is nowhere to place this information in a way that is of any use. Or, it would be nice to straight up reject any package that has existed for less than 3 months. Or being able to namespace dependencies to ensure they come from the right repository.

Python is here paying the decision early on that no packaging/repository standard would be made and to let the community create one organically. It's biting back hard now. More recent languages did not repeat that mistake.

PS. Don't talk to me about solutions like Nexus which try to solve the problem on the client-side but don't really have any extra information to work with and so just end up adding an extra layer of frustration. Until the necessary information is available in machine readable form no client-side tooling can help.

PyTorch and the PyPI supply chain

Posted Jan 13, 2023 4:01 UTC (Fri) by pabs (subscriber, #43278) [Link]

I feel like distributed code review ala crev, along with reproducible and bootstrappable builds is the right model here.

https://github.com/crev-dev/
https://reproducible-builds.org/
https://bootstrappable.org/

PyTorch and the PyPI supply chain

Posted Jan 18, 2023 1:15 UTC (Wed) by hazmat (subscriber, #668) [Link] (1 responses)

re distros, too much plurality and too much complexity to get across the swathe when viewed in aggregate for a developer/publisher of a python dependency, ie language tools exist for a reason. i'd be willing to settle for simple signature upload and verification (sans gpg, say cosign) on the language repos.. but there has been a slow moving effort to try and make things better albeit glacial speeds.. https://blogs.vmware.com/opensource/2020/09/17/tuf-pypi-i... updates are scattered across internet/github issues / tulip chat logs. i do wonder if some of the ossf projects will help on the broader ecosystem. its unclear if we're not just creating another checkbox for enterprises sometimes vs actually moving the needle.

PyTorch and the PyPI supply chain

Posted Jan 18, 2023 15:34 UTC (Wed) by hazmat (subscriber, #668) [Link]

found the best link for current state at
https://github.com/pypi/warehouse/issues/10672

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 3:00 UTC (Thu) by mpr22 (subscriber, #60784) [Link] (5 responses)

> I keep coming back to the question why every language needs their own package manager with the usual set of problems to (a) discover and (b) solve in incompatible manners...

I am less than half joking when I say "blame Perl".

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 3:14 UTC (Thu) by bferrell (subscriber, #624) [Link] (1 responses)

Until VERY recently if you wrote PERL code that did this, the dev community would fetch piles of wood and burn you to the ground.

A few years back a VERY common module got re-written and made major changes to the behavior of the code... With no documentation. They just thought it was a "good idea (tm)".

Post hasty, that got changed and while the new behavior WAS a good idea and kept, it became a "turn it on with a variable if you want it" vs "here, let me shove this down your throat".

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 14:36 UTC (Thu) by smoogen (subscriber, #97) [Link]

I wonder if that is because the Perl community learned the lessons the hard way. I spent way too much time in the late 90's and early 00's fixing 1 am outages to undo some developer's 'grab the latest from CPAN' which 'fixed' whatever bug they had but added 200 new ones in a myriad of dependencies (or my favorite.. why is the perl on each web server or application different? Oh because each team of dev's did a CPAN update and compiled a new version as part of that..) Things became more stable after that... but I also was dealing with perl less and less because various web devs I worked with were finding it 'too stodgy' and moving to Ruby, Python and then Node because speed in module changes were there.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 15:03 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

In Perl's defense, the modern distribution system didn't exist yet when CPAN started. CPAN was built at more or less the same time as modern distribution packaging systems, so waiting for packages to go up on the distribution wasn't a serious option. Even if people had been willing to wait a few years for that system to develop, nobody knew that it was going to develop, or even that Linux was going to win the Unix wars, so some kind of homebrew packaging system was necessary.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 17:01 UTC (Thu) by Sesse (subscriber, #53779) [Link] (1 responses)

Which is, ironically, one of the language package managers that is the least painful to integrate into distributions, and one of the languages with the best track of backwards compatibility in modules.

PyTorch and the PyPI supply chain

Posted Jan 22, 2023 9:42 UTC (Sun) by oldtomas (guest, #72579) [Link]

Perhaps ironical, but not surprising.

I think Perl's growth happened at a time where "fitting in an environment" was the obvious thing to do. One data point? POD has as one of its main targets man pages.

Python (re- [1]) started a trend which I'll call "language monotheism", where each language had (or thought it had) to fight for absolute dominance. I think this might be something for computer sociologists to study some day.

[1] Not the first round, mind you. Older people might remember C vs Pascal, quiche eaters and that. Of course, nowadays, in the era of overabundance, survival and money are more at stake than back then.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 3:05 UTC (Thu) by flussence (guest, #85566) [Link]

Gentoo has also recently started adding the ability to verify upstream tarballs against upstream public keys and signatures, which is better than nothing, but is pointedly next to nothing. If the package in question gets its code from a git repo with signed commits, there's nothing to check that. If the package *itself* lives in a signed git repo, you can't reuse the download-checking key management (it actively fights you if you do) and have to figure out through trial and error how to manually fetch and manage GPG keys for a non-root system account.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 8:45 UTC (Thu) by ms (subscriber, #41272) [Link] (4 responses)

> To me the solution is pretty simple and nothing that hasn't been implemented elsewhere: identify the repository in the dependency specification.

I think that certainly helps. But there have also been lots of examples of submitting PRs that get malicious code in to repos; along with social engineering to take over code repos; and e.g. established chrome extensions being sold to a new owner and then malicious code gets injected. In these cases, the name of the repository hasn't changed.

Another thing that helps is getting away from this mantra of "always fetch the latest version that satisfies your semver constraints". If you take the Go approach of _minimal_ version rather than maximal, then the blast radius is much reduced: it is no longer sufficient to release a new compromised version - that on its own will not get picked up. You would also have to modify the deps of a repo that imports that, and of that, and so on, all the way up to the top.
https://research.swtch.com/vgo-mvs (the section on "Upgrade Timing" is most relevant here).
I'm certainly not claiming Go is the only language to do this; it is simply the one with which I'm most familiar.

What I absolutely detest is the attitude that "this is a people problem, we shouldn't try to solve it with technical means". Correct - you won't be able to _solve_ it. But that's not the point. The point is to reduce the probability of these farcical messes from occurring. And there is plenty of prior art out there that helps. Refusing to learn from that is just sticking your head in the sand.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 12:35 UTC (Thu) by khim (subscriber, #9252) [Link]

> You would also have to modify the deps of a repo that imports that, and of that, and so on, all the way up to the top.

Sounds like cultivation of Log4Shells instead of “dependency confusion”.

But yeah, that's definitely fit well into “simple non-solutions” scheme Go practices.

PyTorch and the PyPI supply chain

Posted Jan 14, 2023 19:42 UTC (Sat) by KJ7RRV (subscriber, #153595) [Link] (2 responses)

Doesn't using the minimum version instead of maximum result in not receiving security updates for dependencies until the depending package is also updated? That seems like a much worse outcome for security, especially considering dependencies of dependencies, etc. Or does Go have a way of specifying that an update is a security update?

PyTorch and the PyPI supply chain

Posted Jan 15, 2023 11:14 UTC (Sun) by farnz (subscriber, #17727) [Link] (1 responses)

There cannot be a way to specify that an update is a security update without losing any gains from the "minimal version" route; there is no way to distinguish "malicious actor flags version with back door as security update" from "good actor flags version removing back door as security update".

As with so many things, it all comes down to trust. If you trust upstream to release good updates, you want to take their latest code. If you don't trust upstream, you should be locking exact versions, and reviewing every new release upstream manually before you bring it in (which, in turn, has to be your top priority in case the fixes are security relevant to your code).

PyTorch and the PyPI supply chain

Posted Jan 15, 2023 12:18 UTC (Sun) by ms (subscriber, #41272) [Link]

Exactly. There is tooling to help find vulnerabilities, but yes, the basic premise is that you the developer must explicitly give permission for some dependency (even transitively) to be updated. Doing anything less gated than this is really just giving other developers permission to execute arbitrary code on your machine.

Both of these are relevant:
https://go.dev/blog/vuln
https://go.dev/blog/supply-chain

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 12:19 UTC (Thu) by khim (subscriber, #9252) [Link] (2 responses)

> I keep coming back to the question why every language needs their own package manager with the usual set of problems to (a) discover and (b) solve in incompatible manners...

Because Gentoo is not macOS or Windows, basically.

Newbies to the programming would, inevitably, use one of these two. And if your language doesn't support them well then it's chances of being used in place of more popular alternative is almost null.

And if you have something that works for beginners… people continue to use it for other things, because why not?

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 20:14 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

I started using gentoo precisely because I needed features that were not available in the latest version of my then distro (SUSE). I could have used the project's own install setup, but I would rather use the distro, so I changed to one that made it easy.

However, I wouldn't recommend gentoo to newbies ... (unless, of course, they want to do things the hard way :-)

Cheers,
Wol

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 21:19 UTC (Thu) by khim (subscriber, #9252) [Link]

The problem if distros is not technical, ultimately, it's social.

Most distro makers drank too much FSF kool aid and now believe others want to create sources for others to use.

Nothing could be further from the truth!

Neither users nor developers are interested in the software for software sake.

Their goal is to produce binary and to either give it away or use it.

That's why disconnect is so deep.

In a world where creation of software source is the goal you have to support various versions of dependencies (because this increases usability of your sources) and then, on top of that, you can afford “curated repos”, then, on top of that, you can provide “long term support” and all these other things.

In a world where software source only exists because it's not very convenient to write in machine code directly… situation is radically different: developers assume that they would decide what dependencies they would use and what targets they would support and users decide they would decide what version of application they would download and use.

Given insane disconnect between expectations it's no wonder no one is happy.

Gentoo, NixOS and other such distros support that mode, but they make assumption that this desire to control everything goes to the core… but most developers and users don't go that far: they are ~~happy to use OS that hardware maker gives them~~ too scared to replace OS that hardware maker gives to them, they want to control things on top of that.

Maybe if Gentoo or Nix supported macOS and Windows this would have been an acceptable compromise, but alas, they don't do that (at least they don't make it easy enough to use for newbie), thus we have no alternative to per-language package managers.