PyTorch and the PyPI supply chain [LWN.net]

PyTorch and the PyPI supply chain

Posted Jan 26, 2023 17:27 UTC (Thu) by irvingleonard (guest, #156786)
Parent article: PyTorch and the PyPI supply chain

Why is everyone blaming all kinds of ancillary stuff? The blames lies in the pytorch team. You're using pip, which uses the PYPI by default, hence you have 3 options:
1. You could use your private index by disabling PYPI altogether and provide every possible dependency. It would work for your package but break every other one out there.
2. You could embrace PYPI and don't use a private index. This might be not feasible for political (or technical?) reasons.
3. You could use them both at the same time, which is what they ended up doing.

Now, the problem is that if you use PYPI you have to play by its rules. Package names are an asset on PYPI: the first one that claims it will own it. They obviously didn't read that memo and got bitten by it. The "solution" was as simple as publishing a dummy package in PYPI with a very low version number for every "private" package that only lived in their private index. That dummy package could be a simple readme with the instructions on how to reach the private index and with that they would have prevent the hijacking.

Am I wrong in this analysis?

to post comments

PyTorch and the PyPI supply chain

Posted Sep 11, 2023 19:49 UTC (Mon) by snnn (guest, #155862) [Link]

I know why PyTorch is in a separated index:

1. Their packages are huge. One file could be 1-2 GB. But PyPI is free. PyPI cannot be so generous to provide so much free storage for every PyPI project.
2. You may build PyTorch with different build configs. For example, different CUDA versions. PyTorch community wants to keep all of them under the same name: pytorch. Otherwise it would harder to other packages to setup dependency on PyTorch. Therefore, PyTorch chose two different approaches: 1. put all of them in the same index and use local version to distinguish them. 2. Put each of them into a different index. However, both approaches are not supported by PyPI.

This problem is a very general. Almost all machine learning packages with GPU acceleration capabilities need to deal with this. I believe every non- casual user should setup their private pypi index. Even the original problem is fixed, as long as you have multiple indexes, you are still at risk. You may think the problem in a different way: how much can I trust the Facebook's pypi index servers? What if someone puts a fake "wheel" package in PyTorch's PyPI index? Don't think no Facebook employee's account can be hacked if you still could remember last year Nvidia lost their GPG key.