|
|
Subscribe / Log in / New account

PyTorch and the PyPI supply chain

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 11:12 UTC (Thu) by summentier (guest, #100638)
Parent article: PyTorch and the PyPI supply chain

Package management and build automation is hard. I get that, I do.

But trying to coerce setuptools to do what you want it to do is not fun. Its code is an undocumented mess, its abstractions are leaky and incoherent, and its architecture like a Jenga tower resting on top of a pile of Mikado sticks. Look at nontrivial setup scripts bundled (e.g., for project such as numpy or tensorflow): they resemble ancient incantations much more than actual code.

So I do not envy pip's job. But much of what ails setuptools also seems to have infected pip: its documentation is ... terse, to say the very least, its code isn't great either, and it does like to act and fail in ever-surprising ways. Moreover, coming from Rust or Julia, it is very hard to be satisfied with the hodgepodge of virtualenvs one has to set up in case of dependency conflicts. So, respectfully, it seems in character that pip does something sub-optimal and I think a doc fix is not going to fix those deep structural issues. (Anaconda, while certainly well-intentioned, tends to make everything worse, particularly on supercomputers.)

I understand that pip is in a tight spot now with respect to backwards compatibility.
Hopefully new projects (such as poetry) will improve this, I have to say, rather sorry state.


to post comments

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 17:10 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> But trying to coerce setuptools to do what you want it to do is not fun. Its code is an undocumented mess, its abstractions are leaky and incoherent, and its architecture like a Jenga tower resting on top of a pile of Mikado sticks. Look at nontrivial setup scripts bundled (e.g., for project such as numpy or tensorflow): they resemble ancient incantations much more than actual code.

I think at one point, NumPy's additions to setuptools were on the same order of size as setuptools itself. SciPy probably didn't make things any easier.

I will agree about the undocumented mess 100% though. Figuring out what could go into some fields (globs, symlink traversal, etc.) involved tracing the value(s) through the code to where they hit some active API that actually used them. The duck typing helps with being able to get things done by abusing things like `../` traversal to grab things, but really hinders with making anyone aware of what is possible (and what of that is actually intended).


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds