LWN: Comments on "Remote imports for Python?"

Remote imports for Python?

Garak — Sun, 10 Sep 2017 05:32:53 +0000

DWA, my new TLA of the day, thx. I think the DWA angle is interestingly core here. The modern world being a smaller place has had this evolution of putting oddities, and perhaps a shrinking amount of scattered human suffering under a metaphoric microscope. The sorts of news stories that make the feeds as well as the mainstream news are a lot like this. And unfortunately, the sensationalism outweighing a more rationally weighted perception seems to be the rule in this day and age. (Of course a lot more was more easily hidden from the public in the past, not claiming things overall were obviously better historically).

What I'm thinking in response to your comment is that such arguments of trying to protect people from shooting themselves in the foot don't seem that persuasive to me. I see a vast diverse ecosystem of developers, and I'm personally not bothered by the DWA factor. The news industry won't let the DWA factor fade, they'll just add another lense to that microscope and give people the entertaining idiot story to laugh at or be scared of for a moment.

Of course the example that comes to mind above is bannning swimming pools to save children because of course some parents who choose to have pools will fail, and kids will die. Somehow, for reasons I couldn't map out in a thesis, I'm more worried about the swimming pools than the dangers being talked about here if the feature gets added. And I've been able to swim for 3 decades now. I mean seriously, there will always be ways for software developers and businesses that rely on them to metaphorically shoot themselves in the foot. I don't see this as significantly effecting that in the long term. But OTOH, the obvious FOSS answer is - Fork Python if you care enough. Let the fittest thrive the most in the ecosystem. Somehow I doubt anyone cares about this feature that much. But it made for some sensationalist reaction commentary with academic entertainment value.

Remote imports for Python?

johan — Sun, 03 Sep 2017 07:13:41 +0000

> It's not really any different then how people get software now. It just cuts out steps.

It could potentially cut out pip from the loop yes, but how hard is a "pip install" anyway?
We are not talking about simplifying something that is currently complex, we are talking about shaving of a few minutes at max in their workflow.
So why should we need yet another way to do the same thing in a slightly less secure manner?

Personally i see this httpimport as a great library, but I don't see much reason to add it to the python standard library.
There's probably more people afraid of the security of python than who wants this feature, so adding it to the python standard library doesn't make sense.

Remote imports for Python?

sasha — Sat, 02 Sep 2017 17:34:29 +0000

Here in Russia, the government authorities can block access to some IPs. At some point they blocked access to GitHub (I do not remember the reason: hate speech or substances). It broke a lot of things for large corporations, so GitHub was immediately unblocked. The programmers explained that they were following "the world best practices".

The possibility to replace a curl script with direct python import changes nothing here.

Remote imports for Python?

robert_s — Sat, 02 Sep 2017 11:45:16 +0000

The "distro build package" step you mention here is actually more like: distro build package, run tests, make sure it works properly with all the other components of the distro.

And most importantly, if a package author goes AWOL or starts adding things that might not be in the users interest, the distro has the ability to patch the code or even transparently switch to a different upstream.

Remote imports for Python?

syops — Fri, 01 Sep 2017 13:52:59 +0000

I do find myself wishing Subresource Integrity could be generalized to work with any HTTP GET. As it stands, SRI provides the resource in the src or href attribute, and the hash (or signature?) in a separate attribute. But that doesn't help with any arbitrary simple http application or library. Requests, pip and our favorite curl | sudo bash aren't designed to checksum the data they're fetching. I'd like to think this could be solved in a future implementation of http (maybe rolling hashes could be used to avoid the pitfall of the client having to stage and hash a very large download before writing it to disk), but I'm a dreamer.

Then again, there's always IPFS. As I understand it, with IPFS, the hash is the address of the content.

Remote imports for Python?

bandrami — Fri, 01 Sep 2017 13:20:07 +0000

from leftpad import unpredictability

Remote imports for Python?

amarao — Fri, 01 Sep 2017 10:07:01 +0000

There is a huge difference between 'direct from http' and apt-get (yum) install.

1. when we compile anything from external git we clone it into our git.
2. Normally all code comes with tests and those tests are executed at build time (at CI)
3. Packages are artifacts, they are stored and used every time in pristine manner to rebuild working environment on each run.

If you replace whole build->publish->install cycle, you will have flaky production. What if there was a hiccup in the connectivity during installation? What if author pushed new version between deploying two servers with same dependencies (but their subdependencies unpinned?).

Best practice for DevOps:

1. Everything can be rebuild and deployed automatically.
2. And we have it sources.
3. On our premises
4. Even build system and jobs can be rebuild from git.
5. Every deploy can be repeated in precise manner.
6. Everythin committed is covered by tests.

Remote imports for Python?

anselm — Fri, 01 Sep 2017 01:12:56 +0000

Good engineering discipline is in precious short supply nowadays...

It seems to me that a software system whose installation procedure is based on piping the output of curl into a local root shell forfeits any claim to “good engineering discipline” right there.

Remote imports for Python?

ThinkRob — Fri, 01 Sep 2017 00:53:21 +0000

> From the article I get the impression that this feature is meant for really rapid prototyping or testing.

The road to hell is paved with good intentions.

There is no doubt in my mind that as soon as this feature hits mainline, that a whole bunch of shops with "Agile" in their job descriptions are gonna go "Sweet, we don't need a build system anymore!" and run with it.

Remember: this is the reality in which something like 20% or whatever of npm modules broke when 'leftpad' went away. Good engineering discipline is in precious short supply nowadays...

Remote imports for Python?

lsl — Thu, 31 Aug 2017 19:38:40 +0000

Right, although it's even more layered than that.

To the compiler, import paths are opaque strings used for identifying a given module/package.

The 'go' build tool (roughly equivalent to what you would use CMake or whatever for in a C-based project) interprets them as file system references inside GOPATH and uses them to figure out what source files to pass to the compiler.

Only 'go get' tries to infer a remote location from them. What 'go get' does is let you say "clone this repo without making me figure out whether it uses Git, Mercurial or SVN and put it where the build system will pick it up". It's a convenience wrapper around VCS tools and is inherently targeted at developers, not users. You don't want to use it at deploy time.

Thus, Torakis' Go analogy is a bogus one.

Remote imports for Python?

k8to — Thu, 31 Aug 2017 19:32:07 +0000

Python is definitely incapable of meaningfully sandboxing.

That aside, the proposal for supporting it with a hash seems sort of vaguely OK, but I don't see the point. If you know the content you want to run ahead of time, why do you need to load it dynamically? I expect the major use pattern at that point will be people who write some code to generate the hash dynamically and then httpimport it, or in other words, the path of laziness.

Remote imports for Python?

cesarb — Thu, 31 Aug 2017 16:44:55 +0000

I don't know if it has already been mentioned on the thread, but there's another problem with remote imports: reliability. A local import can work even offline; a remote import depends on both the remote server and the network being up every time the process is started.

Remote imports for Python?

gioele — Thu, 31 Aug 2017 15:57:16 +0000

> It searches for it, makes an RPM spec file, builds and installs it, and submits the source package for inclusion in Fedora?

It is not that automated, but the Debian packages for many Rubygems are created, built and semi-automatically updated in a similar way, using gem2deb and gemwatch.

Remote imports for Python?

bernat — Thu, 31 Aug 2017 12:09:26 +0000

Go doesn't have this feature either. Package names look like an URL but are just names. "go get" will interpret them as URL (with a builtin set of rules), fetch them and put them in GOPATH for the compiler to find them.

Remote imports for Python?

dunlapg — Thu, 31 Aug 2017 08:52:58 +0000

As Torakis's age post indicates, there may be something of a generation gap surrounding this issue. The GitHub-centric, rapid-fire development style meets the grizzled graybeards who still sport some of the scars of security issues past.

I think it's worth noting that in the world the actual greybeards grew up in was a world of open ports and unsecured SMTP relays -- a world where you typed your password in the clear into telnet and ftp. In other words, a world just as trusting and impatient as the "curl http://somecode.com/install.sh | sudo sh" crowd. The difference is not one of culture or personality, it's one of experience.

Remote imports for Python?

Sesse — Thu, 31 Aug 2017 08:45:26 +0000

So what happens when you want to move from e.g. Gitlab to Github (or your URL becomes inaccessible for any other reason), but there's 200k users out there with your module URL hard-coded in their imports?

/* Steinar */

Remote imports for Python?

lamby — Thu, 31 Aug 2017 08:44:20 +0000

"age post" ?

Remote imports for Python?

jwilk — Thu, 31 Aug 2017 07:17:22 +0000

> [HTTPS] surely was not in widespread use by 1995.

Indeed. HTTPS support was added to stdlib in Python 2.0, released in 2000.

Source: https://docs.python.org/3/whatsnew/2.0.html#module-changes

Remote imports for Python?

smckay — Thu, 31 Aug 2017 03:12:38 +0000

Homebrew is pretty dang good these days. It's basically Portage on OSX except the default build configuration has binaries available.

Remote imports for Python?

drag — Thu, 31 Aug 2017 01:20:34 +0000

> So maybe not an import-from-URL, but more of an "auto-pip", where everything is hosted and vetted?

I suppose so.

I like how Debian does it with apt-get. You have a signed list of packages with their checksums and locations and mirrors. You could mirror the file local to the list or on any server really. Then you don't have to really care where or how they are stored because you have their checksums in a secure manner. Https vs http vs ftp vs nfs mount or whatever... doesn't matter.

Nothing revolution or weird or remarkable or even that much different. The difference between this sort of setup versus distro packages is that it would be OS agnostic since it would be largely source code based. The same list of packages would be for OS X vs Linux vs Windows or whatever.

Of course the crappy part is that some packages would require all libraries being present for compiling. I know that is relatively easy for Linux, but I don't know what that is like to setup for OS X or Windows.

Remote imports for Python?

epa — Wed, 30 Aug 2017 17:11:20 +0000

It searches for it, makes an RPM spec file, builds and installs it, and submits the source package for inclusion in Fedora?

Remote imports for Python?

adam820 — Wed, 30 Aug 2017 16:49:06 +0000

So maybe not an import-from-URL, but more of an "auto-pip", where everything is hosted and vetted? If you stick an import in the code, and then try to run it but it doesn't have it, it just searches for it and auto-downloads it?

Remote imports for Python?

jezuch — Wed, 30 Aug 2017 16:41:40 +0000

> Nowhere is mentioned that this should come anywhere near production code.

You're right, but that doesn't mean that it won't. I guess a Daily WTF article about some "quick hack" like this that ended up in production ("just temporarily, promise!"), won't be long coming :)

Remote imports for Python?

drag — Wed, 30 Aug 2017 16:29:46 +0000

It's not really any different then how people get software now. It just cuts out steps.

Currently if you use pip it goes like this:

https server --> pip install foo --> import foo

If you use distro supplied packages it is like this:

https server ---> distro build package ---> https server ---> yum install python-foo --> import foo

In a real sense it's not a whole lot different then just going:

https server ---> import foo

In every case the original source of trust is the https server. Even the deb or rpm package signatures doesn't confirm that the original code wasn't pulled from a compromised source. Sometimes distros can audit the code, but that only happens for a minority of packaged software.

If security is really the priority then the real way to do it is to have the original developers sign a tarball or package of the source code prior to ever being posted to anything touching the internet and then establish a chain of trust all the way through to the end user. The easiest way to do that may just be to eliminate as many layers between end users and developers as possible. This doesn't prevent a third party from auditing the source code and giving their official blessing to specific revisions.

Remote imports for Python?

niner — Wed, 30 Aug 2017 07:22:35 +0000

From the article I get the impression that this feature is meant for really rapid prototyping or testing. The httpimport module's documentation talks about testing pull requests. Nowhere is mentioned that this should come anywhere near production code. Having to get a checksum of some remote module first negates the speed benefits. And people who think about security at all will know not to use httpimport in production code.

That said, it _is_ a good idea when downloading remote code is indeed wanted. Some Perl 6 modules for example are just bindings for native libraries. To simplify installation on Windows, their installation may involve downloading DLLs and it's common to use checksums to secure that.

Remote imports for Python?

mokki — Wed, 30 Aug 2017 06:53:08 +0000

Java has tried to support this for 20+ years with signed remote packages and a sandbox.

99% of security bugs reported against java in the last 5 years have been about remote code escaping the sandbox.

Why would python want that security circus? It just makes the language seem bad when actually 99.9% of code never uses the remote execute feature and thys does not even enable the sandbox.

Enabling this in python without sandbox would be security nightmare and supporting a sandbox is known to be security nightmare.

Remote imports for Python?

josh — Wed, 30 Aug 2017 05:04:49 +0000

Likewise, that would be quite helpful.

Or the hash of a public key matching the private key the module is signed with.

Remote imports for Python?

smckay — Wed, 30 Aug 2017 03:53:34 +0000

I find it difficult to take the guy at face value. Compared to the alternatives, the value of remote code loading as a language feature is mostly making exploits easier. Everyone who actually wants this terrible misfeature can google "Python import from URL" and be good to go in 2 minutes.

Remote imports for Python?

luto — Wed, 30 Aug 2017 03:39:31 +0000

I'm surprised no one has suggested doing this but making it mandatory for the import call to include a hash of the module.

Remote imports for Python?

bferrell — Wed, 30 Aug 2017 02:57:38 +0000

I'm guessing the lesson from "deprecated" javascript module was lost on this person.