Remote imports for Python?

Importing a module into a Python program is a pretty invasive operation; it directly runs code in the current process that has access to anything the process can reach. So it is not wildly surprising that a suggestion to add a way to directly import modules from remote sites was met with considerable doubt—if not something approaching hostility. It turns out that the person suggesting the change was not unaware of the security implications of the idea, but thought it had other redeeming qualities; others in the discussion were less sanguine.

In his first post to the python-ideas mailing list, security researcher John Torakis proposed imports via HTTP and HTTPS be added as a core Python feature. He also filed an enhancement request; both the post and the bug pointed to his httpimport repository on GitHub that has a prototype implementation. The justification he cited was always likely to set off some alarm bells:

Chris Angelico vehemently opposed the feature, at least for core Python: "This is a security bug magnet; can you imagine trying to ensure that malicious code is not executed, in an arbitrary execution context?" If the feature is explicitly enabled (via, say, pip), it is much less worrisome, Angelico said. The idea of allowing imports over regular HTTP is one that should be dropped, he said; even HTTPS imports would require being "absolutely sure that your certificate chains are 100% dependable".

Oleg Broytman also opposed the idea, suggesting that it would require a Python Enhancement Proposal (PEP), instead of simply filing an enhancement request, to truly be considered. He also noted that there is a difference for Go's remote imports: those happen at compile time for Go, while they would be done at runtime for Python.

The README for Torakis's httpimport repository mentions it being used as a "staging protocol for covertutils backdoors". Torakis's covertutils project is described as a "framework for backdoor programming"; that also got Angelico's attention:

Torakis responded to those complaints (in a posting with non-standard quoting). The backdoor work is evidently part of his day job and httpimport could be useful for that work, especially for rapid prototyping, testing, and debugging purposes. He did agree that HTTP imports are dangerous, but noted that doing so locally (i.e. only to localhost or perhaps trusted local systems) could be useful for testing. He also objected to the complaint about HTTPS, however, noting that certificate checks would be done to eliminate the man-in-the-middle threat. In another posting, he put it this way: "if you can't trust your certificate store now, and you are afraid of Remote code execution through HTTPS, stop using pip altogether".

While indicating that httpimport would make an excellent addition to the Python Package Index (PyPI), Paul Moore pointed out another flaw with the idea of making the feature a core part of the language: it removes the ability for an organization's security team to disallow it. On the other hand, the team could restrict the ability to install modules from PyPI or simply blacklist certain entries such as httpimport. He said:

This is not the first time this idea has come up, Guido van Rossum noted; it was first proposed (and rejected) in 1995 or so—before HTTPS was invented, he said. He may be misremembering either the date or the status of HTTPS, as Wikipedia gives 1994 for its creation, though it surely was not in widespread use by 1995. As with others in the thread, Van Rossum was happy to see a third-party httpimport available to those who need it, but it is too much of a security concern to ever consider adding to the standard library.

Torakis said that he was two-years old at the time of that decision and that "times have changed". He said that he is willing to make changes to make httpimport acceptable for the standard library. It would make working with Python much easier for him and others:

There was no support for adding it as a core feature in the thread, though. A simple "pip install httpimport" is all that would be needed to get access to the feature (once Torakis gets it added to PyPI, anyway). So some thread participants wondered why it was so imperative that it become part of the standard library. As Stephen J. Turnbull put it: "It's an attractive nuisance unless you're a security person, and then pip is not a big deal." Echoing Moore to some extent, Nick Coghlan added another concern:

There is at least one popular scripting language that makes this feature available as part of the language: PHP. Originally, directives like include and require could contain a URL; code would be retrieved from that URL, then executed. Over time, the wisdom of that choice has been questioned; these days, two configuration options (allow_url_fopen and allow_url_include) govern the behavior and remote inclusion is disallowed by default. While many scripting-style languages have ways to accomplish remote inclusion, it is considered something of a trap for the unwary, thus not elevated to be a top-level language feature.

Something that bears noting from the discussion is that installing code over HTTPS from PyPI is no more (or less) dangerous than doing so from GitHub—at least from a man-in-the-middle perspective. There is still the danger of using code from a malicious GitHub repository but, in truth, the same problem exists for PyPI. There is no active vetting of either GitHub repositories or packages uploaded to PyPI. HTTPS can ensure that you are connecting to the server holding the proper key, but it cannot protect you from asking for the wrong thing from that server.

As Torakis's age post indicates, there may be something of a generation gap surrounding this issue. The GitHub-centric, rapid-fire development style meets the grizzled graybeards who still sport some of the scars of security issues past. It is certainly true that it is easy enough to add remote imports to a program (via httpimport or something hand-rolled), but the idea is that programmers will have reached a certain level of understanding when they get to that point—hopefully enough to recognize the dangers of doing so. In any case, by not making it a top-level, supported feature, abuse of it is not the responsibility of the Python core team. Avoiding that kind of "attractive nuisance" (and the bugs it can spawn) is another lesson that the Python graybeards have learned along the way.