Python virtual environments [LWN.net]

By Jake Edge
June 13, 2018

In a short session at the 2018 Python Language Summit, Steve Dower brought up the shortcomings of Python virtual environments, which are meant to create isolated installations of the language and its modules. He said his presentation was "co-written with Twitter" and, indeed, most of his slides were of tweets. At the end, he also slipped in an announcement of his plans for hosting a core development sprint in September.

The title of the session was taken from David Beazley's tweet on May 1: "Virtual environments. Not even once." Thomas Wouters defended virtual environments in a response:

The correct justification is that for the average person, not using a virtualenv all too soon creates confusion, pain, and very difficult to fix breakage. Starting with a virtualenv is the easiest way to avoid that, at *very* little cost.

But Beazley and others (including Dower) think that starting Python tutorials or training classes with a 20-minute digression on setting up a virtual environment is wasted time. It does stop pip install from messing with the global environment, but it has little or nothing to do with actually learning Python. Dower noted that Pipenv is supposed to solve some of the problems with virtual environments, but it "feels a bit clunky", according to a tweet by Trey Hunner.

In another Twitter "thread", there was a discussion of potential changes to pip so that it would gain the notion of local versus global installation. That might be a path toward solving the problems that folks see with virtual environments and Pipenv. Dower said he is willing to create a PEP if there is a consensus on a way forward.

He would like to see a way to do local package installation without using virtual environments. He also would like to have a way to invoke the "right" Python (the right version from the right location) without using virtual environments. But for those who are using virtual environments, he would like them to be relocatable, so that users can copy them elsewhere and have them still be functional. Barry Warsaw suggested making pip --user the default as it is in Debian and Ubuntu; Dower said that only "localizes the damage" and doesn't really solve the problem.

Core development sprint

Dower has volunteered to host a core development sprint to work on CPython. He has scheduled it for September 10-14, 2018 in Redmond, Washington on the campus of his employer, Microsoft. They will have an entire building to use for the sprint. There will be a hotel block reserved in Bellevue, since it is a more interesting place to stay, he said. Around 25-30 developers will be invited to attend; active developers or those with a PEP under consideration should expect to get an invite. He is hoping that the Python Software Foundation will pick up the travel expenses for the invitees, but any core developer is welcome to attend.

Index entries for this article
Conference	Python Language Summit/2018
Python	Virtual environments

Python virtual environments

Posted Jun 13, 2018 21:30 UTC (Wed) by unsignedint (subscriber, #92715) [Link] (3 responses)

I don't bother with virtualenv at all - just start a docker container based on the official python image. Boom - isolated environment

Python virtual environments

Posted Jun 13, 2018 23:48 UTC (Wed) by jhoblitt (subscriber, #77733) [Link]

I too am a heavy docker user but having having to deal with bind mounts or constantly reinstalling from pypi sounds like a major PITA. Especially compared to the ruby world where you merely have to `cd` into the project.

Python virtual environments

Posted Jun 14, 2018 7:50 UTC (Thu) by karkhaz (subscriber, #99844) [Link] (1 responses)

I don't faff around with containers any more---each of my python installs install goes into its own AppVM on Qubes, physically separated from each other by cutting segments out of the hard disk platter so that it resembles a ceiling fan.

Even that hasn't entirely mitigated environmental problems, though, so I'm now considering buying a new machine for each python package. Py3 packages will each have an Arch Linux box, while Py2 (or "Python Heritage") programs can have a PDP-11 to run on, with power supplies separated by audiophile-grade isolation transformers, each located in a separate Biosafety Level 4 containment facility. Typing "pip install --user" through the gauntlets of my positive air-pressure spacesuit will be a tad cumbersome, but it sure beats expending the modicum of effort required to get something packaged by my distribution.

Python virtual environments

Posted Jun 22, 2018 8:11 UTC (Fri) by fredcadete (guest, #81023) [Link]

This approach sounds great!

If only you would give it a cool name, logo, and set up conferences with lots of handouts... then I could use it in production.

Python virtual environments

Posted Jun 14, 2018 7:50 UTC (Thu) by robert_s (subscriber, #42402) [Link] (3 responses)

When I have the choice I skip language "toy" package managers entirely and use Nix shells

Python virtual environments

Posted Jun 14, 2018 14:54 UTC (Thu) by servilio-ap (guest, #56287) [Link]

When I have the choice I skip language "toy" package managers entirely and use Nix shells

And this have the advantage of being able to express dependencies that are not part of the language-specific package manager, e.g.: OpenSSL, libxml2.

Python virtual environments

Posted Jun 15, 2018 11:16 UTC (Fri) by sbaugh (guest, #103291) [Link]

I feel the same way, I much prefer Nix shells over whatever the popular Python virtual environment tool is today.

Apparently people find it painful to have to start any kind of Python training with 20 minutes of Python package manager/virtual environment tooling training. I know that one reason I would find it painful to teach people about those kind of tools is that they are not tools for life. Effort invested into learning Python tools will just be wasted when the tool of the day changes, or I want to use a different language.

But if people had a long-lasting packaging tool they could use for all languages, I don't think I would mind teaching that tool at the start of Python lessons; I'd know it would be long-term useful knowledge for people. I think Nix (or something like it) could meet this criteria.

Python virtual environments

Posted Jun 18, 2018 15:58 UTC (Mon) by civodul (guest, #58311) [Link]

Agreed, I feel that Nix shells and guix environment provide similar functionality but are more robust (they make no assumptions about the environment) and not limited to Python.

Python virtual environments

Posted Jun 14, 2018 13:13 UTC (Thu) by gdamjan (subscriber, #33634) [Link] (5 responses)

PYTHONUSERBASE=$PWD/py-env pip install --user A-LOT-OF-DEPENDENCIES

single line, no need of virtualenv, the directory is relocatable, no need for hardlinking or copying of the python executable (ugh how I hate that), and only needs setting a single environment variable to setup.

(ps. in uwsgi just set env = PYTHONUSERBASE=/here/there/anywhere)

Python virtual environments

Posted Jun 14, 2018 13:41 UTC (Thu) by cortana (subscriber, #24596) [Link] (3 responses)

Good grief, why on earth did they bother with venv and *shudder* virtualenv if it was that simple all along!

Python virtual environments

Posted Jun 15, 2018 18:27 UTC (Fri) by k8to (guest, #15413) [Link] (2 responses)

venv/virtualenv are essentially just a crappy hack that exploited a behavior of the default python logic to find its library location.

Essentially it's an extremely overwrought way to say something like export PYTHON_LIB=/a/path

Adding onto this, its magical nature and overuse means that you can get extremely hard to debug problems when you create a virtualenv and install a tool that itself uses a virtualenv.

It's just brain damage, and it's high time that python simplified this stuff.

Python virtual environments

Posted Jun 18, 2018 21:32 UTC (Mon) by stevedower (guest, #116614) [Link] (1 responses)

Yep, it really is this simple. I too do it all the time, which is why I was confident enough to stand up and say it should just be automatic. The only challenge is agreeing on "what magic marker means we should put a certain directory in sys.path", and I think we'll be okay there (likely __pypackages__ - if that directory exists, you can import everything from it and pip will default to installing into it).

Python virtual environments

Posted Jun 19, 2018 15:57 UTC (Tue) by k8to (guest, #15413) [Link]

In my projects I typically just do manipulate sys.path to point to an included directory during startup of my main python file, which I find vastly simpler than all the package managers and environment variables in the world.

However, that's obviously not sufficient for some problems, so I salute modernizing sys.path setup.

Python virtual environments

Posted Jun 25, 2018 13:37 UTC (Mon) by jsmith45 (guest, #125263) [Link]

My biggest gripe against virtual environments is that they break direct execution of a python script.

You cannot just type /path/to/my/script.py from a newly opened shell and have it work, unless you install globally which is not recommended. This also means you cannot simply run by double clicking in Windows either.

Compare that with node.js. Sure you do need to run `npm install` once, but after you have done so, as long as you don't change the script, you can just directly execute in a newly opened shell and everything works fine.

That alone is a large part of the reason that many projects try rather hard to avoid using anything not in the standard library, since if they do so they must either contaminate the global environment, or lose the ability to run the script directly (they would need to create some shell script wrapper to enter a virtual environment and then execute it).

Python virtual environments

Posted Jun 15, 2018 0:04 UTC (Fri) by flewellyn (subscriber, #5047) [Link] (13 responses)

And...why is it bad to use pip to install systemwide packages? I have never found conclusive answers to this question.

Python virtual environments

Posted Jun 15, 2018 1:40 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (7 responses)

It can cause conflicts with modules installed via distribution packages. There are ways around that including

https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Python virtual environments

Posted Jun 15, 2018 3:23 UTC (Fri) by flewellyn (subscriber, #5047) [Link] (1 responses)

Ahh, I see. So the reason it never bit me on the ass is that I always run pip as a regular user, and it installs just to my local, or to my venv, eh?

Makes sense. I suppose there's less issue if you use pip to install inside a container?

Python virtual environments

Posted Jun 15, 2018 10:28 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

Yes, installing as yourself or using a container is just fine. Installing system wide is generally not recommended

Python virtual environments

Posted Jun 15, 2018 6:16 UTC (Fri) by ceplm (subscriber, #41334) [Link] (3 responses)

And why not to use packages from the distribution?

Python virtual environments

Posted Jun 15, 2018 7:02 UTC (Fri) by kushal (subscriber, #50806) [Link] (1 responses)

Because in workshops we generally use latest packages from PyPI, which are not available from system. Also, this may install some other verison than the system version, which may break other applications on the system.

Python virtual environments

Posted Jun 20, 2018 2:19 UTC (Wed) by k8to (guest, #15413) [Link]

My view is that if you want to use virtualenvs in workshops (I would suggest giving them a container or vm instead), just have the virtualenv on when they log in. You can provide a flyer on setting up virtualenvs if you think that's necessary information to read when they try to use the skills.

Personally I've found virtualenvs to be so fragile that I will never use them again. I would advise teaching students a path that doesn't fall over so easily

Python virtual environments

Posted Jun 15, 2018 10:32 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

> And why not to use packages from the distribution?

Not everything is packaged by distributions and available in the latest version. Pip is unaware of distribution installed version and will overwrite it if a module is a dependency and vice versa.

Python virtual environments

Posted Jun 20, 2018 17:52 UTC (Wed) by cortana (subscriber, #24596) [Link]

I believe that in Debian, `pip install foo` run as root will install to /usr/local/lib/python$ver/dist-packages -- nicely out of the way of packaged modules, which all live under /usr/lib.

Python virtual environments

Posted Jun 21, 2018 7:55 UTC (Thu) by callegar (guest, #16148) [Link] (4 responses)

> Why is it bad to use pip to install systemwide packages?

Because python does not support versioned packages, never had.

So you cannot have on your system 2 different versions of the same package in the same way as you can have on your systems two different versions of e.g., a C library.

You cannot say anything like

import xxxyyyzzz as xyz at {versions}

from xxxyyyzzz at {versions} import abc

where versions is some version specifier e.g. ['>2.5.0', '<3.0.2']

so, when you pip install something you end up "hiding", or "breaking" another version of that same something that you may have already had installed (e.g. by your distro) and on which some part of your system may depend.

Not that this is not a problem that gets magically solved with virtual environments either. Virtual environments only mitigate it.

If you write program A that depends on package B, which in turns depends on C at version Y and then you realize that you also need D that also require D that depens on C at version Z, then you are in big trouble.

So the real question should probably be: "why does not python support versioned packages"?

Python virtual environments

Posted Jun 21, 2018 18:06 UTC (Thu) by raven667 (guest, #5198) [Link] (3 responses)

Honestly, this is one reason why containers exist, why solve this problem individually and separately for every different kind of language/runtime when you can solve it systematically at the OS level once for everything.

Python virtual environments

Posted Jun 22, 2018 6:49 UTC (Fri) by marcH (subscriber, #57642) [Link]

Clear Linux has a single version number for the whole system.

Source code used to be versioned file by file. Git and others made the single version number popular. Then submodules and repo stroke back.

Etc. Modularity is *hard* and what goes around comes around.

Python virtual environments

Posted Jun 22, 2018 12:15 UTC (Fri) by callegar (guest, #16148) [Link] (1 responses)

For instance, because containers, just like virtual environments (that may be seen as a lower form of containerization) only mitigate the issue, do not solve it.

As mentioned before if you write program A that depends on package B, which in turns depends on C at version Y and then you realize that you also need D which depends on C at version Z, then containers or virtual environments are not going to help.

But what I find weird is that versioned "libraries" or "packages", that is a simple solution that was invented together with shared libraries many tens years ago, which would cost very little to implement in a high level interpreted language and which would not preclude at all the use of containers or virtual environments did not make into Python that, for other things, does not seem to completely avoid rather complex solutions and their associated cost.

The funny thing is that the very language developers feel the need to introduce "naming" conventions to work around this limitation (e.g., because versioned packages do not exist, they end up attaching the major version number to the package name itself as in urllib vs urllib2). Same goes for extension packages on Pypi (e.g., gmpy vs gmpy2).

Python virtual environments

Posted Jul 3, 2018 1:24 UTC (Tue) by ghane (guest, #1805) [Link]

... and python2 and python3 :-)

Python virtual environments

Posted Jun 15, 2018 9:56 UTC (Fri) by auc (subscriber, #45914) [Link] (1 responses)

It's a bit sad none mentions conda environments, which solve a lot of trouble.

Python virtual environments

Posted Jun 18, 2018 21:33 UTC (Mon) by stevedower (guest, #116614) [Link]

They sure do, but the context for this discussion was "what can the core Python development team do or change to improve things". conda lives outside of this scope, so it doesn't really get to enter into the discussion (though we've certainly discussed these issues with the conda folks).