How to Manage Python Virtual Environments

Introduction

Using virtual environments for your Python project deployments allows you to isolate your project’s dependencies in an environment that is unaffected by the system Python installation(s). This makes it possible to run multiple projects with otherwise conflicting dependencies side-by-side on the same system. It also gives you peace of mind knowing that your code will run on any system with Python installed, regardless of whether that system has any of your required Python libraries installed system-wide.

.

Prerequisites

• A desktop or cloud server running Windows, OS X, Linux or any other operating system supported by Python
• A working Python installation
• (optional) If you plan to install Python libraries that have C extensions (such as NumPy) then you will need to have a C compiler and the Python development headers installed.

.

What Is a Virtual Environment?

Think of a virtual environment as a complete Python environment that is entirely separate and distinct from any Python environment installed at the operating system (OS) level. When we say “environment” here, we are referring to the Python standard library and header files, any third-party Python packages ( including those with C extensions), and any binary executables related to these. Having a separate environment for each project you are working on makes it much easier to manage dependencies. When you activate a Python virtual environment, your PATH changes to prioritize the Python binary inside your virtual environment over the one(s) installed to your OS. Using the Python binary in the virtual environment also means you will be using Python libraries installed in the virtual environment rather than the ones installed on your OS.
.

Python Installation

The good news is that Python version 3.3 or later comes with a virtual environment manager (venv) built in. If you use an earlier Python version, however, you can install the virtual environment manager (virtualenv) using pip:

sudo pip install virtualenv

There is little difference between venv and virtualenv, and this article will stick to discussing the shared features.
.

Managing your Python Virtual Environments

Now that you have a virtual environment manager (either venv or virtualenv), it’s time to start creating some isolated Python environments!

To simplify the discussion, from here on out we will refer to the environment manager as venv, and you can assume that any commands will work equally well with virtualenv just by replacing pyvenv with virtualenv in the command. If you are using venv your installed venv command may have a version suffix, such as pyvenv-3.5. Use that in place of pyvenv in the examples given.

Now, to get started, open a command prompt to a sandbox directory and enter the command

pyvenv venv-test

You should now have a venv-test directory structure that looks like this:

venv-test
├── bin (called Scripts in Windows)
├── include
├── lib (called Lib in Windows)
├── lib64 -> lib
└── pyvenv.cfg (venv only)

This directory structure is very similar to what you will see in a system-wide Python install, but it stands alone.

Now let’s activate it. At the same command prompt:

source venv-test/bin/activate

# or in Windows:
venv-test/Scripts/activate

You may see your command prompt change to append your virtual environment’s name (in our case (venv-test)) to the beginning of the prompt. To see that the virtual environment is active, run

pip --version

The output should indicate a path for the pip binary within the virtual environment directory.

Using pip in a Virtual Environment

You can easily install packages to your new virtual environment using pip, just as you would in your “real” Python environment. For example, let’s install the Django web framework in this virtual environment:

pip install django

Notice how administrator privileges were not required? That is because the virtual environment is completely owned by the user who created it. Let’s take a look at our packages now:

pip freeze

The pip freeze command shows a list of all Python packages installed and the version that is installed. You should see that Django has been installed. Since we are running pip within an activated virtual environment, the list includes only packages installed to the virtual environment, not system-wide packges.

You can use pip in a virtual environment exactly as you would use the system-wide pip with one crucial distinction: administrator privileges are not required, which is very convenient. You can see all the Python packages for your virtual environment under the venv-test/lib/python?.?/site-packages/ directory (venv-test/Lib/site-packages on Windows) which closely mirrors where packages are installed in the global Python directory structure.

For a quick refesher on pip, check out our article on other useful pip commands

.

More Virtual Environment Options

As with most UNIX commands, you can add --help after pyvenv in a terminal window to get a complete list of options that can be used with the command. We’ll go over a few of them in more detail here.

–clear

You can use this option to clean out an existing virtual environment, so you can start from scratch. This option will remove any packages you’ve installed within the virtual environment. We could use this option on our example virtual environment like so:

pyvenv --clear venv-test

You can run the --clear option with the virtual environment activated or not. You can verify that the virtual environment is clear by activating it and running pip freeze.

–system-site-packages

This option makes it so that any Python running in the virtual environment can access both packages installed to the virtual environment as well as packages installed system-wide. It can be useful if you have certain packages installed system-wide that you often use and want to use in all the projects you work on and the version of the package isn’t critical. We could use this option to create a new virtual environment like so:

pyvenv --system-site-packages venv-system-test

Now if you activate the new virtual environment and run pip freeze, you will see all your system-wide packages as well as the packages installed only in the virtual environment.

–without-pip (–no-pip)

While pip is a wonderful tool that most of us Python developers rely on completely, there may be legitimate situations where one might not want to include pip in a virtual environment. For example, if storage space were at a premium and the virtual environment is for a production deployment where you don’t need pip, it could be useful to create a virtual environment without it. You can create a virtual environment without pip like so:

pyvenv --without-pip venv-nopip-test

Note: the option is different in virtualenv.

virtualenv –no-pip venv-nopip-test

.

Deactivation of a Virtual Environment

When you are finished working in a virtual environment, you must deactivate it to return to your system’s global environment. Otherwise, any commands you make in the same terminal window will be in that virtual environment.

deactivate

Alternatively, you could simply close the terminal window.
.

Additional tools

While virtual environments are an incredible tool in their own right, the Python community has come up with a few ways to make virtual environments even better to work with. We’ll discuss a few of them in more detail below.

virtualenvwrapper

One of the most popular tools used with virtual environments is virtualenvwrapper. You can install it by running

sudo pip install virtualenvwrapper

# or, on Windows:
pip install virtualenvwrapper-win

virtualenvwrapper organizes all virtual environments in one place, provides convenient wrappers for managing virtual environments, and gives you tab completion for commands that take a virtual environment as an argument.

autoenv

Another popular tool that goes beyond the basics of virtual environments is autoenv, another hit from the prolific Kenneth Reitz, creator of python-requests.

sudo pip install autoenv

With autoenv, you can automatically activate virtual environments when you use cd to enter the directory. This addition really comes in handy for local development.

tox

If you work on Python packages that support several different Python versions or different versions of a major dependency, like Django, tox can help manage the prospect of testing these different environments.

sudo pip install tox

When you specify all the different environments you want to support in a tox.ini file, you can then run the tox command to run your unit test suite in all the environments specified. tox automatically creates virtual environments for all the different Python versions specified, installs required packages, and runs the test suite.
.

Final Thoughts

I hope by now you are convinced of the usefulness and necessity of Python virtual environments, in both development and production environments. Try them out, along with the additional tools provided by the community, and find the workflow that works best for you. And if you have an idea for improvement, share it. You’ll find that the Python community is very receptive to hearing new ideas.

About Atlantic.Net

Atlantic.Net provides leading cloud, dedicated and managed hosting services. Learn more about our reliable HIPAA-compliant cloud hosting solutions.

.
.