Explain the principle and use of Python virtual environment

Yu Liang 2020-11-17 17:05:42
explain principle use python virtual

Click on Machine learning algorithm and Python Study , Choose star

Wonderful content doesn't get lost

This paper first introduces the basic knowledge of virtual environment and how to use it , Then I will introduce the working principle behind the virtual environment .( Environmental Science : stay macOS Mojave Use the latest version of Python 3.7.x)


  • Why use virtual environments ?
  • What is a virtual environment ?
  • Using virtual environments
  • management environment
  • How virtual environments work ?

1. Why use virtual environments ?

Virtual environments provide simple solutions to a range of potential problems , Especially in the following aspects :

  • Allow different projects to use different versions of packages , To solve the dependency problem . for example , Can be Project A v2.7 be used for Project X, And will Package A v1.3 be used for Project Y.
  • By capturing all package dependencies in the requirements document , Make the project self-contained and reproducible .
  • Install the package on a host without administrator rights .
  • Just one project , No need to install software packages on a system wide basis , To keep the whole picture site-packages / The catalogue is neat and tidy .

It sounds convenient , isn't it? ? When you start building more complex projects and working with others , The importance of virtual environment will be highlighted . Many data scientists also need to be familiar with the multilingual aspects of virtual environments Conda Environmental Science .

You can use it in order !

2. What is a virtual environment ?

What is virtual environment ?

Virtual environments are used for dependency management and project isolation Python Tools , allow Python Site package ( Third party Library ) Installed in the isolated directory of a local specific project , Instead of global installation ( That is, as system wide Python Part of ).

That sounds good , But what exactly is a virtual environment ? A virtual environment is just a directory with three important components :

  • With third party libraries installed site-packages / Folder .
  • Installed on the system Python Of the executable symlink A symbolic link .
  • To ensure implementation Python The script of the code uses... Installed in a given virtual environment Python Interpreter and site package .

The last point is that unexpected mistakes can occur , I'll talk about that later , But let's take a look at how to actually use virtual environments in practice .

3. Using virtual environments

Create a virtual environment

Suppose you want to create a project called test-project/ Virtual environment for , The project has the following tree :


├── data ├── deliver # Final analysis, code, & presentations ├── develop # Notebooks for exploratory analysis ├── src # Scripts & local project modules └── tests

You need to perform venv modular , It is Python Part of the standard library .

% cd test-project/ % python3 -m venv venv/ # Creates an environment called venv/

Be careful : You can replace... With a different environment name “venv/”.

look ! Virtual environment was born . Now the project becomes :

├── data
├── deliver
├── develop
├── src
├── tests
└── venv # There it is!

remind : The virtual environment itself is a directory .

The only thing to do is to run the script mentioned earlier “ Activate ” Environmental Science .

% source venv/bin/activate (venv) % # Fancy new command prompt

Now we're in the active virtual environment ( Indicated by a command prompt , Prefix with the name of the active environment ).

We will deal with the project as usual , Make sure that the project is completely isolated from the rest of the system . In a virtual environment , We can't access system wide site packages , And cannot access the installation package outside of the virtual environment .

On completion of project work , You can exit the environment with the following code :

(venv) % deactivate % # Old familiar command prompt

Installation package

By default , Install only in new environments pip and setuptools.

(venv) % pip list # Inside an active environmentPackage Version ---------- ------- pip 19.1.1 setuptools 40.8.0

If you want to install a specific version of the third-party library , such as numpyv1.15.3, Can be used as usual pip.

(venv) % pip install numpy==1.15.3 (venv) % pip listPackage Version ---------- ------- numpy 1.15.3 pip 19.1.1 setuptools 40.8.0

Now available in script or active Python shell Import numpy. for example , Suppose the project contains the following lines of script tests / imports-test.py.

#!/usr/bin/env python3
import numpy as np

When you run this script directly from the command line , available :

(venv) % tests/imports-test.py (venv) % # Look, Ma, no errors!

success . Script import numpy No fault .

4. management environment

Requirements document

The easiest way to make our work available for reuse is at the root of the project ( Top level directory ) Add a requirement document to . So , Need to run pip freeze, The installed third-party packages and their version numbers are listed below :

(venv) % pip freeze

And write the output to a file , We call it requirements.txt.

(venv) % pip freeze > requirements.txt

When updating a package or installing a new package , You can rewrite the requirements file with the same command .

Now? , Anyone who shares a project can use requirements.txt file , Run the project on the system by copying the environment .

Copy the environment

wait —— How did it work ?

Imagine , Our teammates Sara From the team's GitHub Test project deleted from Repository . On her system , The directory tree for the project is shown below :

├── data
├── deliver
├── develop
├── requirements.txt
├── src
└── tests

Notice something unusual ? Yes , you 're right ! No, venv / Folder .

We've taken it from the team's GitHub Delete from repository , Because its existence may cause trouble . This is the use of requirements.txt One reason why files are critical to copying project code .

To run the test project on the machine ,Sara All you need to do is create a virtual environment in the root of the project :

Sara% cd test-project/
Sara% python3 -m venv venv/

And use pip install -r requirements.txt Install the project's dependencies in an active virtual environment .

Sara% source venv/bin/activate
(venv) Sara% pip install -r requirements.txt
Collecting numpy==1.15.3 (from -r i (line 1))
Installing collected packages: numpy
Successfully installed numpy-1.15.3

Now? ,Sara The project environment on the system is exactly the same as ours . It's neat , isn't it? ?


Unfortunately things don't always go as planned , There are always problems . Maybe by mistake updating a specific site package and finding myself in Dependency Hell The Ninth level of , Cannot run single line project code . Maybe it's not that bad , Maybe you will find yourself in the seventh level .

No matter how much you find yourself , The easiest way to solve the problem and see hope again is to recreate the virtual environment of the project .

% rm -r venv/ # Nukes the old environment
% python3 -m venv venv/ # Makes a blank new one
% pip install -r requirements.txt # Re-installs dependencies

Be accomplished , Thanks a lot requirements.txt file , It's back to normal again . However, another reason is that the requirements document should always be included in the project .

5. How virtual environments do this ?

Want to learn more about virtual environments ? such as , How to use the right environment Python Explain the program and find the right third-party library ?

echo $ PATH

It all boils down to PATH The value of , It tells shell What to use Python Examples and where to find website packages . On the basis of shell in ,PATH It seems more or less like this .

% echo $PATH

call Python Interpreter or run .py Script time ,shell Will search in order PATH The contents listed in , Until I met Python example . To see PATH First found Python example , Please run which python3.

% which python3 /usr/local/bin/python3 # Your output may differ

Through the site module ( This is a Python Part of the standard library ) Find this Python It's also easy for instance to find the location of the site package .

% python3 # Activates a Python shell >>> import site >>> site.getsitepackages() # Points to site-packages folder[ /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages ]

Run script venv / bin / activate modify PATH, In order to shell Search the project's local binaries before searching the system's global binaries .

% cd ~/test-project/ % source venv/bin/activate (ven) % echo $PATH~/test-project/venv/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin

Now? shell Know how to use the project's local Python example :

(venv) % which python3 ~/test-project/venv/bin/python3

Where can I find the local site package for the project ?

(venv) % python3 >>> import site >>> site.getsitepackages()[ ~/test-project/venv/lib/python3.7/site-packages ] # Ka-ching

Sanity check

Remember the old tests / imports-test.py Script ? It looks like the following :

#!/usr/bin/env python3

import numpy as np

We can run this script in an active environment , There's no problem , It's because of the environment Python Instances can access the project's local site package .

What happens if you run the same script from outside the project's virtual environment ?

% tests/imports-test.py # Look, no active environmentTraceback (most recent call last): File "tests/imports-test.py", line 3, in <module> import numpy as npModuleNotFoundError: No module named numpy

Yes , An error occurred , But we should do this . If we don't , That means we can access the project's local site package from outside the project , And thus undermines the whole purpose of having a virtual environment . The fact that something went wrong proved that our project was completely isolated from the rest of the system .

The directory tree of the environment

One thing can help you organize all this information , That is, to understand the appearance of the environment tree clearly .

test-project/venv/ # Our environment s root directory ├── bin │ ├── activate # Scripts to activate │ ├── activate.csh # our project s │ ├── activate.fish # virtual environment. │ ├── easy_install │ ├── easy_install-3.7 │ ├── pip │ ├── pip3 │ ├── pip3.7 │ ├── python -> /usr/local/bin/python # Symlinks to system-wide │ └── python3 -> python3.7 # Python instances. ├── include ├── lib │ └── python3.7 │ └── site-packages # Stores local site packages └── pyvenv.cfg

This article is from WeChat official account. - Machine learning algorithm and Python Study (MLPython)

The source and reprint of the original text are detailed in the text , If there is any infringement , Please contact the yunjia_community@tencent.com Delete .

Original publication time : 2020-11-11

Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .

本文为[Yu Liang]所创,转载请带上原文链接,感谢

  1. 利用Python爬虫获取招聘网站职位信息
  2. Using Python crawler to obtain job information of recruitment website
  3. Several highly rated Python libraries arrow, jsonpath, psutil and tenacity are recommended
  4. Python装饰器
  5. Python实现LDAP认证
  6. Python decorator
  7. Implementing LDAP authentication with Python
  8. Vscode configures Python development environment!
  9. In Python, how dare you say you can't log module? ️
  10. 我收藏的有关Python的电子书和资料
  11. python 中 lambda的一些tips
  12. python中字典的一些tips
  13. python 用生成器生成斐波那契数列
  14. python脚本转pyc踩了个坑。。。
  15. My collection of e-books and materials about Python
  16. Some tips of lambda in Python
  17. Some tips of dictionary in Python
  18. Using Python generator to generate Fibonacci sequence
  19. The conversion of Python script to PyC stepped on a pit...
  20. Python游戏开发,pygame模块,Python实现扫雷小游戏
  21. Python game development, pyGame module, python implementation of minesweeping games
  22. Python实用工具,email模块,Python实现邮件远程控制自己电脑
  23. Python utility, email module, python realizes mail remote control of its own computer
  24. 毫无头绪的自学Python,你可能连门槛都摸不到!【最佳学习路线】
  25. Python读取二进制文件代码方法解析
  26. Python字典的实现原理
  27. Without a clue, you may not even touch the threshold【 Best learning route]
  28. Parsing method of Python reading binary file code
  29. Implementation principle of Python dictionary
  30. You must know the function of pandas to parse JSON data - JSON_ normalize()
  31. Python实用案例,私人定制,Python自动化生成爱豆专属2021日历
  32. Python practical case, private customization, python automatic generation of Adu exclusive 2021 calendar
  33. 《Python实例》震惊了,用Python这么简单实现了聊天系统的脏话,广告检测
  34. "Python instance" was shocked and realized the dirty words and advertisement detection of the chat system in Python
  35. Convolutional neural network processing sequence for Python deep learning
  36. Python data structure and algorithm (1) -- enum type enum
  37. 超全大厂算法岗百问百答(推荐系统/机器学习/深度学习/C++/Spark/python)
  38. 【Python进阶】你真的明白NumPy中的ndarray吗?
  39. All questions and answers for algorithm posts of super large factories (recommended system / machine learning / deep learning / C + + / spark / Python)
  40. [advanced Python] do you really understand ndarray in numpy?
  41. 【Python进阶】Python进阶专栏栏主自述:不忘初心,砥砺前行
  42. [advanced Python] Python advanced column main readme: never forget the original intention and forge ahead
  43. python垃圾回收和缓存管理
  44. java调用Python程序
  45. java调用Python程序
  46. Python常用函数有哪些?Python基础入门课程
  47. Python garbage collection and cache management
  48. Java calling Python program
  49. Java calling Python program
  50. What functions are commonly used in Python? Introduction to Python Basics
  51. Python basic knowledge
  52. Anaconda5.2 安装 Python 库(MySQLdb)的方法
  53. Python实现对脑电数据情绪分析
  54. Anaconda 5.2 method of installing Python Library (mysqldb)
  55. Python implements emotion analysis of EEG data
  56. Master some advanced usage of Python in 30 seconds, which makes others envy it
  57. python爬取百度图片并对图片做一系列处理
  58. Python crawls Baidu pictures and does a series of processing on them
  59. python链接mysql数据库
  60. Python link MySQL database