Using line_ Profiler to evaluate and optimize the performance of Python code

itread01 2021-01-20 21:14:30
using line_ line profiler evaluate


# The significance of performance testing is to finish a python After the project , We often have to consider optimizing the performance of the software . So we need a software optimization idea , First of all, we need to make clear the bottleneck of software code and function , The ideal situation is to have such a tool , The ability to evaluate the performance of each line of code for a target function , In this way, we can target the worst part of all the code , For targeted optimization . Open source library `line_profiler` I did a job like this , Open source address :[github.com/rkern/line_profiler](github.com/rkern/line_profiler). Let's take a look at the installation and use details of the tool .# line_profiler Installation of `line_profiler` The installation of support source code installation and pip Installation of , Here we only introduce pip Form of installation , It's easier , Please refer to the official open source address for source code installation .```bash[ [email protected] line_profiler]$ python3 -m pip install line_profilerCollecting line_profiler Downloading line_profiler-3.1.0-cp38-cp38-manylinux2010_x86_64.whl (65 kB) |████████████████████████████████| 65 kB 221 kB/s Requirement already satisfied: IPython in /home/dechin/anaconda3/lib/python3.8/site-packages (from line_profiler) (7.19.0)Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (3.0.8)Requirement already satisfied: backcall in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.2.0)Requirement already satisfied: pexpect>4.3; sys_platform != "win32" in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.8.0)Requirement already satisfied: setuptools>=18.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (50.3.1.post20201107)Requirement already satisfied: jedi>=0.10 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.17.1)Requirement already satisfied: decorator in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.4.2)Requirement already satisfied: traitlets>=4.2 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (5.0.5)Requirement already satisfied: pygments in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (2.7.2)Requirement already satisfied: pickleshare in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.7.5)Requirement already satisfied: wcwidth in /home/dechin/anaconda3/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->IPython->line_profiler) (0.2.5)Requirement already satisfied: ptyprocess>=0.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from pexpect>4.3; sys_platform != "win32"->IPython->line_profiler) (0.6.0)Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from jedi>=0.10->IPython->line_profiler) (0.7.0)Requirement already satisfied: ipython-genutils in /home/dechin/anaconda3/lib/python3.8/site-packages (from traitlets>=4.2->IPython->line_profiler) (0.2.0)Installing collected packages: line-profilerSuccessfully installed line-profiler-3.1.0``` Here is an additional introduction to a temporary use pip The source of the installation program , Here is what Tencent provides pypi Source :```bashpython3 -m pip install -i https://mirrors.cloud.tencent.com/pypi/simple line_profiler``` If you need a permanent storage source, you can modify it `~/.pip/pip.conf` Archives , A reference example is as follows ( Using the image source of Huawei cloud ):```bash[global]index-url = https://mirrors.huaweicloud.com/repository/pypi/simpletrusted-host = mirrors.huaweicloud.comtimeout = 120```# Reference in code that needs debugging optimization line_profiler Let's look directly at a case :```python# line_profiler_test.pyfrom line_profiler import LineProfilerimport numpy as [email protected] test_profiler(): for i in range(100): a = np.random.randn(100) b = np.random.randn(1000) c = np.random.randn(10000) return Noneif __name__ == '__main__': test_profiler()``` In this case , We define a function that needs to be tested `test_profiler`, In this function, there are several lines of modules to be analyzed `numpy.random.randn`. The way to use it is first import Come in `LineProfiler` Function , Then, the function named `profile` The decorator , It's done. line_profiler Configuration of effectiveness analysis . About python The use and principle of decorators , You can refer to this article [ Blog ](https://www.cnblogs.com/dechinphy/p/decoretor.html) Introduction to . Another thing to note is ,line_profiler The scope of analysis is limited to the function with decorator , If there are other calls in the function and so on , I won't go into other functions for analysis , Except for the nested functions that are embedded .# Use line_profiler Perform a simple performance analysis line_profiler It's also easy to use , There are two main steps : First use `kernprof` analysis , Then use python The analysis results are obtained by execution .1. After defining the function modules that need to be analyzed , use `kernprof` It's a binary system `lprof` Archives :```bash[dechin-manjaro line_profiler]# kernprof -l line_profiler_test.py Wrote profile results to line_profiler_test.py.lprof``` At the end of the command , Will generate a... In the current directory `lprof` Archives :```bash[dechin-manjaro line_profiler]# ll Total dosage 8-rw-r--r-- 1 dechin dechin 304 1 month 20 16:00 line_profiler_test.py-rw-r--r-- 1 root root 185 1 month 20 16:00 line_profiler_test.py.lprof```2. Use `python3` Execute `lprof` Binary files :```bash[dechin-manjaro line_profiler]# python3 -m line_profiler line_profiler_test.py.lprof Timer unit: 1e-06 sTotal time: 0.022633 sFile: line_profiler_test.pyFunction: test_profiler at line 5Line # Hits Time Per Hit % Time Line Contents============================================================== 5 @profile 6 def test_profiler(): 7 101 40.0 0.4 0.2 for i in range(100): 8 100 332.0 3.3 1.5 a = np.random.randn(100) 9 100 2092.0 20.9 9.2 b = np.random.randn(1000) 10 100 20169.0 201.7 89.1 c = np.random.randn(10000) 11 1 0.0 0.0 0.0 return None``` Here we directly get the line by line efficiency analysis conclusion . Briefly introduce the meaning of each column : The line number of the code in the code file 、 The number of calls 、 The total execution time of the line 、 Time spent on a single execution 、 The proportion of execution time under this function , The last column is the specific code content . In fact , About `line_profiler` This is the end of the introduction , But we hope to analyze it through another actual case line_profiler The function of , Interested readers can continue to read on .# Use line_profiler Analyze different function libraries and calculate sine function sin We need to test the efficiency of multiple libraries ` Sine function `, It includes our own use of fortran Built in `SIN` Function . In the demo line_profiler Before the performance test of , Let's first look at how to make a fortran Of `f90` File conversion to python Callable DLL files .1. First, in the Manjaro Linux Installed on the platform gfotran```bash[dechin-manjaro line_profiler]# pacman -S gcc-fortran Resolving dependencies ... Querying package conflicts ... Software package (1) gcc-fortran-10.2.0-4 Download size : 9.44 MiB Full installation size : 31.01 MiB:: Do you want to install ? [Y/n] Y:: Getting package ...... gcc-fortran-10.2.0-4-x86_64 9.4 MiB 6.70 MiB/s 00:01 [#######################################################################################] 100%(1/1) Checking the key in the key ring [#######################################################################################] 100%(1/1) Checking package integrity [#######################################################################################] 100%(1/1) Loading package file [#######################################################################################] 100%(1/1) Checking for file conflicts [#######################################################################################] 100%(1/1) Checking available storage [#######################################################################################] 100%:: Processing package changes ...(1/1) Installing gcc-fortran [#######################################################################################] 100%:: Executing post transaction hook function ...(1/2) Arming ConditionNeedsUpdate...(2/2) Updating the info directory file...```2. Build a simple fortran Archives `fmath.f90`, The function is to return the value of the sine function :```fortransubroutine fsin(theta,result) implicit none real*8::theta real*8,intent(out)::result result=SIN(theta)end subroutine```3. use f2py Will be fortran The file is compiled into a file called `fmath` Dynamic link library of :```bash[dechin-manjaro line_profiler]# f2py -c -m fmath fmath.f90 running buildrunning config_ccunifing config_cc, config, build_clib, build_ext, build commands --compiler optionsrunning config_fcunifing config_fc, config, build_clib, build_ext, build commands --fcompiler optionsrunning build_srcbuild_srcbuilding extension "fmath" sourcesf2py options: []f2py:> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.ccreating /tmp/tmpup5ia9lf/src.linux-x86_64-3.8Reading fortran codes... Reading file 'fmath.f90' (format:free)Post-processing... Block: fmath Block: fsinPost-processing (stage 2)...Building modules... Building module "fmath"... Constructing wrapper function "fsin"... result = fsin(theta) Wrote C/API module "fmath" to file "/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c" adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c' to sources. adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8' to include_dirs.copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8build_src: building npy-pkg config filesrunning build_extcustomize UnixCCompilercustomize UnixCCompiler using build_extget_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'customize Gnu95FCompilerFound executable /usr/bin/gfortrancustomize Gnu95FCompilercustomize Gnu95FCompiler using build_extbuilding 'fmath' extensioncompiling C sourcesC compiler: gcc -pthread -B /home/dechin/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPICcreating /tmp/tmpup5ia9lf/tmpcreating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lfcreating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8compile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'gcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.cgcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.cIn file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c:15:/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: Warning :#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~~~~~In file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c:2:/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: Warning :#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~~~~~compiling Fortran sourcesFortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loopsFortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loopsFortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loopscompile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'gfortran:f90: fmath.f90/usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.o /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.o /tmp/tmpup5ia9lf/fmath.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -lgfortran -o ./fmath.cpython-38-x86_64-linux-gnu.soRemoving build directory /tmp/tmpup5ia9lf``` There will be some alarms in the middle , But it doesn't affect our normal use , After compiling , You can see a so Archives ( If it is windows Platforms may be other types of DLL files ):```bash[dechin-manjaro line_profiler]# ll Total dosage 120-rwxr-xr-x 1 root root 107256 1 month 20 16:40 fmath.cpython-38-x86_64-linux-gnu.so-rw-r--r-- 1 root root 150 1 month 20 16:40 fmath.f90-rw-r--r-- 1 dechin dechin 304 1 month 20 16:00 line_profiler_test.py-rw-r--r-- 1 root root 185 1 month 20 16:00 line_profiler_test.py.lprof```3. use ipython Test the function of the DLL :```bash[dechin-manjaro line_profiler]# ipythonPython 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more informationIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: from fmath import fsinIn [2]: print (fsin(3.14))0.0015926529164868282In [3]: print (fsin(3.1415926))5.3589793170057245e-08``` Here we can see that based on fortran The function of sine function of has been completed , Next, let's formally compare the performance of several sine function implementations ( The underlying implementation is likely to repeat , This is a black box for performance testing ). First , We still need to set up the test python Archives `sin_profiler_test.py`:```python# sin_profiler_test.pyfrom line_profiler import LineProfilerimport randomfrom numpy import sin as numpy_sinfrom math import sin as math_sin# from cupy import sin as cupy_sinfrom cmath import sin as cmath_sinfrom fmath import fsin as [email protected] test_profiler(): for i in range(100000): r = random.random() a = numpy_sin(r) b = math_sin(r) # c = cupy_sin(r) d = cmath_sin(r) e = fortran_sin(r) return Noneif __name__ == '__main__': test_profiler()``` Here `line_profiler` The definition of is consistent with the previous example , Our main test objects are `numpy,math,cmath` Sine function implementation of four open source libraries and one of their own fortran Sine function of , Through the above `f2py` The dynamic link library is constructed with python Seamless docking . Because of the `cupy` The library was not installed successfully , So there's no way to test it here for the time being, and it's commented out . And then it's the same , Through `kernprof` Compile build :```bash[dechin-manjaro line_profiler]# kernprof -l sin_profiler_test.py Wrote profile results to sin_profiler_test.py.lprof``` Finally passed `python3` To execute :```bash[dechin-manjaro line_profiler]# python3 -m line_profiler sin_profiler_test.py.lprof Timer unit: 1e-06 sTotal time: 0.261304 sFile: sin_profiler_test.pyFunction: test_profiler at line 10Line # Hits Time Per Hit % Time Line Contents============================================================== 10 @profile 11 def test_profiler(): 12 100001 28032.0 0.3 10.7 for i in range(100000): 13 100000 33995.0 0.3 13.0 r = random.random() 14 100000 86870.0 0.9 33.2 a = numpy_sin(r) 15 100000 33374.0 0.3 12.8 b = math_sin(r) 16 # c = cupy_sin(r) 17 100000 40179.0 0.4 15.4 d = cmath_sin(r) 18 100000 38854.0 0.4 14.9 e = fortran_sin(r) 19 1 0.0 0.0 0.0 return None``` From this result we can see that , In the four libraries of this test ,`math` The computational efficiency of is the highest ,`numpy` The computational efficiency of is the lowest , And we wrote it ourselves `fortran Interface functions ` It's even better than that `numpy` It's twice as fast , Only after `math` The realization of . In fact , Here, the value refers to the performance test of a single function , We can also go through `ipython` Built in `timeit` To test :```bash[dechin-manjaro line_profiler]# ipythonPython 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more informationIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: from fmath import fsinIn [2]: import randomIn [3]: %timeit fsin(random.random())145 ns ± 2.38 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)In [4]: from math import sin as math_sinIn [5]: %timeit math_sin(random.random())107 ns ± 0.116 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)In [6]: from numpy import sin as numpy_sinIn [7]: %timeit numpy_sin(random.random())611 ns ± 4.28 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)In [8]: from cmath import sin as cmath_sinIn [9]: %timeit cmath_sin(random.random())151 ns ± 1.01 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)``` In this result, we can see that the ranking trend is still consistent with the previous one , But because it will `random` Modules and computing modules together , There are some differences in the time values given .# Summary this paper focuses on python A line by line performance analysis tool `line_profiler`, Through a simple decorator call, we can analyze the performance bottleneck of the program , So as to carry out targeted optimization . in addition , In the process of testing, we can also find that , Different forms of sine trigonometric functions are implemented , There are differences in effectiveness , It's just that it's not perceived when the frequency of daily use is low . What we need to know is , Even sine functions have many different implementations , For example, series expansion , And now it's the most popular 、 The most efficient way of Computing , In fact, it's still by looking up the table . therefore , Different calculus 、 Different language implementations , Will lead to completely different results . In terms of testing , The known performance ranking is :`math`<`fortran`<`cmath`<`numpy` The execution time increases from left to right .# Copyright notice this article first link :[https://www.cnblogs.com/dechinphy/p/line-profiler.html](https://www.cnblogs.com/dechinphy/p/line-profiler.html) author ID:DechinPhy For more original articles, please refer to :[https://www.cnblogs.com/dechinphy/](https://www.cnblogs.com/dec
版权声明
本文为[itread01]所创,转载请带上原文链接,感谢
https://pythonmana.com/2021/01/20210120211222832N.html

  1. 利用Python爬虫获取招聘网站职位信息
  2. Using Python crawler to obtain job information of recruitment website
  3. Several highly rated Python libraries arrow, jsonpath, psutil and tenacity are recommended
  4. Python装饰器
  5. Python实现LDAP认证
  6. Python decorator
  7. Implementing LDAP authentication with Python
  8. Vscode configures Python development environment!
  9. In Python, how dare you say you can't log module? ️
  10. 我收藏的有关Python的电子书和资料
  11. python 中 lambda的一些tips
  12. python中字典的一些tips
  13. python 用生成器生成斐波那契数列
  14. python脚本转pyc踩了个坑。。。
  15. My collection of e-books and materials about Python
  16. Some tips of lambda in Python
  17. Some tips of dictionary in Python
  18. Using Python generator to generate Fibonacci sequence
  19. The conversion of Python script to PyC stepped on a pit...
  20. Python游戏开发,pygame模块,Python实现扫雷小游戏
  21. Python game development, pyGame module, python implementation of minesweeping games
  22. Python实用工具,email模块,Python实现邮件远程控制自己电脑
  23. Python utility, email module, python realizes mail remote control of its own computer
  24. 毫无头绪的自学Python,你可能连门槛都摸不到!【最佳学习路线】
  25. Python读取二进制文件代码方法解析
  26. Python字典的实现原理
  27. Without a clue, you may not even touch the threshold【 Best learning route]
  28. Parsing method of Python reading binary file code
  29. Implementation principle of Python dictionary
  30. You must know the function of pandas to parse JSON data - JSON_ normalize()
  31. Python实用案例,私人定制,Python自动化生成爱豆专属2021日历
  32. Python practical case, private customization, python automatic generation of Adu exclusive 2021 calendar
  33. 《Python实例》震惊了,用Python这么简单实现了聊天系统的脏话,广告检测
  34. "Python instance" was shocked and realized the dirty words and advertisement detection of the chat system in Python
  35. Convolutional neural network processing sequence for Python deep learning
  36. Python data structure and algorithm (1) -- enum type enum
  37. 超全大厂算法岗百问百答(推荐系统/机器学习/深度学习/C++/Spark/python)
  38. 【Python进阶】你真的明白NumPy中的ndarray吗?
  39. All questions and answers for algorithm posts of super large factories (recommended system / machine learning / deep learning / C + + / spark / Python)
  40. [advanced Python] do you really understand ndarray in numpy?
  41. 【Python进阶】Python进阶专栏栏主自述:不忘初心,砥砺前行
  42. [advanced Python] Python advanced column main readme: never forget the original intention and forge ahead
  43. python垃圾回收和缓存管理
  44. java调用Python程序
  45. java调用Python程序
  46. Python常用函数有哪些?Python基础入门课程
  47. Python garbage collection and cache management
  48. Java calling Python program
  49. Java calling Python program
  50. What functions are commonly used in Python? Introduction to Python Basics
  51. Python basic knowledge
  52. Anaconda5.2 安装 Python 库(MySQLdb)的方法
  53. Python实现对脑电数据情绪分析
  54. Anaconda 5.2 method of installing Python Library (mysqldb)
  55. Python implements emotion analysis of EEG data
  56. Master some advanced usage of Python in 30 seconds, which makes others envy it
  57. python爬取百度图片并对图片做一系列处理
  58. Python crawls Baidu pictures and does a series of processing on them
  59. python链接mysql数据库
  60. Python link MySQL database