Using line_ Profiler to evaluate and optimize the performance of Python code

itread01 2021-01-20 21:14:30
using line_ line profiler evaluate


# The significance of performance testing is to finish a python After the project , We often have to consider optimizing the performance of the software . So we need a software optimization idea , First of all, we need to make clear the bottleneck of software code and function , The ideal situation is to have such a tool , The ability to evaluate the performance of each line of code for a target function , In this way, we can target the worst part of all the code , For targeted optimization . Open source library `line_profiler` I did a job like this , Open source address :[github.com/rkern/line_profiler](github.com/rkern/line_profiler). Let's take a look at the installation and use details of the tool .# line_profiler Installation of `line_profiler` The installation of support source code installation and pip Installation of , Here we only introduce pip Form of installation , It's easier , Please refer to the official open source address for source code installation .```bash[ [email protected] line_profiler]$ python3 -m pip install line_profilerCollecting line_profiler Downloading line_profiler-3.1.0-cp38-cp38-manylinux2010_x86_64.whl (65 kB) |████████████████████████████████| 65 kB 221 kB/s Requirement already satisfied: IPython in /home/dechin/anaconda3/lib/python3.8/site-packages (from line_profiler) (7.19.0)Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (3.0.8)Requirement already satisfied: backcall in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.2.0)Requirement already satisfied: pexpect>4.3; sys_platform != "win32" in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.8.0)Requirement already satisfied: setuptools>=18.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (50.3.1.post20201107)Requirement already satisfied: jedi>=0.10 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.17.1)Requirement already satisfied: decorator in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.4.2)Requirement already satisfied: traitlets>=4.2 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (5.0.5)Requirement already satisfied: pygments in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (2.7.2)Requirement already satisfied: pickleshare in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.7.5)Requirement already satisfied: wcwidth in /home/dechin/anaconda3/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->IPython->line_profiler) (0.2.5)Requirement already satisfied: ptyprocess>=0.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from pexpect>4.3; sys_platform != "win32"->IPython->line_profiler) (0.6.0)Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from jedi>=0.10->IPython->line_profiler) (0.7.0)Requirement already satisfied: ipython-genutils in /home/dechin/anaconda3/lib/python3.8/site-packages (from traitlets>=4.2->IPython->line_profiler) (0.2.0)Installing collected packages: line-profilerSuccessfully installed line-profiler-3.1.0``` Here is an additional introduction to a temporary use pip The source of the installation program , Here is what Tencent provides pypi Source :```bashpython3 -m pip install -i https://mirrors.cloud.tencent.com/pypi/simple line_profiler``` If you need a permanent storage source, you can modify it `~/.pip/pip.conf` Archives , A reference example is as follows ( Using the image source of Huawei cloud ):```bash[global]index-url = https://mirrors.huaweicloud.com/repository/pypi/simpletrusted-host = mirrors.huaweicloud.comtimeout = 120```# Reference in code that needs debugging optimization line_profiler Let's look directly at a case :```python# line_profiler_test.pyfrom line_profiler import LineProfilerimport numpy as np@profiledef test_profiler(): for i in range(100): a = np.random.randn(100) b = np.random.randn(1000) c = np.random.randn(10000) return Noneif __name__ == '__main__': test_profiler()``` In this case , We define a function that needs to be tested `test_profiler`, In this function, there are several lines of modules to be analyzed `numpy.random.randn`. The way to use it is first import Come in `LineProfiler` Function , Then, the function named `profile` The decorator , It's done. line_profiler Configuration of effectiveness analysis . About python The use and principle of decorators , You can refer to this article [ Blog ](https://www.cnblogs.com/dechinphy/p/decoretor.html) Introduction to . Another thing to note is ,line_profiler The scope of analysis is limited to the function with decorator , If there are other calls in the function and so on , I won't go into other functions for analysis , Except for the nested functions that are embedded .# Use line_profiler Perform a simple performance analysis line_profiler It's also easy to use , There are two main steps : First use `kernprof` analysis , Then use python The analysis results are obtained by execution .1. After defining the function modules that need to be analyzed , use `kernprof` It's a binary system `lprof` Archives :```bash[dechin-manjaro line_profiler]# kernprof -l line_profiler_test.py Wrote profile results to line_profiler_test.py.lprof``` At the end of the command , Will generate a... In the current directory `lprof` Archives :```bash[dechin-manjaro line_profiler]# ll Total dosage 8-rw-r--r-- 1 dechin dechin 304 1 month 20 16:00 line_profiler_test.py-rw-r--r-- 1 root root 185 1 month 20 16:00 line_profiler_test.py.lprof```2. Use `python3` Execute `lprof` Binary files :```bash[dechin-manjaro line_profiler]# python3 -m line_profiler line_profiler_test.py.lprof Timer unit: 1e-06 sTotal time: 0.022633 sFile: line_profiler_test.pyFunction: test_profiler at line 5Line # Hits Time Per Hit % Time Line Contents============================================================== 5 @profile 6 def test_profiler(): 7 101 40.0 0.4 0.2 for i in range(100): 8 100 332.0 3.3 1.5 a = np.random.randn(100) 9 100 2092.0 20.9 9.2 b = np.random.randn(1000) 10 100 20169.0 201.7 89.1 c = np.random.randn(10000) 11 1 0.0 0.0 0.0 return None``` Here we directly get the line by line efficiency analysis conclusion . Briefly introduce the meaning of each column : The line number of the code in the code file 、 The number of calls 、 The total execution time of the line 、 Time spent on a single execution 、 The proportion of execution time under this function , The last column is the specific code content . In fact , About `line_profiler` This is the end of the introduction , But we hope to analyze it through another actual case line_profiler The function of , Interested readers can continue to read on .# Use line_profiler Analyze different function libraries and calculate sine function sin We need to test the efficiency of multiple libraries ` Sine function `, It includes our own use of fortran Built in `SIN` Function . In the demo line_profiler Before the performance test of , Let's first look at how to make a fortran Of `f90` File conversion to python Callable DLL files .1. First, in the Manjaro Linux Installed on the platform gfotran```bash[dechin-manjaro line_profiler]# pacman -S gcc-fortran Resolving dependencies ... Querying package conflicts ... Software package (1) gcc-fortran-10.2.0-4 Download size : 9.44 MiB Full installation size : 31.01 MiB:: Do you want to install ? [Y/n] Y:: Getting package ...... gcc-fortran-10.2.0-4-x86_64 9.4 MiB 6.70 MiB/s 00:01 [#######################################################################################] 100%(1/1) Checking the key in the key ring [#######################################################################################] 100%(1/1) Checking package integrity [#######################################################################################] 100%(1/1) Loading package file [#######################################################################################] 100%(1/1) Checking for file conflicts [#######################################################################################] 100%(1/1) Checking available storage [#######################################################################################] 100%:: Processing package changes ...(1/1) Installing gcc-fortran [#######################################################################################] 100%:: Executing post transaction hook function ...(1/2) Arming ConditionNeedsUpdate...(2/2) Updating the info directory file...```2. Build a simple fortran Archives `fmath.f90`, The function is to return the value of the sine function :```fortransubroutine fsin(theta,result) implicit none real*8::theta real*8,intent(out)::result result=SIN(theta)end subroutine```3. use f2py Will be fortran The file is compiled into a file called `fmath` Dynamic link library of :```bash[dechin-manjaro line_profiler]# f2py -c -m fmath fmath.f90 running buildrunning config_ccunifing config_cc, config, build_clib, build_ext, build commands --compiler optionsrunning config_fcunifing config_fc, config, build_clib, build_ext, build commands --fcompiler optionsrunning build_srcbuild_srcbuilding extension "fmath" sourcesf2py options: []f2py:> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.ccreating /tmp/tmpup5ia9lf/src.linux-x86_64-3.8Reading fortran codes... Reading file 'fmath.f90' (format:free)Post-processing... Block: fmath Block: fsinPost-processing (stage 2)...Building modules... Building module "fmath"... Constructing wrapper function "fsin"... result = fsin(theta) Wrote C/API module "fmath" to file "/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c" adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c' to sources. adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8' to include_dirs.copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8build_src: building npy-pkg config filesrunning build_extcustomize UnixCCompilercustomize UnixCCompiler using build_extget_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'customize Gnu95FCompilerFound executable /usr/bin/gfortrancustomize Gnu95FCompilercustomize Gnu95FCompiler using build_extbuilding 'fmath' extensioncompiling C sourcesC compiler: gcc -pthread -B /home/dechin/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPICcreating /tmp/tmpup5ia9lf/tmpcreating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lfcreating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8compile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'gcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.cgcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.cIn file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c:15:/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: Warning :#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~~~~~In file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c:2:/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: Warning :#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~~~~~compiling Fortran sourcesFortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loopsFortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loopsFortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loopscompile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'gfortran:f90: fmath.f90/usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.o /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.o /tmp/tmpup5ia9lf/fmath.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -lgfortran -o ./fmath.cpython-38-x86_64-linux-gnu.soRemoving build directory /tmp/tmpup5ia9lf``` There will be some alarms in the middle , But it doesn't affect our normal use , After compiling , You can see a so Archives ( If it is windows Platforms may be other types of DLL files ):```bash[dechin-manjaro line_profiler]# ll Total dosage 120-rwxr-xr-x 1 root root 107256 1 month 20 16:40 fmath.cpython-38-x86_64-linux-gnu.so-rw-r--r-- 1 root root 150 1 month 20 16:40 fmath.f90-rw-r--r-- 1 dechin dechin 304 1 month 20 16:00 line_profiler_test.py-rw-r--r-- 1 root root 185 1 month 20 16:00 line_profiler_test.py.lprof```3. use ipython Test the function of the DLL :```bash[dechin-manjaro line_profiler]# ipythonPython 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more informationIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: from fmath import fsinIn [2]: print (fsin(3.14))0.0015926529164868282In [3]: print (fsin(3.1415926))5.3589793170057245e-08``` Here we can see that based on fortran The function of sine function of has been completed , Next, let's formally compare the performance of several sine function implementations ( The underlying implementation is likely to repeat , This is a black box for performance testing ). First , We still need to set up the test python Archives `sin_profiler_test.py`:```python# sin_profiler_test.pyfrom line_profiler import LineProfilerimport randomfrom numpy import sin as numpy_sinfrom math import sin as math_sin# from cupy import sin as cupy_sinfrom cmath import sin as cmath_sinfrom fmath import fsin as fortran_sin@profiledef test_profiler(): for i in range(100000): r = random.random() a = numpy_sin(r) b = math_sin(r) # c = cupy_sin(r) d = cmath_sin(r) e = fortran_sin(r) return Noneif __name__ == '__main__': test_profiler()``` Here `line_profiler` The definition of is consistent with the previous example , Our main test objects are `numpy,math,cmath` Sine function implementation of four open source libraries and one of their own fortran Sine function of , Through the above `f2py` The dynamic link library is constructed with python Seamless docking . Because of the `cupy` The library was not installed successfully , So there's no way to test it here for the time being, and it's commented out . And then it's the same , Through `kernprof` Compile build :```bash[dechin-manjaro line_profiler]# kernprof -l sin_profiler_test.py Wrote profile results to sin_profiler_test.py.lprof``` Finally passed `python3` To execute :```bash[dechin-manjaro line_profiler]# python3 -m line_profiler sin_profiler_test.py.lprof Timer unit: 1e-06 sTotal time: 0.261304 sFile: sin_profiler_test.pyFunction: test_profiler at line 10Line # Hits Time Per Hit % Time Line Contents============================================================== 10 @profile 11 def test_profiler(): 12 100001 28032.0 0.3 10.7 for i in range(100000): 13 100000 33995.0 0.3 13.0 r = random.random() 14 100000 86870.0 0.9 33.2 a = numpy_sin(r) 15 100000 33374.0 0.3 12.8 b = math_sin(r) 16 # c = cupy_sin(r) 17 100000 40179.0 0.4 15.4 d = cmath_sin(r) 18 100000 38854.0 0.4 14.9 e = fortran_sin(r) 19 1 0.0 0.0 0.0 return None``` From this result we can see that , In the four libraries of this test ,`math` The computational efficiency of is the highest ,`numpy` The computational efficiency of is the lowest , And we wrote it ourselves `fortran Interface functions ` It's even better than that `numpy` It's twice as fast , Only after `math` The realization of . In fact , Here, the value refers to the performance test of a single function , We can also go through `ipython` Built in `timeit` To test :```bash[dechin-manjaro line_profiler]# ipythonPython 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more informationIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: from fmath import fsinIn [2]: import randomIn [3]: %timeit fsin(random.random())145 ns ± 2.38 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)In [4]: from math import sin as math_sinIn [5]: %timeit math_sin(random.random())107 ns ± 0.116 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)In [6]: from numpy import sin as numpy_sinIn [7]: %timeit numpy_sin(random.random())611 ns ± 4.28 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)In [8]: from cmath import sin as cmath_sinIn [9]: %timeit cmath_sin(random.random())151 ns ± 1.01 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)``` In this result, we can see that the ranking trend is still consistent with the previous one , But because it will `random` Modules and computing modules together , There are some differences in the time values given .# Summary this paper focuses on python A line by line performance analysis tool `line_profiler`, Through a simple decorator call, we can analyze the performance bottleneck of the program , So as to carry out targeted optimization . in addition , In the process of testing, we can also find that , Different forms of sine trigonometric functions are implemented , There are differences in effectiveness , It's just that it's not perceived when the frequency of daily use is low . What we need to know is , Even sine functions have many different implementations , For example, series expansion , And now it's the most popular 、 The most efficient way of Computing , In fact, it's still by looking up the table . therefore , Different calculus 、 Different language implementations , Will lead to completely different results . In terms of testing , The known performance ranking is :`math`<`fortran`<`cmath`<`numpy` The execution time increases from left to right .# Copyright notice this article first link :[https://www.cnblogs.com/dechinphy/p/line-profiler.html](https://www.cnblogs.com/dechinphy/p/line-profiler.html) author ID:DechinPhy For more original articles, please refer to :[https://www.cnblogs.com/dechinphy/](https://www.cnblogs.com/dec
版权声明
本文为[itread01]所创,转载请带上原文链接,感谢
https://pythonmana.com/2021/01/20210120211222832N.html

  1. A series of problems and solutions in Java calling Python
  2. python自动化爬取淘宝商品数据导入execl表格
  3. Using Python to automatically punch in the pin / enterprise wechat
  4. 【分享】python+requests接口测试基础
  5. Python automatically crawls Taobao product data and imports it into excel table
  6. C++/Python描述 628. 三个数的最大乘积
  7. Python的容器有哪些?分别有什么作用?
  8. python+requests接口测试基础
  9. 20 行代码:Serverless 架构下用 Python 轻松搞定图像分类和预测
  10. python+requests接口测试基础
  11. [share] Python + requests interface test foundation
  12. C + + / Python description 628. Maximum product of three numbers
  13. What are Python containers and what are their functions?
  14. Testing foundation of Python + requests interface
  15. 20 lines of code: easy to do image classification and prediction with Python under serverless architecture
  16. Python爬取优质高清壁纸网站:彼岸
  17. Testing foundation of Python + requests interface
  18. 【人生苦短,我学 Python】基础篇——列表(Day8)
  19. Python crawls high quality HD Wallpaper website: the other side
  20. Python图像增强与特效-利用百度AI进行黑白图像上色
  21. 【七天搞定Python】day01.Python环境配置、pip、IDE、注释、变量,数据类型、标识符/关键字、输出、输入
  22. Life is short, I learn Python
  23. Python image enhancement and special effects - using Baidu AI to color black and white images
  24. Python environment configuration, Pip, IDE, comment, variable, data type, identifier / keyword, output, input
  25. 为什么说Python是最伟大的语言?看图就知道了 - 知乎
  26. Why is Python the greatest language? Just look at the picture. - Zhihu
  27. 通过创建视频游戏来学习 Python
  28. Learn Python by creating video games
  29. Python3版本下创建计算给定日期范围内工作日方法
  30. Creating a method to calculate working days within a given date range in Python 3
  31. 图解爬虫,用几个最简单的例子带你入门Python爬虫
  32. Graphical crawler, with a few of the simplest examples to take you to the introduction of Python crawler
  33. python+requests基础知识
  34. Basic knowledge of Python + requests
  35. python自定义windowsr日志支持文件分割
  36. python+requests基础知识
  37. Python custom Windowsr log supports file segmentation
  38. Basic knowledge of Python + requests
  39. 高级测试 | Python笔试题
  40. 火了!开源的 Python 抢票神器,过年回家就看这一波了!
  41. Python 爬虫进阶 - 前后端分离有什么了不起,过程超详细!
  42. 【python】使用pip提示ModuleNotFoundError
  43. 【python】虚拟环境搭建
  44. Advanced test | Python written test questions
  45. Fire! Open source Python ticket grabbing artifact, come home to see this wave of New Year!
  46. Python crawler advanced - before and after the end of the separation of what great, super detailed process!
  47. [Python] prompt modulenotfounderror with PIP
  48. Building a virtual environment
  49. Serverless 架构下用 Python 轻松搞定图像分类和预测
  50. Easy image classification and prediction with Python under serverless architecture
  51. python协程爬取某网站的老赖数据
  52. Python coroutine crawls Laolai data of a website
  53. 使用Python分析姿态估计数据集COCO的教程
  54. Using Python to analyze the data set coco of attitude estimation
  55. win环境 python3 flask 上手整理 环境搭建(一)
  56. Getting started with win environment python3 flash
  57. Python实现一个论文下载器,赶紧收藏
  58. win环境 python3 flask 上手整理 快速上手-基础操作(二)
  59. Python 中常见的配置文件写法
  60. Python to achieve a paper Downloader, quickly collect