Using line_ Profiler evaluates and optimizes the performance of Python code

DechinPhy 2021-01-20 20:13:22
using line_ line profiler evaluates


The significance of performance testing

After finishing a python After the project , We often have to consider optimizing the performance of the software . So we need a software optimization idea , First of all, we need to make clear the bottleneck of software code and function , The ideal situation is to have such a tool , We can evaluate the performance of every line of code of an objective function , So we can target the worst performing part of all the code , For targeted optimization . Open source library line_profiler I did a job like this , Open source address :github.com/rkern/line_profiler. Let's take a look at the installation and use details of the tool .

line_profiler Installation

line_profiler Installation support source code installation and pip Installation , Here we only introduce pip Form of installation , It's easier , Please refer to the official open source address for source code installation .

[dechin@dechin-manjaro line_profiler]$ python3 -m pip install line_profiler
Collecting line_profiler
Downloading line_profiler-3.1.0-cp38-cp38-manylinux2010_x86_64.whl (65 kB)
|████████████████████████████████| 65 kB 221 kB/s
Requirement already satisfied: IPython in /home/dechin/anaconda3/lib/python3.8/site-packages (from line_profiler) (7.19.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (3.0.8)
Requirement already satisfied: backcall in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.2.0)
Requirement already satisfied: pexpect>4.3; sys_platform != "win32" in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.8.0)
Requirement already satisfied: setuptools>=18.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (50.3.1.post20201107)
Requirement already satisfied: jedi>=0.10 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.17.1)
Requirement already satisfied: decorator in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.4.2)
Requirement already satisfied: traitlets>=4.2 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (5.0.5)
Requirement already satisfied: pygments in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (2.7.2)
Requirement already satisfied: pickleshare in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.7.5)
Requirement already satisfied: wcwidth in /home/dechin/anaconda3/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->IPython->line_profiler) (0.2.5)
Requirement already satisfied: ptyprocess>=0.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from pexpect>4.3; sys_platform != "win32"->IPython->line_profiler) (0.6.0)
Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from jedi>=0.10->IPython->line_profiler) (0.7.0)
Requirement already satisfied: ipython-genutils in /home/dechin/anaconda3/lib/python3.8/site-packages (from traitlets>=4.2->IPython->line_profiler) (0.2.0)
Installing collected packages: line-profiler
Successfully installed line-profiler-3.1.0

Here is an additional introduction to a temporary use pip The source of the installation program , Here is what Tencent provides pypi Source :

python3 -m pip install -i https://mirrors.cloud.tencent.com/pypi/simple line_profiler

If you need to save the source permanently, you can modify it ~/.pip/pip.conf file , A reference example is as follows ( Using the image source of Huawei cloud ):

[global]
index-url = https://mirrors.huaweicloud.com/repository/pypi/simple
trusted-host = mirrors.huaweicloud.com
timeout = 120

Reference... In code that needs debugging optimization line_profiler

Let's look directly at a case :

# line_profiler_test.py
from line_profiler import LineProfiler
import numpy as np
@profile
def test_profiler():
for i in range(100):
a = np.random.randn(100)
b = np.random.randn(1000)
c = np.random.randn(10000)
return None
if __name__ == '__main__':
test_profiler()

In this case , We define a function that needs to be tested test_profiler, In this function, there are several lines of performance modules to be analyzed numpy.random.randn. The way to use it is first import Come in LineProfiler function , Then, the function named profile The decorator , It's done. line_profiler Configuration of performance analysis . About python The use and principle of decorators , You can refer to this one Blog The content of . Another thing to note is ,line_profiler The scope that can be analyzed is limited to the function content with decorator , If there are other calls in the function and so on , I won't go into other functions for analysis , Except for embedded nested functions .

Use line_profiler Perform a simple performance analysis

line_profiler It's also easy to use , There are two main steps : First use kernprof analysis , Then use python The analysis results are obtained by execution .

  1. After defining the function modules that need to be analyzed , use kernprof Parse into binary lprof file :
[dechin-manjaro line_profiler]# kernprof -l line_profiler_test.py
Wrote profile results to line_profiler_test.py.lprof

After the execution of the command , Will generate a... In the current directory lprof file :

[dechin-manjaro line_profiler]# ll
Total usage 8
-rw-r--r-- 1 dechin dechin 304 1 month 20 16:00 line_profiler_test.py
-rw-r--r-- 1 root root 185 1 month 20 16:00 line_profiler_test.py.lprof
  1. Use python3 function lprof Binary :
[dechin-manjaro line_profiler]# python3 -m line_profiler line_profiler_test.py.lprof
Timer unit: 1e-06 s
Total time: 0.022633 s
File: line_profiler_test.py
Function: test_profiler at line 5
Line # Hits Time Per Hit % Time Line Contents
==============================================================
5 @profile
6 def test_profiler():
7 101 40.0 0.4 0.2 for i in range(100):
8 100 332.0 3.3 1.5 a = np.random.randn(100)
9 100 2092.0 20.9 9.2 b = np.random.randn(1000)
10 100 20169.0 201.7 89.1 c = np.random.randn(10000)
11 1 0.0 0.0 0.0 return None

Here we directly get the line by line performance analysis conclusion . Briefly introduce the meaning of each column : The line number of the code in the code file 、 Number of calls 、 The total execution time of the line 、 Time spent on a single execution 、 The proportion of execution time under this function , The last column is the specific code content . Actually , About line_profiler This is the end of the introduction , But we hope to analyze it through another actual case line_profiler The function of , Interested readers can continue to read on .

Use line_profiler Analyze different function libraries and calculate sine function sin The efficiency of

We need to test the implementation in multiple libraries Sine function , It includes our own use of fortran Built in SIN function .

In the demo line_profiler Before the performance test of , Let's first look at how to make a fortran Of f90 File conversion to python Callable DLL file .

  1. First, in the Manjaro Linux On the platform gfotran
[dechin-manjaro line_profiler]# pacman -S gcc-fortran
Resolving dependencies ...
Looking for package conflicts ...
software package (1) gcc-fortran-10.2.0-4
Download size : 9.44 MiB
Full installation size : 31.01 MiB
:: Do you want to install ? [Y/n] Y
:: Getting package ......
gcc-fortran-10.2.0-4-x86_64 9.4 MiB 6.70 MiB/s 00:01 [#######################################################################################] 100%
(1/1) Checking the key in the key ring [#######################################################################################] 100%
(1/1) Checking package integrity [#######################################################################################] 100%
(1/1) Loading package file [#######################################################################################] 100%
(1/1) Checking for file conflicts [#######################################################################################] 100%
(1/1) Checking available storage [#######################################################################################] 100%
:: Processing package changes ...
(1/1) Installing gcc-fortran [#######################################################################################] 100%
:: Running post transaction hook function ...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the info directory file...
  1. Create a simple fortran file fmath.f90, The function is to return the value of sine function :
subroutine fsin(theta,result)
implicit none
real*8::theta
real*8,intent(out)::result
result=SIN(theta)
end subroutine
  1. use f2py Will be fortran The file is compiled into a file called fmath Dynamic link library for :
[dechin-manjaro line_profiler]# f2py -c -m fmath fmath.f90
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "fmath" sources
f2py options: []
f2py:> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c
creating /tmp/tmpup5ia9lf/src.linux-x86_64-3.8
Reading fortran codes...
Reading file 'fmath.f90' (format:free)
Post-processing...
Block: fmath
Block: fsin
Post-processing (stage 2)...
Building modules...
Building module "fmath"...
Constructing wrapper function "fsin"...
result = fsin(theta)
Wrote C/API module "fmath" to file "/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c"
adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c' to sources.
adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8' to include_dirs.
copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8
copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'fmath' extension
compiling C sources
C compiler: gcc -pthread -B /home/dechin/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
creating /tmp/tmpup5ia9lf/tmp
creating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf
creating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8
compile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'
gcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c
gcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c
In file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13,
from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c:15:
/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: Warning :#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
In file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13,
from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c:2:
/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: Warning :#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'
gfortran:f90: fmath.f90
/usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.o /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.o /tmp/tmpup5ia9lf/fmath.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -lgfortran -o ./fmath.cpython-38-x86_64-linux-gnu.so
Removing build directory /tmp/tmpup5ia9lf

There will be some alarms in the middle , But it doesn't affect our normal use , After compiling , You can see a so file ( If it is windows Platforms may be other types of DLL files ):

[dechin-manjaro line_profiler]# ll
Total usage 120
-rwxr-xr-x 1 root root 107256 1 month 20 16:40 fmath.cpython-38-x86_64-linux-gnu.so
-rw-r--r-- 1 root root 150 1 month 20 16:40 fmath.f90
-rw-r--r-- 1 dechin dechin 304 1 month 20 16:00 line_profiler_test.py
-rw-r--r-- 1 root root 185 1 month 20 16:00 line_profiler_test.py.lprof
  1. use ipython Test the function of the DLL :
[dechin-manjaro line_profiler]# ipython
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from fmath import fsin
In [2]: print (fsin(3.14))
0.0015926529164868282
In [3]: print (fsin(3.1415926))
5.3589793170057245e-08

Here we can see that based on fortran The function of sine function has been realized , Next, let's formally compare the performance of several sine function implementations ( The underlying implementation is likely to repeat , This is a black box for performance testing ).

First , We still need to create something to be tested python file sin_profiler_test.py

# sin_profiler_test.py
from line_profiler import LineProfiler
import random
from numpy import sin as numpy_sin
from math import sin as math_sin
# from cupy import sin as cupy_sin
from cmath import sin as cmath_sin
from fmath import fsin as fortran_sin
@profile
def test_profiler():
for i in range(100000):
r = random.random()
a = numpy_sin(r)
b = math_sin(r)
# c = cupy_sin(r)
d = cmath_sin(r)
e = fortran_sin(r)
return None
if __name__ == '__main__':
test_profiler()

here line_profiler The definition of is consistent with the previous example , Our main test objects are numpy,math,cmath Sine function implementation of four open source libraries and one of their own fortran The sine function of , Through the above introduction f2py The dynamic link library constructed follows python Achieve seamless docking . Because of the cupy The library was not installed successfully , So there's no way to test it here for the time being, and it's commented out . And then it's the same , adopt kernprof Build it :

[dechin-manjaro line_profiler]# kernprof -l sin_profiler_test.py
Wrote profile results to sin_profiler_test.py.lprof

Finally through python3 To execute :

[dechin-manjaro line_profiler]# python3 -m line_profiler sin_profiler_test.py.lprof
Timer unit: 1e-06 s
Total time: 0.261304 s
File: sin_profiler_test.py
Function: test_profiler at line 10
Line # Hits Time Per Hit % Time Line Contents
==============================================================
10 @profile
11 def test_profiler():
12 100001 28032.0 0.3 10.7 for i in range(100000):
13 100000 33995.0 0.3 13.0 r = random.random()
14 100000 86870.0 0.9 33.2 a = numpy_sin(r)
15 100000 33374.0 0.3 12.8 b = math_sin(r)
16 # c = cupy_sin(r)
17 100000 40179.0 0.4 15.4 d = cmath_sin(r)
18 100000 38854.0 0.4 14.9 e = fortran_sin(r)
19 1 0.0 0.0 0.0 return None

From this result we can see that , In the four libraries of this test ,math The computational efficiency of is the highest ,numpy The computational efficiency of is the lowest , And we wrote it ourselves fortran The interface function It's even better than that numpy It's twice as fast , Second only to math The implementation of the . Actually , Here, the value involves the performance test of a single function , We can also go through ipython The built-in timeit To test it :

[dechin-manjaro line_profiler]# ipython
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from fmath import fsin
In [2]: import random
In [3]: %timeit fsin(random.random())
145 ns ± 2.38 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
In [4]: from math import sin as math_sin
In [5]: %timeit math_sin(random.random())
107 ns ± 0.116 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
In [6]: from numpy import sin as numpy_sin
In [7]: %timeit numpy_sin(random.random())
611 ns ± 4.28 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [8]: from cmath import sin as cmath_sin
In [9]: %timeit cmath_sin(random.random())
151 ns ± 1.01 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In this result, we can see that the ranking trend is still consistent with the previous one , But because it will random Module and computing module together , There are some differences in the time values given .

Summary

This article focuses on the introduction python A line by line performance analysis tool line_profiler, Through the simple call of decorator, we can analyze the performance bottleneck of the program , So as to carry out targeted optimization . in addition , In the process of testing, we can also find that , Different forms of sine trigonometric functions are realized , There are differences in performance , It's just that it's not perceived when the frequency of daily use is low . What we need to know is , Even sine functions have many different implementations , For example, series expansion , And now it's the most popular 、 The highest performance computing method , In fact, it's still by looking up the table . therefore , Different algorithm implementations 、 Different language implementations , Will lead to completely different results . In terms of testing , The known performance ranking is :math<fortran<cmath<numpy The running time increases from left to right .

Copyright notice

The first link to this article is :https://www.cnblogs.com/dechinphy/p/line-profiler.html
author ID:DechinPhy
For more original articles, please refer to :https://www.cnblogs.com/dechinphy/

版权声明
本文为[DechinPhy]所创,转载请带上原文链接,感谢
https://pythonmana.com/2021/01/20210120194038977l.html

  1. A series of problems and solutions in Java calling Python
  2. python自动化爬取淘宝商品数据导入execl表格
  3. Using Python to automatically punch in the pin / enterprise wechat
  4. 【分享】python+requests接口测试基础
  5. Python automatically crawls Taobao product data and imports it into excel table
  6. C++/Python描述 628. 三个数的最大乘积
  7. Python的容器有哪些?分别有什么作用?
  8. python+requests接口测试基础
  9. 20 行代码:Serverless 架构下用 Python 轻松搞定图像分类和预测
  10. python+requests接口测试基础
  11. [share] Python + requests interface test foundation
  12. C + + / Python description 628. Maximum product of three numbers
  13. What are Python containers and what are their functions?
  14. Testing foundation of Python + requests interface
  15. 20 lines of code: easy to do image classification and prediction with Python under serverless architecture
  16. Python爬取优质高清壁纸网站:彼岸
  17. Testing foundation of Python + requests interface
  18. 【人生苦短,我学 Python】基础篇——列表(Day8)
  19. Python crawls high quality HD Wallpaper website: the other side
  20. Python图像增强与特效-利用百度AI进行黑白图像上色
  21. 【七天搞定Python】day01.Python环境配置、pip、IDE、注释、变量,数据类型、标识符/关键字、输出、输入
  22. Life is short, I learn Python
  23. Python image enhancement and special effects - using Baidu AI to color black and white images
  24. Python environment configuration, Pip, IDE, comment, variable, data type, identifier / keyword, output, input
  25. 为什么说Python是最伟大的语言?看图就知道了 - 知乎
  26. Why is Python the greatest language? Just look at the picture. - Zhihu
  27. 通过创建视频游戏来学习 Python
  28. Learn Python by creating video games
  29. Python3版本下创建计算给定日期范围内工作日方法
  30. Creating a method to calculate working days within a given date range in Python 3
  31. 图解爬虫,用几个最简单的例子带你入门Python爬虫
  32. Graphical crawler, with a few of the simplest examples to take you to the introduction of Python crawler
  33. python+requests基础知识
  34. Basic knowledge of Python + requests
  35. python自定义windowsr日志支持文件分割
  36. python+requests基础知识
  37. Python custom Windowsr log supports file segmentation
  38. Basic knowledge of Python + requests
  39. 高级测试 | Python笔试题
  40. 火了!开源的 Python 抢票神器,过年回家就看这一波了!
  41. Python 爬虫进阶 - 前后端分离有什么了不起,过程超详细!
  42. 【python】使用pip提示ModuleNotFoundError
  43. 【python】虚拟环境搭建
  44. Advanced test | Python written test questions
  45. Fire! Open source Python ticket grabbing artifact, come home to see this wave of New Year!
  46. Python crawler advanced - before and after the end of the separation of what great, super detailed process!
  47. [Python] prompt modulenotfounderror with PIP
  48. Building a virtual environment
  49. Serverless 架构下用 Python 轻松搞定图像分类和预测
  50. Easy image classification and prediction with Python under serverless architecture
  51. python协程爬取某网站的老赖数据
  52. Python coroutine crawls Laolai data of a website
  53. 使用Python分析姿态估计数据集COCO的教程
  54. Using Python to analyze the data set coco of attitude estimation
  55. win环境 python3 flask 上手整理 环境搭建(一)
  56. Getting started with win environment python3 flash
  57. Python实现一个论文下载器,赶紧收藏
  58. win环境 python3 flask 上手整理 快速上手-基础操作(二)
  59. Python 中常见的配置文件写法
  60. Python to achieve a paper Downloader, quickly collect