使用line_profiler對python程式碼效能進行評估優化

itread01 2021-01-20 21:13:22
Python 使用 line line_profiler profiler


# 效能測試的意義在做完一個python專案之後,我們經常要考慮對軟體的效能進行優化。那麼我們需要一個軟體優化的思路,首先我們需要明確軟體本身程式碼以及函式的瓶頸,最理想的情況就是有這樣一個工具,能夠將一個目標函式的程式碼每一行的效能都評估出來,這樣我們可以針對所有程式碼中效能最差的那一部分,來進行鍼對性的優化。開源庫`line_profiler`就做了一個這樣的工作,開源地址:[github.com/rkern/line_profiler](github.com/rkern/line_profiler)。下面讓我們一起看下該工具的安裝和使用詳情。# line_profiler的安裝`line_profiler`的安裝支援原始碼安裝和pip的安裝,這裡我們僅介紹pip形式的安裝,也比較容易,原始碼安裝方式請參考官方開源地址。```bash[ [email protected] line_profiler]$ python3 -m pip install line_profilerCollecting line_profiler Downloading line_profiler-3.1.0-cp38-cp38-manylinux2010_x86_64.whl (65 kB) |████████████████████████████████| 65 kB 221 kB/s Requirement already satisfied: IPython in /home/dechin/anaconda3/lib/python3.8/site-packages (from line_profiler) (7.19.0)Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (3.0.8)Requirement already satisfied: backcall in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.2.0)Requirement already satisfied: pexpect>4.3; sys_platform != "win32" in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.8.0)Requirement already satisfied: setuptools>=18.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (50.3.1.post20201107)Requirement already satisfied: jedi>=0.10 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.17.1)Requirement already satisfied: decorator in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (4.4.2)Requirement already satisfied: traitlets>=4.2 in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (5.0.5)Requirement already satisfied: pygments in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (2.7.2)Requirement already satisfied: pickleshare in /home/dechin/anaconda3/lib/python3.8/site-packages (from IPython->line_profiler) (0.7.5)Requirement already satisfied: wcwidth in /home/dechin/anaconda3/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->IPython->line_profiler) (0.2.5)Requirement already satisfied: ptyprocess>=0.5 in /home/dechin/anaconda3/lib/python3.8/site-packages (from pexpect>4.3; sys_platform != "win32"->IPython->line_profiler) (0.6.0)Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from jedi>=0.10->IPython->line_profiler) (0.7.0)Requirement already satisfied: ipython-genutils in /home/dechin/anaconda3/lib/python3.8/site-packages (from traitlets>=4.2->IPython->line_profiler) (0.2.0)Installing collected packages: line-profilerSuccessfully installed line-profiler-3.1.0```這裡額外介紹一種臨時使用pip的源進行安裝的方案,這裡用到的是騰訊所提供的pypi源:```bashpython3 -m pip install -i https://mirrors.cloud.tencent.com/pypi/simple line_profiler```如果需要永久儲存源可以修改`~/.pip/pip.conf`檔案,一個參考示例如下(採用華為雲的映象源):```bash[global]index-url = https://mirrors.huaweicloud.com/repository/pypi/simpletrusted-host = mirrors.huaweicloud.comtimeout = 120```# 在需要除錯優化的程式碼中引用line_profiler讓我們直接來看一個案例:```python# line_profiler_test.pyfrom line_profiler import LineProfilerimport numpy as [email protected] test_profiler(): for i in range(100): a = np.random.randn(100) b = np.random.randn(1000) c = np.random.randn(10000) return Noneif __name__ == '__main__': test_profiler()```在這個案例中,我們定義了一個需要測試的函式`test_profiler`,在這個函式中有幾行待分析效能的模組`numpy.random.randn`。使用的方式就是先import進來`LineProfiler`函式,然後在需要逐行進行效能分析的函式上方引用名為`profile`的裝飾器,就完成了line_profiler效能分析的配置。關於python裝飾器的使用和原理,可以參考這篇[部落格](https://www.cnblogs.com/dechinphy/p/decoretor.html)的內容介紹。還有一點需要注意的是,line_profiler所能夠分析的範圍僅限於加了裝飾器的函式內容,如果函式內有其他的呼叫之類的,不會再進入其他的函式進行分析,除了內嵌的巢狀函式。# 使用line_profiler進行簡單效能分析line_profiler的使用方法也較為簡單,主要就是兩步:先用`kernprof`解析,再採用python執行得到分析結果。1. 在定義好需要分析的函式模組之後,用`kernprof`解析成二進位制`lprof`檔案:```bash[dechin-manjaro line_profiler]# kernprof -l line_profiler_test.py Wrote profile results to line_profiler_test.py.lprof```該命令執行結束後,會在當前目錄下產生一個`lprof`檔案:```bash[dechin-manjaro line_profiler]# ll總用量 8-rw-r--r-- 1 dechin dechin 304 1月 20 16:00 line_profiler_test.py-rw-r--r-- 1 root root 185 1月 20 16:00 line_profiler_test.py.lprof```2. 使用`python3`執行`lprof`二進位制檔案:```bash[dechin-manjaro line_profiler]# python3 -m line_profiler line_profiler_test.py.lprof Timer unit: 1e-06 sTotal time: 0.022633 sFile: line_profiler_test.pyFunction: test_profiler at line 5Line # Hits Time Per Hit % Time Line Contents============================================================== 5 @profile 6 def test_profiler(): 7 101 40.0 0.4 0.2 for i in range(100): 8 100 332.0 3.3 1.5 a = np.random.randn(100) 9 100 2092.0 20.9 9.2 b = np.random.randn(1000) 10 100 20169.0 201.7 89.1 c = np.random.randn(10000) 11 1 0.0 0.0 0.0 return None```這裡我們就直接得到了逐行的效能分析結論。簡單介紹一下每一列的含義:程式碼在程式碼檔案中對應的行號、被呼叫的次數、該行的總共執行時間、單次執行所消耗的時間、執行時間在該函式下的佔比,最後一列是具體的程式碼內容。其實,關於`line_profiler`的使用介紹到這裡就可以結束了,但是我們希望通過另外一個實際案例來分析line_profiler的功能,感興趣的讀者可以繼續往下閱讀。# 使用line_profiler分析不同函式庫計算正弦函式sin的效率我們這裡需要測試多個庫中所實現的`正弦函式`,其中包含我們自己使用的fortran內建的`SIN`函式。在演示line_profiler的效能測試之前,讓我們先看看如何將一個fortran的`f90`檔案轉換成python可呼叫的動態連結庫檔案。1. 首先在Manjaro Linux平臺上安裝gfotran```bash[dechin-manjaro line_profiler]# pacman -S gcc-fortran正在解析依賴關係...正在查詢軟體包衝突...軟體包 (1) gcc-fortran-10.2.0-4下載大小: 9.44 MiB全部安裝大小: 31.01 MiB:: 進行安裝嗎? [Y/n] Y:: 正在獲取軟體包...... gcc-fortran-10.2.0-4-x86_64 9.4 MiB 6.70 MiB/s 00:01 [#######################################################################################] 100%(1/1) 正在檢查金鑰環裡的金鑰 [#######################################################################################] 100%(1/1) 正在檢查軟體包完整性 [#######################################################################################] 100%(1/1) 正在載入軟體包檔案 [#######################################################################################] 100%(1/1) 正在檢查檔案衝突 [#######################################################################################] 100%(1/1) 正在檢查可用儲存空間 [#######################################################################################] 100%:: 正在處理軟體包的變化...(1/1) 正在安裝 gcc-fortran [#######################################################################################] 100%:: 正在執行事務後鉤子函式...(1/2) Arming ConditionNeedsUpdate...(2/2) Updating the info directory file...```2. 建立一個簡單的fortran檔案`fmath.f90`,功能為返回正弦函式的值:```fortransubroutine fsin(theta,result) implicit none real*8::theta real*8,intent(out)::result result=SIN(theta)end subroutine```3. 用f2py將該fortran檔案編譯成名為`fmath`的動態連結庫:```bash[dechin-manjaro line_profiler]# f2py -c -m fmath fmath.f90 running buildrunning config_ccunifing config_cc, config, build_clib, build_ext, build commands --compiler optionsrunning config_fcunifing config_fc, config, build_clib, build_ext, build commands --fcompiler optionsrunning build_srcbuild_srcbuilding extension "fmath" sourcesf2py options: []f2py:> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.ccreating /tmp/tmpup5ia9lf/src.linux-x86_64-3.8Reading fortran codes... Reading file 'fmath.f90' (format:free)Post-processing... Block: fmath Block: fsinPost-processing (stage 2)...Building modules... Building module "fmath"... Constructing wrapper function "fsin"... result = fsin(theta) Wrote C/API module "fmath" to file "/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c" adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c' to sources. adding '/tmp/tmpup5ia9lf/src.linux-x86_64-3.8' to include_dirs.copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8copying /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpup5ia9lf/src.linux-x86_64-3.8build_src: building npy-pkg config filesrunning build_extcustomize UnixCCompilercustomize UnixCCompiler using build_extget_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'customize Gnu95FCompilerFound executable /usr/bin/gfortrancustomize Gnu95FCompilercustomize Gnu95FCompiler using build_extbuilding 'fmath' extensioncompiling C sourcesC compiler: gcc -pthread -B /home/dechin/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPICcreating /tmp/tmpup5ia9lf/tmpcreating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lfcreating /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8compile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'gcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.cgcc: /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.cIn file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.c:15:/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: 警告:#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~~~~~In file included from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.h:13, from /tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.c:2:/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: 警告:#warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~~~~~compiling Fortran sourcesFortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loopsFortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loopsFortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loopscompile options: '-I/tmp/tmpup5ia9lf/src.linux-x86_64-3.8 -I/home/dechin/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/dechin/anaconda3/include/python3.8 -c'gfortran:f90: fmath.f90/usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fmathmodule.o /tmp/tmpup5ia9lf/tmp/tmpup5ia9lf/src.linux-x86_64-3.8/fortranobject.o /tmp/tmpup5ia9lf/fmath.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib -lgfortran -o ./fmath.cpython-38-x86_64-linux-gnu.soRemoving build directory /tmp/tmpup5ia9lf```這中間會有一些告警,但是並不影響我們的正常使用,編譯好之後,可以在當前目錄下看到一個so檔案(如果是windows平臺可能是其他型別的動態連結庫檔案):```bash[dechin-manjaro line_profiler]# ll總用量 120-rwxr-xr-x 1 root root 107256 1月 20 16:40 fmath.cpython-38-x86_64-linux-gnu.so-rw-r--r-- 1 root root 150 1月 20 16:40 fmath.f90-rw-r--r-- 1 dechin dechin 304 1月 20 16:00 line_profiler_test.py-rw-r--r-- 1 root root 185 1月 20 16:00 line_profiler_test.py.lprof```3. 用ipython測試該動態連結庫的功能是否正常:```bash[dechin-manjaro line_profiler]# ipythonPython 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more informationIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: from fmath import fsinIn [2]: print (fsin(3.14))0.0015926529164868282In [3]: print (fsin(3.1415926))5.3589793170057245e-08```這裡我們可以看到基於fortran的正弦函式的功能已經完成實現了,接下來讓我們正式對比幾種正弦函式實現的效能(底層的實現有可能重複,這裡作為黑盒來進行效能測試)。首先,我們還是需要建立好待測試的python檔案`sin_profiler_test.py`:```python# sin_profiler_test.pyfrom line_profiler import LineProfilerimport randomfrom numpy import sin as numpy_sinfrom math import sin as math_sin# from cupy import sin as cupy_sinfrom cmath import sin as cmath_sinfrom fmath import fsin as [email protected] test_profiler(): for i in range(100000): r = random.random() a = numpy_sin(r) b = math_sin(r) # c = cupy_sin(r) d = cmath_sin(r) e = fortran_sin(r) return Noneif __name__ == '__main__': test_profiler()```這裡`line_profiler`的定義跟前面定義的例子一致,我們主要測試的物件為`numpy,math,cmath`四個開源庫的正弦函式實現以及自己實現的一個fortran的正弦函式,通過上面介紹的`f2py`構造的動態連結庫跟python實現無縫對接。由於這裡的`cupy`庫沒有安裝成功,所以這裡暫時沒辦法測試而註釋掉了。接下來還是一樣的,通過`kernprof`進行編譯構建:```bash[dechin-manjaro line_profiler]# kernprof -l sin_profiler_test.py Wrote profile results to sin_profiler_test.py.lprof```最後通過`python3`來執行:```bash[dechin-manjaro line_profiler]# python3 -m line_profiler sin_profiler_test.py.lprof Timer unit: 1e-06 sTotal time: 0.261304 sFile: sin_profiler_test.pyFunction: test_profiler at line 10Line # Hits Time Per Hit % Time Line Contents============================================================== 10 @profile 11 def test_profiler(): 12 100001 28032.0 0.3 10.7 for i in range(100000): 13 100000 33995.0 0.3 13.0 r = random.random() 14 100000 86870.0 0.9 33.2 a = numpy_sin(r) 15 100000 33374.0 0.3 12.8 b = math_sin(r) 16 # c = cupy_sin(r) 17 100000 40179.0 0.4 15.4 d = cmath_sin(r) 18 100000 38854.0 0.4 14.9 e = fortran_sin(r) 19 1 0.0 0.0 0.0 return None```從這個結果上我們可以看出,在這測試的四個庫中,`math`的計算效率是最高的,`numpy`的計算效率是最低的,而我們自己編寫的`fortran介面函式`甚至都比`numpy`的實現快了一倍,僅次於`math`的實現。其實,這裡值涉及到了單個函式的效能測試,我們還可以通過`ipython`中自帶的`timeit`來進行測試:```bash[dechin-manjaro line_profiler]# ipythonPython 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more informationIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: from fmath import fsinIn [2]: import randomIn [3]: %timeit fsin(random.random())145 ns ± 2.38 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)In [4]: from math import sin as math_sinIn [5]: %timeit math_sin(random.random())107 ns ± 0.116 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)In [6]: from numpy import sin as numpy_sinIn [7]: %timeit numpy_sin(random.random())611 ns ± 4.28 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)In [8]: from cmath import sin as cmath_sinIn [9]: %timeit cmath_sin(random.random())151 ns ± 1.01 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)```在這個結果中我們看到排名的趨勢依然跟之前的保持一致,但是由於將`random`模組和計算模組放在一起,在給出的時間數值上有些差異。# 總結概要本文重點介紹了python的一款逐行效能分析的工具`line_profiler`,通過簡單的裝飾器的呼叫就可以分析出程式的效能瓶頸,從而進行鍼對性的優化。另外,在測試的過程中我們還可以發現,不同形式的正弦三角函式實現,效能是存在差異的,只是在日常使用頻率較低的情況下是不感知的。需要了解的是,即使是正弦函式也有很多不同的實現方案,比如各種級數展開,而目前最流行、效能最高的計算方式,其實還是通過查表法。因此,不同的演算法實現、不同的語言實現,都會導致完全不一樣的結果。就測試情況而言,已知的效能排名為:`math`<`fortran`<`cmath`<`numpy`從左到右執行時長逐步增加。# 版權宣告本文首發連結為:[https://www.cnblogs.com/dechinphy/p/line-profiler.html](https://www.cnblogs.com/dechinphy/p/line-profiler.html)作者ID:DechinPhy更多原著文章請參考:[https://www.cnblogs.com/dechinphy/](https://www.cnblogs.com/dec
版权声明
本文为[itread01]所创,转载请带上原文链接,感谢
https://www.itread01.com/content/1611147908.html

  1. 利用Python爬虫获取招聘网站职位信息
  2. Using Python crawler to obtain job information of recruitment website
  3. Several highly rated Python libraries arrow, jsonpath, psutil and tenacity are recommended
  4. Python装饰器
  5. Python实现LDAP认证
  6. Python decorator
  7. Implementing LDAP authentication with Python
  8. Vscode configures Python development environment!
  9. In Python, how dare you say you can't log module? ️
  10. 我收藏的有关Python的电子书和资料
  11. python 中 lambda的一些tips
  12. python中字典的一些tips
  13. python 用生成器生成斐波那契数列
  14. python脚本转pyc踩了个坑。。。
  15. My collection of e-books and materials about Python
  16. Some tips of lambda in Python
  17. Some tips of dictionary in Python
  18. Using Python generator to generate Fibonacci sequence
  19. The conversion of Python script to PyC stepped on a pit...
  20. Python游戏开发,pygame模块,Python实现扫雷小游戏
  21. Python game development, pyGame module, python implementation of minesweeping games
  22. Python实用工具,email模块,Python实现邮件远程控制自己电脑
  23. Python utility, email module, python realizes mail remote control of its own computer
  24. 毫无头绪的自学Python,你可能连门槛都摸不到!【最佳学习路线】
  25. Python读取二进制文件代码方法解析
  26. Python字典的实现原理
  27. Without a clue, you may not even touch the threshold【 Best learning route]
  28. Parsing method of Python reading binary file code
  29. Implementation principle of Python dictionary
  30. You must know the function of pandas to parse JSON data - JSON_ normalize()
  31. Python实用案例,私人定制,Python自动化生成爱豆专属2021日历
  32. Python practical case, private customization, python automatic generation of Adu exclusive 2021 calendar
  33. 《Python实例》震惊了,用Python这么简单实现了聊天系统的脏话,广告检测
  34. "Python instance" was shocked and realized the dirty words and advertisement detection of the chat system in Python
  35. Convolutional neural network processing sequence for Python deep learning
  36. Python data structure and algorithm (1) -- enum type enum
  37. 超全大厂算法岗百问百答(推荐系统/机器学习/深度学习/C++/Spark/python)
  38. 【Python进阶】你真的明白NumPy中的ndarray吗?
  39. All questions and answers for algorithm posts of super large factories (recommended system / machine learning / deep learning / C + + / spark / Python)
  40. [advanced Python] do you really understand ndarray in numpy?
  41. 【Python进阶】Python进阶专栏栏主自述:不忘初心,砥砺前行
  42. [advanced Python] Python advanced column main readme: never forget the original intention and forge ahead
  43. python垃圾回收和缓存管理
  44. java调用Python程序
  45. java调用Python程序
  46. Python常用函数有哪些?Python基础入门课程
  47. Python garbage collection and cache management
  48. Java calling Python program
  49. Java calling Python program
  50. What functions are commonly used in Python? Introduction to Python Basics
  51. Python basic knowledge
  52. Anaconda5.2 安装 Python 库(MySQLdb)的方法
  53. Python实现对脑电数据情绪分析
  54. Anaconda 5.2 method of installing Python Library (mysqldb)
  55. Python implements emotion analysis of EEG data
  56. Master some advanced usage of Python in 30 seconds, which makes others envy it
  57. python爬取百度图片并对图片做一系列处理
  58. Python crawls Baidu pictures and does a series of processing on them
  59. python链接mysql数据库
  60. Python link MySQL database