Python crawler -- regular expression

Hee hee hee hee hee 2020-11-13 01:54:15
python crawler regular expression

Regular expressions

One 、 Introduction to regular expressions

The role of regular expressions

Actually, there are four main steps for reptiles :

  1. Clear objectives ( Know where you're going to search... Or where you're going to search )
  2. climb ( All the content of the website will be crawled down )
  3. take ( Get rid of data that's not useful to us )
  4. Processing data ( Store and use in the way we want )

Regular expressions are often used for retrieval 、 Replace those that match a pattern ( The rules ) The text of .

Regular expression is a logical formula for string operation , It is to use some specific characters defined in advance 、 And the combination of these specific characters , Form a “ Rule string ”, This “ Rule string ” A filter logic used to express strings .

Purpose of using regular expressions

Whether the given string conforms to the filtering logic of regular expression (“ matching ”);
Through regular expressions , Get the specific part we want from the text string (“ Filter ”).
 Insert picture description here

Two 、 Regular expression matching rules

####### Common characters
 Insert picture description here

Predefined character set

 Insert picture description here


 Insert picture description here

Boundary matching

 Insert picture description here

Logical grouping

 Insert picture description here

3、 ... and 、re modular

Regular expressions use Escape special characters , So if we're going to use the original string , Just add one r Prefix ,

re Module general use steps
  1. Use compile() Function to compile a regular expression as a string Pattern object

  2. adopt Pattern Object provides a series of methods for matching text , Get a match , One Match object .

  3. Finally using Match Object provides properties and methods to get information , Do other operations as needed

compile function

compile Function to compile regular expressions , Generate a Pattern object , Its general usage is as follows :

import re
# Compile regular expressions into Pattern object
pattern = re.compile(r'\d+')
Pattern object

Regular expressions are compiled into Pattern object , You can use pattern A series of methods for text matching to find .

Pattern Some commonly used methods of object mainly include :
match Method : Find from start , One match
search Method : Search from anywhere , One match
findall Method : All match , Returns a list of
finditer Method : All match , Return iterator
split Method : Split string , Returns a list of
sub Method : Replace
match Method

match Method to find the header of a string ( You can also specify the starting position ), It's a match , As long as a match is found, it will return , Instead of finding all the matches . Its general usage is as follows :
 Insert picture description here string String to match
pos Starting position of string , The default value is 0
endpos The end of the string , The default value is len ( String length )

group([group1, …]) Method is used to get a string that matches one or more groups , When you want to get the whole matching substring , Can be used directly group() or group(0);

start([group]) Method is used to get the starting position of the substring in the whole string ( Index of the first character of the substring ), Parameter default The value is 0;

end([group]) Method is used to get the end position of the substring in the whole string ( Index of the last character of the substring +1), Parameters The default value is 0;

span([group]) Method returns (start(group), end(group)).

search Method

search Method is used to find any location of a string , It's also a match , As long as a match is found, it will return , Instead of looking for all Matching results , Its general usage is as follows : Insert picture description here
When the match is successful , Return to one Match object , If there is no match , Then return to None.

findall Methods and finditer Method

 Insert picture description here

split Method

split Method to split the string according to the matching substring and return the list , It is used in the following form : Insert picture description here
maxsplit Specify the maximum number of divisions , Do not specify to split all

sub Method

sub Method to replace . It is used in the following form :
 Insert picture description here
repl It can be a string or a function :
1). Is string , Use repl To replace every matching substring of a string , And return the replaced string ,
2). If repl Is the function , This method should take only one parameter (Match object ), And return a string to replace .

count Used to specify the maximum number of replacements , Replace all if not specified .

Greedy mode and non greedy mode :abbbc
  1. Greedy mode : On the premise that the whole expression matches successfully , As many matches as possible ( .* );
    Regular expressions using greedy quantifiers ab* , Matching results : abbb.
    • Decided to match as much as possible b, therefore a All the back b There were .
  2. Non greedy model : On the premise that the whole expression matches successfully , As few matches as possible ( ? );
    Regular expressions using non greedy quantifiers ab*? , Matching results : a. .*? .+?
    Even if there is * , however ? Decided to match as little as possible b, So there was no b.
  3. Python The quantifier is greedy by default .
本文为[Hee hee hee hee hee]所创,转载请带上原文链接,感谢

  1. 利用Python爬虫获取招聘网站职位信息
  2. Using Python crawler to obtain job information of recruitment website
  3. Several highly rated Python libraries arrow, jsonpath, psutil and tenacity are recommended
  4. Python装饰器
  5. Python实现LDAP认证
  6. Python decorator
  7. Implementing LDAP authentication with Python
  8. Vscode configures Python development environment!
  9. In Python, how dare you say you can't log module? ️
  10. 我收藏的有关Python的电子书和资料
  11. python 中 lambda的一些tips
  12. python中字典的一些tips
  13. python 用生成器生成斐波那契数列
  14. python脚本转pyc踩了个坑。。。
  15. My collection of e-books and materials about Python
  16. Some tips of lambda in Python
  17. Some tips of dictionary in Python
  18. Using Python generator to generate Fibonacci sequence
  19. The conversion of Python script to PyC stepped on a pit...
  20. Python游戏开发,pygame模块,Python实现扫雷小游戏
  21. Python game development, pyGame module, python implementation of minesweeping games
  22. Python实用工具,email模块,Python实现邮件远程控制自己电脑
  23. Python utility, email module, python realizes mail remote control of its own computer
  24. 毫无头绪的自学Python,你可能连门槛都摸不到!【最佳学习路线】
  25. Python读取二进制文件代码方法解析
  26. Python字典的实现原理
  27. Without a clue, you may not even touch the threshold【 Best learning route]
  28. Parsing method of Python reading binary file code
  29. Implementation principle of Python dictionary
  30. You must know the function of pandas to parse JSON data - JSON_ normalize()
  31. Python实用案例,私人定制,Python自动化生成爱豆专属2021日历
  32. Python practical case, private customization, python automatic generation of Adu exclusive 2021 calendar
  33. 《Python实例》震惊了,用Python这么简单实现了聊天系统的脏话,广告检测
  34. "Python instance" was shocked and realized the dirty words and advertisement detection of the chat system in Python
  35. Convolutional neural network processing sequence for Python deep learning
  36. Python data structure and algorithm (1) -- enum type enum
  37. 超全大厂算法岗百问百答(推荐系统/机器学习/深度学习/C++/Spark/python)
  38. 【Python进阶】你真的明白NumPy中的ndarray吗?
  39. All questions and answers for algorithm posts of super large factories (recommended system / machine learning / deep learning / C + + / spark / Python)
  40. [advanced Python] do you really understand ndarray in numpy?
  41. 【Python进阶】Python进阶专栏栏主自述:不忘初心,砥砺前行
  42. [advanced Python] Python advanced column main readme: never forget the original intention and forge ahead
  43. python垃圾回收和缓存管理
  44. java调用Python程序
  45. java调用Python程序
  46. Python常用函数有哪些?Python基础入门课程
  47. Python garbage collection and cache management
  48. Java calling Python program
  49. Java calling Python program
  50. What functions are commonly used in Python? Introduction to Python Basics
  51. Python basic knowledge
  52. Anaconda5.2 安装 Python 库(MySQLdb)的方法
  53. Python实现对脑电数据情绪分析
  54. Anaconda 5.2 method of installing Python Library (mysqldb)
  55. Python implements emotion analysis of EEG data
  56. Master some advanced usage of Python in 30 seconds, which makes others envy it
  57. python爬取百度图片并对图片做一系列处理
  58. Python crawls Baidu pictures and does a series of processing on them
  59. python链接mysql数据库
  60. Python link MySQL database