Python 3 crawler series (1) -- climbing blind date websites

Active smiling face 2021-11-25 10:18:44
python crawler series climbing blind

use python3 Do a web crawler , The website to climb this time is http://www.lovewzly.com/jiaoyou.html Dating website , I don't say much nonsense , Go straight to the code :
First import the corresponding module :

import requests
import os

Then write the corresponding fractional function :

# Set the age 
def set_age():
age = int(input(" Please enter the expected age of the other party :")) # Force string to integer 
if 21 <= age <=30:
startage = 21
endage = 30
elif 31 <= age <=40:
startage = 31
endage = 40
return startage,endage
# Set gender 
def set_sex():
sex = input(" Please enter the expected gender of the other party :")
if sex == ' male ':
gender = 1
elif sex == ' Woman ':
gender = 2
return gender
# Set city 
def set_city():
city = input(" Please enter the other party's expected city :")
if city == ' Beijing ':
cityid = 52
elif city == ' Guangzhou ':
cityid = 76
elif city == ' fuzhou ':
cityid = 53
elif city == ' Xiamen ':
cityid = 60
elif city == ' Hangzhou ':
cityid = 383
elif city == ' Qingdao ':
cityid = 284
elif city == ' Changsha ':
cityid = 197
elif city == ' jinan ':
cityid = 283
elif city == ' nanjing ':
cityid = 220
elif city == ' Hong Kong ':
cityid = 395
elif city == ' Shanghai ':
cityid = 321
elif city == ' Chengdu ':
cityid = 322
elif city == ' wuhan ':
cityid = 180
elif city == ' Suzhou ':
cityid = 221
elif city == ' Shenzhen ':
cityid = 77
return cityid
# Set the height 
def set_height():
height = int(input(" Please enter the other person's expected height :"))
if 0<= height <=150:
startheight = 0
endheight = 150
elif 151 <= height <=160:
startheight = 151
endheight = 160
elif 161 <= height <=170:
startheight = 161
endheight = 170
elif 171 <= height <=180:
startheight = 171
endheight = 180
return startheight,endheight
# Set academic qualifications 
def set_education1():
education1 = input(" Please enter the expected education of the other party :")
if education1 == ' Junior high school ':
education = 10
elif education1 == ' high school ':
education = 20
elif education1 == ' secondary specialized school ':
education = 25
elif education1 == ' junior college ':
education = 30
elif education1 == ' Undergraduate ':
education = 40
elif education1 == ' master ':
education = 50
elif education1 == ' Doctor ':
education = 60
return education
# Set monthly salary 
def set_yue():
yue = int(input(" Please enter the expected monthly salary of the other party :"))
if 2000<= yue <=4999:
salary = 2
elif 5000<= yue <=9999:
salary = 3
elif 10000<= yue <=19999:
salary = 4
elif 20000<= yue :
salary = 5
return salary
# Parse web pages 
def get_info(page,startage,endage,gender,cityid,startheight,endheight ,education ,salary ):
# http://www.lovewzly.com/api/user/pc/list/search?startage=21&endage=30&gender=2&marry=1&page=1
# String formatting %s
url = 'http://www.lovewzly.com/api/user/pc/list/search?startage={}&endage={}&gender={}&cityid={}&startheight={}&endheight={}&marry=1&education={}&salary={}&page={}'.format(startage,endage,gender,cityid,startheight,endheight,education,salary,page)
# response = requests.get(url).json() # Simple writing 
response = requests.get(url)
if response.status_code == 200:
result = response.json()
# 《Response[200]> Response successful Status code 
# print(response)
return result
# Download the file to a custom folder 
def save_images(item,i): # Define a images Folder 
if not os.path.exists('images'):
#os.makedirs(path) # He can create multi-level directories at once 
os.mkdir('images') # Is to create a directory level by level 
images_url = item['avatar'] # get images Information ,avatar yes Picture location 
# print(images_url)images_url
response = requests.get(images_url) # utilize requests Request picture address 
if response.status_code == 200: # Judge whether the address is valid 
print(" Getting :%s Information about " % item['username']) # Set the save address of the picture 
file_path = 'images//{}.jpg'.format(i) # Naming will go wrong , Yes * Names like that 
with open(file_path,'wb') as f: # b Binary system 
f.write(response.content) # Open the corresponding file 

Finally, write the main function :

# The main program 
def get_data():
print(" Please enter your filter criteria , Start this marriage :")
# Call the function written above 
startage,endage = set_age()
gender = set_sex()
cityid = set_city()
startheigth,endheigth = set_height()
education = set_education1()
salary = set_yue()
# To prevent system naming errors , Replace names with numbers 
i = 0
for i in range(1,10): # take 1~10 The content of 
# Get the captured json data 
json = get_info(i,startage,endage,gender,cityid,startheigth,endheigth,education,salary)
#print(json['data']['list'])
# Save the picture 
for item in json['data']['list']:
i += 1
print(item,i)
save_images(item,i)
get_data()

The code ends here , This is the basic version of the crawler, which is more suitable for python For beginners , Bloggers have a perfect version of this crawler , Readers in need can chat with bloggers privately or leave messages in the comment area , The blogger will reply and answer at the first time , If the number is large, the blogger will update the perfect version of the crawler to the blog , Those who like bloggers' blog posts can pay attention to bloggers , I am an active smiling face .

版权声明
本文为[Active smiling face]所创,转载请带上原文链接,感谢
https://pythonmana.com/2021/11/20211109005853881H.html

  1. Sorting out the learning route for Python beginners
  2. The 6-line Python code uses the pdf2docx module converter object to convert PDF into docx file
  3. Batch compression of picture files using Python
  4. Using Python to write djikstra algorithm for robot path planning
  5. python实现手机号获取短信验证码 | 对接打码平台
  6. Detailed explanation of Euler Rodriguez code in Python
  7. Prove that angular displacement is not a vector with Python
  8. Using Python program to deeply understand homogeneous transfer matrix t
  9. Triple product formula of vector and its proof (with Python code)
  10. Derivation of differential formula of DH matrix using Python
  11. Python openpyxl operation on Excel (get the total number of columns, get the value of a row, get the value of a column, and set the cell value)
  12. Realizing Excel data filtering and data matching with Python
  13. Python reads and writes files
  14. Four scenarios of processing Excel files with Python
  15. Python converts dictionary to excel
  16. Python implements file reading and writing
  17. Basic Python syntax -- functions
  18. Python learning thinking
  19. Python basic syntax -- Boolean operation, break, continue, pass
  20. Python basic syntax -- loop
  21. Basic Python syntax -- lists, dictionaries
  22. Python basic syntax -- conditional judgment, input ()
  23. Python first experience - efficient office, data analysis, crawler
  24. Modulenotfounderror: no module named 'Django summernote details
  25. Key points for Django to make personal blog website
  26. Path setting of templates in Django settings
  27. Leetcode 1611. Minimum one bit operations to make integers Zero (Python)
  28. Directory C: \ users \ a \ desktop \ Django_ The blog master is registered as the GIT root, but no git repository details are found there
  29. Django. Core. Exceptions. Improveconfigured: application labels aren't unique, duplicates: admin
  30. How to verify that Django has created the project details correctly
  31. How to create a database when using Django to construct a website
  32. The solution of using Django framework to create project in Windows system
  33. Running Python virtual environment on win10 system to execute ll_ Env \ scripts \ activate: unable to load file elaboration scheme
  34. Detailed explanation of constructing virtual environment with Django in Python 3
  35. Python implementation of affine cipher
  36. RC4 Python implementation
  37. Simple: Python_ Automatic body temperature clock
  38. 用python把两个csv中的日期的列提出年,做出新的一列,再把两个csv表格按照新做出的日期这列和ID号合并为一个表。
  39. python中类实例化后,其对象无法被其他模块调用方法
  40. [JSON] - Python creates JSON file format
  41. Utilisez Python pour proposer l'année de la colonne de date dans les deux CSV, faire une nouvelle colonne, puis combiner les deux tableaux CSV en un seul tableau avec la colonne de date et le numéro d'identification.
  42. 关于#python#的问题,请各位专家解答!
  43. ***
  44. ***
  45. 關於#python#的問題,請各比特專家解答!
  46. S'il vous plaît répondre aux questions de Python!
  47. About the import of Python class
  48. Magic Python property decorator: 1 line of code makes Python methods become properties in seconds
  49. Python 音频调整音量(附代码) | Python工具
  50. Python programming ideas [series of articles]
  51. Python crawler programming idea (67): modify nodes using pyquery
  52. Python crawler programming idea (66): using pyquery to obtain node information
  53. Python crawler programming idea (65): find nodes using pyquery
  54. Python crawler programming idea (64): using CSS selectors in pyquery
  55. Python crawler programming idea (63): basic knowledge of pyquery
  56. Python crawler programming idea (62): project practice: capturing cool dog online red song list
  57. Python crawler programming idea (61): project practice: capturing rental information
  58. Python crawler programming idea (60): get CSS selector code through browser
  59. Python crawler programming idea (58): nested selection nodes with beautiful soup CSS selectors
  60. Python crawler programming idea (56): find method of beautiful soup method selector