What is a reptile

Web crawler （ Also known as web spider , Network robot ） Is to simulate the browser to send network requests , Receive request response , One according to certain rules , Programs that automatically capture Internet Information .
In principle, , As long as it's a browser ( client ) Something that can be done , Reptiles can do .

Crawling web page image implementation steps

First step ： Open the operated website （ Any website ）

http://github.com/

The second step ： adopt python Visit this website

headers = {
'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
response = requests.get('http://github.com/',headers=headers)
print(response.request.headers)

{
'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

The third step ： Click on F12 Search for relevant information

Find picture information

obtain headers：

Step four ： Crawling pictures , Download to local

headers = {
'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
response = requests.get('https://avatars.githubusercontent.com/nplasterer?s=64&v=4',headers=headers)
print(response.request.headers)
with open('icon.ico', 'wb') as f:
f.write(response.content)
print(" Crawling pictures succeeded ")

Step five ： Show tests

img = cv2.imread("icon.ico")
cv2.imshow('icon',img)
cv2.waitKey(0)

Core code

import requests
import cv2
headers = {
'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
response = requests.get('https://avatars.githubusercontent.com/nplasterer?s=64&v=4',headers=headers)
print(response.request.headers)
with open('icon.ico', 'wb') as f:
f.write(response.content)
print(" Crawling pictures succeeded ")
img = cv2.imread("icon.ico")
cv2.imshow('icon',img)
cv2.waitKey(0)