picture

Last time I introduced  mitmproxy  This bag grabbing tool , And demonstrates how to use this tool to change your computer's request to access the Internet and return information , Do you think it's a little interesting . Today, I'd like to introduce how to use this tool to monitor mobile Internet access , And by grabbing bags , Download the data we want .

start-up mitmproxy

First of all, by executing the order  mitmweb  start-up mitmproxy, Let it handle the listening state , The service will listen on this machine 8080 port , After startup, it is as follows :

$ mitmwebWeb server listening at http://127.0.0.1:8081/Proxy server listening at http://*:8080

Mobile network configuration

  1. Make sure the mobile phone and computer are in the same LAN , And check your computer's LAN IP What's the address for standby , You can use the command to view , Such as Windows For system  ipconfig  command ,Mac or Linux Use the command  ifconfig, See the following figure , Find this machine in the local area network IP Address .

 picture

Different operating systems , Maybe the results are different , Here we can see in the red box my machine's IP The address is 192.168.0.108.

  1. Configure mobile proxy address , For example, the following uses  iphone Mobile phone configuration , Open Settings -> WLAN -> Click on the network that you are now connected to -> At the bottom HTTP   Agent configuration , After selecting manual , Configure the proxy address for our computer IP Address and mitmproxy Listening port of , The configuration is shown in the figure below :

 picture

  1. Now we can let the mobile phone access the Internet through the computer , But at the moment, mobile phones  App It's all through HTTPS Encryption request , So we need to install a mitmproxy Of HTTPS   certificate . Next, let's open the mobile browser , Enter url :mitm.it, Open the page as shown in the figure below , Select the operating system of the corresponding mobile phone , Download the certificate and install .

 picture

Download page

Here I'm going to order Apple Download the certificate with the icon , Download and wait for installation :

 picture

To be installed

 picture

Point installation

Click Install above , There may be hints of trust , Just choose a trust certificate , Here is the installed interface, as shown in the figure below :

 picture

Installation successful

Okay , After the configuration is complete, let's turn on a mobile phone APP See the computer mitmweb On the page , The content of the request has appeared .

 picture

Installation successful

Write download scripts

Okay , Now after analyzing the intercepted requests , We started using Python Write code , By analyzing the video url Download the video locally . Let's print the data first , New file  xiaoshipin.py, Then write the following code , The implementation will encode json The string is decoded to python object :

# xiaoshipin.py
import json
def response(flow):    url='https://api.amemv.com/aweme/v1/aweme/post/'    # Sift out the above url For the beginning of the url    if flow.request.url.startswith(url):        text=flow.response.text        # To encode json The string is decoded to python object        data=json.loads(text)        print(data)

Save after editing , And then execute the command :mitmweb -s xiaoshipin.py, Bring this script with you to run our proxy service , Refresh a few videos , You'll see what's shown below :

 picture

It contains the download address information we need , Next we just need to extract the video url , And cache the video locally OK 了 .

Now let's modify the above  xiaoshipin.py  Code for , Perfect the video download function .

#xiaoshipin.py
# Introduction package import json,osimport requests
# Only intercept and process return requests def response(flow):    # Requested url    url='https://api.amemv.com/aweme/v1/aweme/post/'    # Sift out the above url For the beginning of the url    if flow.request.url.startswith(url):        text=flow.response.text        # To encode json The string is decoded to python object        data=json.loads(text)        # I just analyzed and saw all the information of each video        # All in aweme_list in        video_url=data['aweme_list']        # Set download path        path='/Users/xx/shipin'        # If the folder does not exist , The new        if not os.path.exists(path):            os.mkdir(path)
       # Loop through all the videos url        for each in video_url:            # Video description            desc=each['desc']            url=each['video']['play_addr']['url_list'][0]            # Set the video name            filename=path+'/'+desc+'.mp4'            # use request Request video stream            req=requests.get(url=url,verify=False)            # Save video file            with open(filename,'ab') as f:                f.write(req.content)                f.flush()                print(filename,' The download ')

I have annotated the relevant code above , Now we save the edited code , Then execute the command again :mitmweb -s xiaoshipin.py, Start monitoring service . Then turn on the phone and download the video again , You'll see if all the videos have been saved locally .

 picture

summary

This article introduces how to pass  mitmproxy Tools to download small video browsing on the phone , Of course, in addition to video, he can also download music, pictures and so on , As long as you can analyze the content you want in which request through the interception tool , Then search and analyze the content returned by the request , And then use today's tool , And then through the simple Python It's OK to code and process automatically .

Reference resources

Official website :https://mitmproxy.org

Sample code :https://github.com/JustDoPython/python-100-day/