explain

The article was first published in HURUWO My blog station , This platform does synchronous backup release .

If browsing or access to abnormal picture loading failure or related questions, you can go to the original blog comments browsing .

Link to the original text PYTHON Reptile battle _ Garbage man idle fish crawler turn crawler data integration self use second-hand rapid response garbage collection platform _3( The source code is continuously updated ) Click directly to visit .

Links to the entire series

PYTHON Reptile battle : Garbage man, fish, reptile 、 Turn crawler data integration, self use second-hand rapid response garbage collection platform ( The source code is continuously updated )-1

PYTHON Reptile battle : Garbage man, fish, reptile 、 Turn crawler data integration, self use second-hand rapid response garbage collection platform ( The source code is continuously updated )-2

PYTHON Reptile battle : Garbage man, fish, reptile 、 Turn crawler data integration, self use second-hand rapid response garbage collection platform ( The source code is continuously updated )-3

Preface

Content of this chapter

This chapter is about fidder The problem of packet forwarding , That is, after intercepting the data, how to push the desired data to the specified database for processing .

FidderScript Description and use of

FidderScript What is it?

FidderScript yes Fidder Requests for packets reply Modified scripting language . By using FidderScript Scripting inserts can be automated Request to modify forward reply Modify the operating . It's very powerful , People who like data collection must learn .

FidderScript The grammar of

Official documents : http://fiddler2.com/r/?fiddlerscriptcookbook

You don't have to look at the official documents Sinister classview It already includes all the api file Just look at it

Here are some common class methods :

1.Session All requested information / The repository of the reply message Include request response You can make changes . Compare a core class method object .

2.HTTPRequestHeaders and HTTPResponseHeaders headers The mapping class of stay Session It can be extracted and modified .

3. About Fidder Some of the software itself FidderApplication Class method For example, modify UI Of Button, etc , You can even customize an automation tool . Let's not go into details , Just look at it .

I feel that the language is very similar to java, There are objects and classes . It doesn't cost much to learn . But it's really powerful .

FidderScript A simple example

Experience the complete development process of a script .

Enter the script editor

Two ways :

Method 1.

Click to enter FidderScript

Expand the class diagram : Click on ClassView

If you make changes, click :Save Script Save script

Application script

Method 2.

Click Rules > Customize Rules

Rules....Enter FiddlerScript code inside the appropriate function.

Save the file.

Recommend the second way to enter the script editor

Revise it request url

Jump to the specified code Blocks and method bodies Click on goto Select from the drop-down box and click

Get into OnBeforeRequest


static function OnBeforeRequest(oSession: Session) { ......
......
......
}

be-all request The information is contained in oSession Inside

For example, we make a Zhihu's redirection to Baidu

if(oSession.fullUrl.Equals("https://www.zhihu.com/"))
{
oSession.fullUrl = "https://www.baidu.com/";
}

Save and run :

The magic jump from Zhihu to Baidu

The whole process was very easy .

FidderScript Transfer of idle fish data

Data storage in data transfer

1. Local or cloud

2. Whether we need to build a back-end system to process data

3. How the data is displayed on the page

Solution

In the cloud, locally , But it's necessary to build a background system .

Provide a data submission interface to separate data provision and data processing .

Display and processing in the background system (web In the system ) Processed data . About the background system construction, the following article will introduce , Let's assume we've built it .

Background data receiving system

The data submission interface is assumed to be : http://127.0.0.1:8000/xianyu/

The data submission method is POST Data can be submitted directly to the original data , Try to put logic processing in the background

Start the service first :

Write forward scripts

Because idle fish's testing machine is not around Let's use tiktok data for example

Tiktok Grab a link to the list of goods in the window

https://api5-normal-c-lf.amemv.com/aweme/v1/promotion/user/promotion/list/?user_id=54512396382&sec_user_id=MS4wLjABAAAAa54-WneSWxYZ24J_00iS-9SwMQkrM3fOYK_sXDYG-lg&cursor=0&count=20&column_id=0&goods_type=0&shop_version=1&storage_type=0&manifest_version_code=110601&_rticket=1611199065509&app_type=normal&iid=17186708527928&channel=meizu&device_type=M1852&language=zh&cpu_support64=true&host_abi=armeabi-v7a&uuid=86908603665928&resolution=1080*2124&openudid=9c8e3e9910b351a8&update_version_code=11609900&cdid=a5182ee3-c5b7-44b9-830f-71ca10dd4b60&os_api=27&mac_address=D8%3A6C%3A02%3AC9%3AB5%3A58&dpi=480&ac=wifi&device_id=70532699303&mcc_mnc=46001&os_version=8.1.0&version_code=110600&app_name=aweme&version_name=11.6.0&device_brand=Meizu&ssmix=a&device_platform=android&aid=1128&ts=1611199065

Scripts forward packets Core code :

static function OnBeforeResponse(oSession: Session) {
if (m_Hide304s && oSession.responseCode == 304) {
oSession["ui-hide"] = "true";
} if (oSession.fullUrl.Contains("/promotion/user/promotion/list/"))
{
oSession.utilDecodeResponse();// There may be garbled code in the request to eliminate saving // Put the content through ajax http Send it to other places
var _xhr = new ActiveXObject('Microsoft.XMLHTTP');
var url = 'http://127.0.0.1:8000/xianyu'; // Send data parameters
var param = oSession.GetResponseBodyAsString() // You don't need a return value, so you write an empty callback
_xhr.onreadystatechange = function() {}
_xhr.open('POST', url, true);
_xhr.setRequestHeader("Content-Type", "application/json");
_xhr.setRequestHeader("X-Requested-With", "XMLHttpRequest");
_xhr.send(param); } }

Test data reception

Click on the phone Open the window

api The server successfully obtained the data

summary

Finally, let me explain

autoResponder Just replace the return package function There is no function to modify and save the data package , To begin with, I'm running away .

FidderScript It's a powerful tool Only master FidderScript Can be regarded as a real master of the whole Fidder This software .

Reference article Information note

Fiddler4.6.20 Use script Save request or response data to local or post Save it somewhere else

The source code of this project has been uploaded :https://github.com/HuRuWo/PickUpTrash

PYTHON Reptile battle _ Garbage man idle fish crawler turn crawler data integration self use second-hand rapid response garbage collection platform _3( The source code is continuously updated ) More articles about

  1. 13 Line code implementation :Python Real time video capture ( Source code attached )

    One . Preface This article is about < Face recognition complete project combat > Series blog article No 3 part : Program design (Python edition ), The first 1 section <Python Real time video capture program design >, This chapter systematically introduces : be based on Python+open ...

  2. Python Open source face recognition library : The offline recognition rate is as high as 99.38%( Source code attached )

    Python Open source face recognition library : The offline recognition rate is as high as 99.38%( Source code attached ) turn https://cloud.tencent.com/developer/article/1359073   11.11 Wisdom goes up ...

  3. be based on Python Interface automation test framework + Separation of data and code ( Advanced ) Source code attached

    introduction In the last article < be based on Python Interface automation test framework ( Elementary chapter ) Source code attached > Talked about the construction of interface automation testing framework , The core module function is to test database initialization , Let's look at the previous frame structure : It can be seen that testcase ...

  4. Java The actual battle of reptile series : Climb cool dog music website TOP500 Song ( Source code attached )

    In the two essays I shared earlier, I introduced HttpClient and Jsoup And a simple code case : Java Reptiles II : Use HttpClient Grab page HTML Java Reptile series three : Use Jsoup analysis HTML today ...

  5. python Reptiles - Taobao product password ( Graphic tutorial with source code )

    I'm free today , I don't want to be like that in the book , I believe that all the data are regular to look for , Then I analyzed the rules and encryption methods of Taobao's commodity data , Using the simplest knowledge to analyze the data needed . This also taught me , There is more than one way to solve the problem ...

  6. Python Developing Chinese chess practice ( Source code attached )

        Pygame Make Chinese chess , I always like playing chess , Yes python Just try to make one , Level co., LTD. , The level of computer operation is low , Need to update the way the computer works in the next version , I hope the source code can help you learn better python. Total points ...

  7. Java Douban movie crawler —— Little reptile growing up ( Source code attached )

    I've used reptiles before , For example, use nutch Crawl the seed , Search based on crawled data , I've also seen some source code . Of course ,nutch The consideration of reptiles is very comprehensive and meticulous . Every time I see the information of the web page and the processing information on the screen , ...

  8. python_way day10 python And the scope of other languages 、 python2.7 More inheritance and 3.5 The difference between multiple inheritance 、 socket and socketserver Source code ( Support concurrent processing socket, Multi process , Multithreading )

    python_way day10 1.python And the scope of other languages 2.python2.7 More inheritance and 3.5 The difference between multiple inheritance 3.socket and socketserver Source code ( Concurrent processing socket) ...

  9. use python Of TK Module to realize idiom guessing game ( Source code attached )

    explain : This game uses python Module has tkinter,random,hashlib: The whole game is divided into four windows , A window into the game . A close window . A game in progress window and a game end window . There are two main sources py ...

  10. Python: game : Mine clearance ( Source code attached )

    This time we're based on pygame To do a minesweeping , Last time a friend asked me the code python edition , Let me explain , All my code is based on python 3.6 Of . Look at the screenshot first , Modelled on the XP Mine sweeping on the road , Feeling XP It's like ...

Random recommendation

  1. JavaScript The essence of closure “ Lexical scope ”

    You should have written the following similar code , In fact, what I want to express here is that sometimes the place where a method is defined is thousands of miles away from the place where it is used , When that method is implemented , Which variables it can access , Which variables cannot be accessed , How to judge this ? This is the question we need to analyze this time ...

  2. use vs2013 establish windows service Program

    windows services It's a service program running in the background , It can be used vs2013 To create , The steps to create are as follows : 1. open vs2013 , Files -->New Project --> wi ...

  3. Java NIO frame Netty demo

    Netty What is it? Netty It's a java Open source framework .Netty Provide asynchronous . Event driven network application framework and tools , For rapid development of high performance . Highly reliable network server and client programs . in other words ,Netty It's based on NI ...

  4. ZendFramework The deployment environment

    View source code int_autoloader.php In file , Found an application of  AutoloaderFactory The namespace of , The path is written as a relative path , So you need to be in php.ini Define a inclde_pat ...

  5. winform Use code to place controls at the top and bottom

    Sometimes , We may add controls dynamically , And prepare to place it on the top or bottom of the pair . There are two ways to achieve this : One way is to WinForm Use... In the form Controls Control set SetChildIndex Method , This method sets the child control to the specified index ...

  6. poj2236 Wireless network

    The main idea of this question is : We need to use the Internet in disaster relief , These guys built a wireless network out of laptops , But come on , Communication with each other is limited by distance , A computer can only be separated from him for d Computer communication of , Then a computer can also be connected by several computers to communicate in this way . And then it's going to input each ...

  7. JDBC Fourth articles --【 Database connection pool 、DbUtils frame 、 Pagination 】

    1. Database connection pool What is a database connection pool Simply speaking : The database connection pool provides the connection . Why do we use database connection pooling The establishment and closing of database connection is very resource consuming Open frequently . Closing the connection causes poor system performance Write connection pools ...

  8. Java The naming convention of

    1. Class name . The interface name : The first letter of the first word should be capitalized . If there are more than one word , Other words should be capitalized . 2. Package name : All lowercase . 3. Variable name . Method name : The first letter of the first word is lowercase . If there are more than one word , Other words should be capitalized . 4. often ...

  9. Selenium+PhantomJS The reason and solution of using time error

    problem In use today selenium+PhantomJS Dynamic crawling of web pages , The following error message appears : UserWarning: Selenium support for PhantomJS has been dep ...

  10. openCV Video decomposition and synthesis

    1. Video decomposition import cv2 # ************************** # Break up the video cap=cv2.VideoCapture('1.mp4')# Get a video cap isOpe ...