Full stack engineer development manual （ author ： Luan Peng ）
python The whole course
python Data mining series tutorial ——PySpider Framework application solution .
pyspider It's easier to get started , It's easier to operate , Because it adds WEB Interface , Write reptiles quickly , Integrated phantomjs, It can be used to grab js Rendered page . Support multi thread crawling 、JS Dynamic analysis , Provides an operational interface 、 Error retry 、 Timing crawling and so on
PySpider yes binux Do an open source implementation of crawler Architecture . The main functional requirements are ：
Grab 、 Update and schedule specific pages for multiple sites
We need to extract structured information from the page
Flexible and scalable , Stable and controllable
And this is the vast majority python The need of reptiles —— Directional grabbing , Structured parsing . But in the face of different structures of various websites , A single crawl mode is not necessarily satisfied , Flexible grasp control is necessary . In order to achieve this goal , Simple configuration files are often not flexible enough , therefore , Controlling grab through scripts is the last option .
And to reschedule , queue , Grab , exception handling , Monitoring and other functions as a framework , Provided to grab script , And ensure flexibility . Then add web Editing and debugging environment , as well as web Mission monitoring , That's the framework .
pyspider The basis of design is ： With python Script driven grab loop model crawler