Real time access to stock data, free—— Python crawler Sina stock actual combat

Quantity technology house 2021-10-29 10:40:11
real time access stock data

More highlights , Welcome to the official account : Quantity technology house , You can also add Technology Personal micro signal :sljsz01, Communicate with me .

The importance of real-time stock data

For the four tradable assets : Stocks 、 futures 、 option 、 For digital money , futures 、 option 、 Digital currency , Can be provided from the exchange api Receive real-time market data , The quantitative trading interface of stocks is not open to ordinary people , This leads us to want to get real-time data of stocks , Very difficult . And at the same time , Real time stock data , It is also extremely important floor trading data .

For manual traders , On the one hand, real-time data can assist in staring at the disk , On the other hand , You can use the program to simply develop the price reminder , To a certain price , Make entry and exit transactions .

For quantitative traders , Real time market is even more important . After we receive real-time quotes , It is not only necessary to calculate the strategy signal with real-time data , And when the strategy signal needs to place an order for a stock , We also need to know the latest price of the stock 、 Pankou data , Thus according to the price + Handicap , Choose the right price to order . Besides , After real-time data landing , It can also serve our policy backtesting .

The most popular explanation of reptiles is

Reptiles , It is equivalent to imitating the action of web page query , For example, we enter in the browser, The browser returns to Baidu's home page , In fact, this is a request + The process of returning . We're asking for an address , What is returned is data ( Although what we see is Baidu home page , In fact, there are some columns of data behind it , But the subsequent visual display became a web page ).

For requesting stock data , It's the same thing . For example, we request the data of a stock ( With 600000 Take Shanghai Pudong Development Bank as an example ), Enter... In the browser address :, The stock code can be modified at will , The browser returns the following data set and displays :

  Of course, query multiple stocks at the same time , It can also be done , We also enter the address in the browser , for example :,sz000001,sz000002, At the same time, I inquired about Shanghai Pudong Development Bank 、 Ping An Bank 、 vanke A The data of .


that , about Python Come on , How do we use Python To imitate the action of Web query ? There must be a Python Three party Library :request library .

requests It's using Apache2 licensed Permit HTTP library .Request Support HTTP Connection hold and connection pool , Support use cookie Keep talking , Support file upload , Support automatic response content encoding , Support international URL and POST Automatic data coding . so to speak ,requests stay python The built-in module is highly encapsulated , Thus making python When making a network request , Become human , Use Requests It's easy to do whatever the browser can do . meanwhile ,requests Persistent connections are automatically made keep-alive.

requests The above advantages of the library , Plus the convenience of its use , Make it Python The preferred tool for crawlers . We go through requests, Let's repeat the query process of the above web page , The steps are also very simple , First, query a single stock , Return as follows

  Query multiple stocks again , Return as follows , You can see , use requests The result returned by the query , It's exactly the same as what we see on our website , This also shows that requests It perfectly simulates the request action of the browser .


Python Reptiles Sina Stock actual combat

First step ,Sina API + Requests Library calls core functions

By calling Sina Stock API, Query the stock price in real time . We use multi stock query , Use requests Request interface .

Core function logic :code Pass in the stock code , call requests Query the library , And parse the query results , Get the latest price of the stock we want to query 、 The day's rise and fall 、 Yesterday's closing price and other key fields

The second step , adopt threading Multi thread simultaneous query results 、 adopt Queue Implement thread pool

Let's briefly introduce threading and Queue.threading The module contains rich functions about thread operation , Include : Common thread functions , Thread object , Lock object , Recursive lock object , Event object , Conditional variable object , Semaphore object , Timer Objects , Fence object .threading.Thread: Thread object , Important method ,start(): Start thread activity . It will make run() Method is called in a separate control thread , It should be noted that the same thread object start() Method can only be called once , If called more than once , Will report RuntimeError error .run(): This method represents thread activity .

Python Of Queue Synchronous... Is provided in the module 、 Thread safe queue class , Include FIFO( First in, first out ) queue Queue,LIFO( After the first out ) queue LifoQueue, And priority queues PriorityQueue. These queues all implement lock primitives , Can be used directly in multithreading . You can use queues to synchronize threads .Queue.put(item) Written to the queue ;Queue.get([block[, timeout]]) Get the queue

The third step , Modular implementation :Worker class & Stock class

Worker class , Main functions :

  1. Incoming thread instance object , call init、start Method , heavy load run function

  2. work_queue Store the queue to query , Take out one by one (FIFO), Save the query results to result_queue queue , When the queue is full , Retrieve all query results in the result queue , And print



Stock class , Main functions :

  1. Initialize the producer 、 Consumer queue , The producer queue is empty , The maximum number of consumer queues is the number of stocks queried

  2. According to the preset thread data , Initializes the thread pool , binding Worker class

  3. For every stock you query , Join the producer queue

  4. Define the function for the crawler to get data


Last , We call Worker Classes and Stock class , You can get the results you want .

Operation result display

The program can not only obtain individual stock data , You can also get index data , We subscribe to the Shanghai index by default 、 Shenzhen Composite Index 、 vanke A、 Pudong Development Bank ,4 Data streams , The following is a display of the default run results , We printed the latest price 、 applies 、 Yesterday's closing price and other key data . We follow certain time intervals , Get data and print .

  Program extension : At the main program entry , You can modify the number of stocks queried , We support simultaneous query of multiple stocks ; Query interval 、 Modification of the number of threads queried .

  Of course , We can also further expand the crawler program , For example, calculate the quantization strategy signal , The landing data is used as the back test data , wait .

If you want to get this Python Reptiles Sina Stock Program , Welcome to scan code to add quantity technology to wechat , Get full code .


Previous dry goods sharing recommended reading

Digital currency stabilizes currency to grid market making strategy

Digital currency capital cost strategy

Share a yearly 15% The above risk-free arbitrage opportunities

Grid trading system development

Analyze and forecast the stock price through deep learning stock price cross section data

Omega System Trading and Development Club Internal sharing strategy Easylanguage Source code

A complete machine learning solution for real data sets ( Next )

A complete machine learning solution for real data sets ( On )

How to use trading pioneers (TB) Developing a digital currency strategy

Stock index futures high frequency data machine learning prediction

How to use TradingView(TV) Back testing digital currency trading strategy

How to invest in stock funds ? When to buy ? Can I help you? ?

【 Quantity technology house | Quantitative investment strategy series share 】 Stock index futures trading strategy based on moving average index

AMA Index original author Perry Kaufman 100+ Set of transaction strategy source sharing

【 Quantity technology house | Options series share 】 Options strategy “ The lonely nine sword ”

【 Quantity technology house | Financial data series sharing 】 The spread sequence calculation of arbitrage strategy , I'm afraid it's not as simple as you think

【 Quantity technology house | Quantitative investment strategy series share 】 Follow the mature trading strategy of futures

How to get free historical data of digital currency

【 Quantity technology house | Quantitative investment strategy series share 】 Multi period resonant trading strategy

【 Quantity technology house | Financial data analysis series sharing 】 Why Zhongzheng 500(IC) Is the most suitable index for long-term long

Commodity spot data is not easy to get ? The seasonality of goods is hard to track ? One click solution to the trouble free Python Crawler sharing

【 Quantity technology house | Financial data analysis series sharing 】 How to copy bottom commodity futures correctly 、 Commodities

【 Quantity technology house | Quantitative investment strategy series share 】 Stock index futures IF Minute volatility statistics strategy

【 Quantity technology house | Python Crawler series share 】 Real time monitoring of major stock market announcements Python Reptiles


本文为[Quantity technology house]所创,转载请带上原文链接,感谢

  1. J'ai utilisé Python pour ramper 1000 lettres d'amour pour aider mon colocataire à exprimer les fleurs de classe, mais les inverser et les inverser... C'est le secret ultime des fleurs de classe!
  2. python中列表转为矩阵后无法进行矩阵的乘法运算
  3. Python crawler Development and Learning full tutoriel 2nd Edition, banggan 100000 words [recommended Collection]
  4. Python crawler haut de gamme: microstore confus anti - décryptage
  5. La multiplication de la matrice ne peut pas être effectuée lorsque la liste est convertie en matrice en python
  6. Introduction to operators in python (Part 1)
  7. Are the dictionaries in Python ordered
  8. Introduction to dictionaries in Python
  9. List introduction in Python
  10. pandas比较两个dataframe特定数据列的数值是否相同并给出差值:使用np.where函数
  11. Python使用matplotlib绘制透明背景的可视化图像并保存透明背景的可视化结果(transparent background)
  12. Python self study notes -- basic grammar
  13. Python utilise matplotlib pour dessiner une image visuelle de l'arrière - plan transparent et enregistrer les résultats visuels de l'arrière - plan transparent
  14. Pandas compare les valeurs de deux colonnes de données spécifiques à dataframe et donne des valeurs de voyage: en utilisant la fonction np.where
  15. Comment configurer une application ASGI Django avec Postgres, nginx et uvicorn sur Ubuntu 20.04
  16. What are the advantages of Python and how to get started quickly
  17. Python self study notes -- basic data types
  18. Python code reading (Chapter 14): List Union
  19. Analyse statistique de la fonction de données des essais aléatoires Python
  20. Alien invasion project in Python application -- Aliens (Part 2)
  21. Python code reading (Chapter 14): List Union
  22. Lecture du Code Python (article 25): diviser les chaînes multilignes en listes
  23. Python self study notes -- operators
  24. Formation python - différences entre http et HTTPS
  25. Implementation of automatic timing comment function on Python CSDN platform
  26. python+tkinter+treeview子控件快捷键
  27. Raccourcis clavier pour les sous - contrôles Python + tkinter + treeview
  28. Analyse des données Python
  29. python+tkinter+treeview子控件快捷鍵
  30. Devine si je peux attraper Maotai avec la programmation python? Tout est ouvert à github
  31. À propos de pygame.display.set in Python Un petit problème avec mode ()
  32. Implementation of automatic timing comment function on Python CSDN platform
  33. python:dataframe进行iteritem遍历时如何将输出结果按照列分别输出为该列最后一行
  34. python:dataframe進行iteritem遍曆時如何將輸出結果按照列分別輸出為該列最後一行
  35. Python: comment le dataframe affiche les résultats de sortie par colonne à la dernière ligne de la colonne lors de la traversée de l'itemitem
  36. Écrivez un gadget de bureau pour votre fille préférée en python et elle dit que c'est génial!
  37. Introduction to closures in Python 3
  38. Global / nonlocal usage in Python 3
  39. Introduction to context manager in Python 3
  40. Python crawler selenium framework. You can start with these five questions | Python skill tree
  41. Common standard library random, python introductory tutorial 5 or 6 questions a day | Python skill tree
  42. It is said that Python is omnipotent. It's really good to see Liyang photography circle with Python this time
  43. 【Python 爬虫】 4、爬虫基本原理
  44. 【Python 爬蟲】 4、爬蟲基本原理
  45. 【 Python crawler】 4. Principes de base du crawler
  46. 这道python题到底应该要怎么做
  47. Que doit faire exactement ce problème Python
  48. Après l'importation des variables du module Python, les valeurs imprimées sont fixes.
  49. Nouveau singe Muzi Lee: 0 cours de formation Python de base types de hachage pour les opérations Python redis
  50. Looking at problems from a fresh perspective: analyzing selenium principle from the perspective of Python
  51. Insérez le format de date dans la base de données MySQL en python et ne l'exécutez pas.
  52. Try Python 3.10 with CONDA
  53. Répondez en python et demandez à quelqu'un de vous aider.
  54. Un simple problème de travail Python, qui ne fonctionne pas
  55. Problèmes d'écriture Python pour la boucle
  56. Comment Python exécute les commandes du programme à plusieurs reprises au lieu de quitter
  57. YYDS! Dexplot: one line of Python code to easily draw statistical charts!
  58. pandas生成的透视表如何和源数据一起保存
  59. pandas生成的透視錶如何和源數據一起保存
  60. Comment sauvegarder le tableau pivot généré par pandas avec les données sources