Python - thinking about class (self / CLS) and multi process communication

DoHerasYang 2021-04-06 11:57:54
python thinking class self cls


Python- Some thoughts on class and class instance in multi process



1. background

Recently completed a small tool , Complete the log identification 、 Related functions of comparison , Although many functions of this gadget need to be improved , But it doesn't affect my determination to recommend it here : CessTop - CessTop ---- A Smart Tool written in Python to Parse and Compare the Cisco Firewall Config File with TopSec Firewall Config File

In the process , Because I need to refactor my code , I need to play different roles for three different processes , The first process deals with Cisco The content of the configuration document , The second process deals with TopSec The content of the configuration document , The third process waits for the first two processes to finish processing the related data structure , And then compare , That is to say, the third one is equivalent to a service monitoring function in the early stage of operation .

In the process , An independent class is designed for each process to define different data variables , Therefore, you need to create a process for the instance object of each class ;

These are a background for writing this blog ....


A little bit of thinking - spark ( Or the motivation to write this blog ?!):

  • stay Pycharm IDE If you don't define @staticmethod You will always be prompted to convert your newly defined function to Global Function definition of , I don't understand why this is happening , But I think it's necessary to understand the definition rules of functions in a class ;

  • In process creation , We all know Python The multi process implementation of is based on multiprocessing Of Package To achieve , As for how to achieve multi process , stay Windows and class Unix Our system is different , Here I only study class Unix The implementation of the , That is to call fork The problem with functions ;

    1. For thread pools (pool) Call to apply as well as apply_async The problem with functions ;

    2. How to realize the communication between multiple processes to ensure the parameter transfer between processes ? Use Pipe still Queue?


2. Python Functions in class - staticmethod / classmethod

I'm sure many friends are familiar with this concept , Let's talk about it briefly and give some examples :


staticmethod

@staticmethod Defines static functions in a class , This static function has several properties :

  • Can not be called by an instance of a class , That is, it can be called directly from the class , That is, you don't need to declare an instance :

    class A(object):
    @staticmethod
    def demo_method(info:str):
    print(info)
    A.demo_method("This is staticmethod") # This is staticmethod
    
  • Static methods are equivalent to being separated from classes , But it can also be done through self To call private variables in a class , But the premise is to create an instance of the class , Because this function cannot be bound to a class instance , So it needs to be self Pass on the reference , namely self Is a new class instance variable :

    class A(object):
    __private_var = "This is Private Variable"
    @staticmethod
    def demo_method(self):
    print(self.__private_var)
    A_instance = A()
    # Here we need to self Make a parameter , The parameter is the newly created temp
    A.demo_method(A_instance)
    
  • Static methods can be inherited by subclasses , The way of inheritance follows the way of class inheritance :

    class A():
    @staticmethod
    def A_method(info):
    print(info)
    # B Class inheritance A Class A_method
    class B(A):
    @staticmethod
    def B_method(self, info):
    super().A_method(info)
    # Here's to create a new B Example
    B_instance = B()
    B_instance.B_method(B_instance, "This is B invokes A staticmethod")
    # Here you can print out This is B invokes A staticmethod
    # namely B Called A Of A_method
    

    We all know that although @staticmethod Defines a function out of class , Therefore, inherited class instances cannot access private variables in the inherited class , Unless you declare a class instance for the inherited class ;

    The static above can also be written in the following form :

    class A():
    @staticmethod
    def A_method(info):
    print(info)
    # B Class inheritance A Class A_method
    class B(A):
    @classmethod
    def B_method(cls, info):
    super().A_method(info)
    B().B_method("This is B invokes A staticmethod")
    # Here you can print out This is B invokes A staticmethod
    # namely B Called A Of A_method
    

    The specific explanation is given by classmethod For further explanation ;


classmethod

classmethod Defines... In a class Class method , from @classmethod Decorators are defined , Its characteristics are as follows :

  • because With ornaments @staticmethod The modified class method can directly pass the class through cls binding , So the call doesn't need to declare an instance of a class :

    class A():
    @classmethod
    def A_method(cls):
    print("This is A classmethod method")
    A.A_method()
    # Print out : This is A classmethod method
    

    Of course , It doesn't affect you to create a new class instance , And then call the function. :

    class A():
    @classmethod
    def A_method(cls):
    print("This is A classmethod method")
    A_instance = A()
    A_instance.A_method()
    # Print out : This is A classmethod method
    
  • For a declared Class method The function in the class wants to call the variable defined in the class , And private variables , Is that OK ? The answer is yes !

    class A():
    class_var = "This is Class Variable\n"
    __private_var = "This is Class Private Variable\n"
    @classmethod
    def A_method(cls):
    print(cls.class_var)
    print(cls.__private_var)
    A.A_method()
    # Print out :
    # This is Class Variable
    # This is Class Private Variable
    

    But here's a problem , In the absence of instances , That is to say There is no one created in the stack Class instance , If you change the variables of a class , Will the variables of this class be modified ? - - - It's like it's going to change ......

    class A():
    num = 1
    @classmethod
    def A_method(cls):
    print(cls.num)
    @classmethod
    def change_method(cls):
    cls.num = 2
    A.change_method()
    A.A_method()
    # At first, I thought it could not be modified , But the results surprised me , It was changed .... Analyze why it has been modified or what impact it has ...
    # The output is : 2
    # however , At present, I don't think the definition of this class has been changed , So try Define a new Class
    A_instance = A()
    print(A_instance.num)
    # The output is : 2
    # ok , It was modified ....
    # Analyze the process
    

    And then the analysis above continues , We need to know Python The definition of a class , namely Declare this class What is the position of being ?

    class A():
    num = 1
    @classmethod
    def A_method(cls):
    print(cls.num)
    @classmethod
    def change_method(cls):
    cls.num = 2
    print(cls) # 140683689759152
    A.change_method() # 140683689759152
    A.A_method()
    # A print python The location of the function in memory
    print(id(A)) # 140683689759152
    

    That is, the class called above is the definition stored in the same address ;

    therefore , Because it refers to a class variable with the same address , So there are situations where you might change the class definition variables ;

  • Now? , I have understood that in Pyton Constants defined by the class may change , So the inherited subclass calls super What's your address ? namely :super What is called is defined in the class of global variables ? Or the address of the class instance ?

    • If you call the subclass directly (B、C) Instead of creating an instance of a subclass , Then the calling parent class is not a directly defined parent class , That is, the definition in the original parent class will not be changed ! As you can see from the code below , Created in a global variable A、B The addresses of the two classes are different ; stay B Print superclass in ( Parent class ) The address of is not the same as the address of the global variable , Then there is no case of changing the properties defined by the parent class ;

      class A():
      def A_method():
      print("This is A method!")
      class B(A):
      @classmethod
      def B_method(cls):
      print(id(super()))
      class C(A):
      @classmethod
      def C_method(cls):
      print(id(super()))
      print(id(A)) # 140512863619088
      print(id(B)) # 140512863620032
      B.B_method() # 140511333031744
      C.C_method() # 140511869048192
      
    • Verify the definition given above :

      class A():
      num = 1
      def A_method():
      print("This is A method!")
      @classmethod
      def A_ChangeMethod(cls):
      cls.num = 2
      @classmethod
      def A_PrintNum(cls):
      print(cls.num)
      class B(A):
      @classmethod
      def B_method(cls):
      # print(id(super()))
      super().A_ChangeMethod()
      super().A_PrintNum()
      class C(A):
      @classmethod
      def C_method(cls):
      print(super().num)
      # print(id(B))
      B.B_method() # 2
      # print(id(A))
      C.C_method() # 1
      
  • Generate instances of classes , To verify again , That is, it will not be modified !

    class A():
    num = 1
    def A_method():
    print("This is A method!")
    @classmethod
    def A_ChangeMethod(cls):
    cls.num = 2
    @classmethod
    def A_PrintNum(cls):
    print(cls.num)
    class B(A):
    @classmethod
    def B_method(cls):
    super().A_ChangeMethod()
    super().A_PrintNum()
    class C(A):
    @classmethod
    def C_method(cls):
    print(super().num)
    B_instance = B()
    B.B_method() # 2
    C_instance = C()
    C.C_method() # 1
    
  • Defined class instance cls What's the address of ?

    class A():
    def A_method():
    print("This is A method!")
    class B():
    @classmethod
    def B_Method(cls):
    print(id(cls))
    print(id(B)) # 140512865761952
    B_instance = B
    B_instance.B_Method() # 140512865761952
    B_instance_2 = B
    B_instance.B_Method() # 140512865761952
    

staticmethod as well as classmethod Comparison

  • cls as well as self The difference between :

    • There is one of both C++ The feeling of the pointer in the middle ;

    • From the example above, we can see that ,cls Is used to refer to a variable in the current class defined by a function that points to a storage address :

      • If there is no instance of the declared class , that cls When called directly, it points to ( quote ) Define the address of the class for the first time , The address of the global variable class , That is, if you call cls Modify the properties of a class , It will be modified , We should pay great attention to this !;
      • Created an instance of the class , that cls It is also pointed to the address of the first defined class , So do it cls To call properties perhaps Be very careful when modifying the properties of a class , There may be unexpected changes , therefore cls You can add class attributes ;
    • self Used to refer to The current class instance variable , There's nothing to explore ;


A little thought

  • When calling a class reference directly , yes : Define the global variable class call , So if you modify a property, it will result in a modification ;
  • In consideration of inheritance , Every inheritance , The compiler creates ( Deep copy ) A temporary parent class to provide inherited properties and methods , In this case, we don't consider whether to create class instances , That is, whether an instance is created or not, the compiler will deeply copy a parent class , therefore super Does not change the definition of the defined global variable class ,super I think it's very safe ;
  • stay Python Class inheritance of , Subclasses will be deep copies of A parent class , To implement the call Properties and functions of the parent class
    • The advantage of this is : It's very safe for a definition , That is, there will be no unexpected mistakes ;
    • shortcoming : Occupancy resources ;


3. Python Communication between processes in - multiprocessing/Queue

In the most recent build of gadgets , Communication in the process is used in the middle , Please refer to my code for the specific implementation process ;

Here's a little bit of a problem , namely multiprocessing.Pool Communication problems between different processes in the , It is hereby stated that ;

We all know that Python There are related concepts of process pool , The call is very simple , That is to use Pool.add as well as Pool.apply_async perhaps Pool.apply To start the process ;

In three processes , need The first two processes deal with files , The third process is to analyze the processed data , So I want to design a third process for a service , Wait for the first two processes to process the relevant data and return the results , There are many ways to do it , I chose Queue To handle communication between processes , The demo code below illustrates this process :

from multiprocessing import Process, Pool, Queue
if __name__=='__main__':
queue_1 = Queue(maxsize=3)
pool = Pool(3)
with pool as pro:
result = pro.apply_async(f, args=(queue_1,),)
pool.close()
pool.join()

That is, I need to Queue Into the startup function , Complete the communication of parameters in the process , At this time, we encountered the error report :

RuntimeError: Queue objects should only be shared between processes through inheritance

Analyze this mistake

  • First , Inquired about the related API namely ,apply_async Return to one AsyncResult Parameters of type , This parameter returns the state of the process , Because of the call apply_async after ,queues_1 Direct passing into is not supported apply_async The function of ;

  • But in Process The definition in can be passed directly into , That is, the following is supported :

    from multiprocessing import Process, Pool, Queue
    if __name__=='__main__':
    queue_1 = Queue(maxsize=3)
    process_1 = Process(target=f, args=(queue_1,))
    process_2 = Process(target=f, args=(queue_1,))
    process_3 = Process(target=f, args=(queue_1,))
    

resolvent

  • call multiprocess.Manager To create a system that allows multiple processes to communicate multiprocess.Manager.Queue , Then be Pool Object call ;
  • take Pool Change the object to Process object ;

Wrote last

  • In a multiprocess call , If you write a startup process function yourself and don't rewrite it Process.Run function , So you need to define a startup function , If the function in the class is defined as staticmethod And defined self, Then you need to define an instance of a class , And then through Process The ginseng :

    The startup function defined in the class :

    @staticmethod
    # def Start_Processing(self):
    def Start_Processing(self, queue: multiprocessing.Queue):
    try:
    self.access_list = self.Process_Cisco_LogFile_ToList(filename=self.filename)
    self.LogFileList_toPandasDF(self, Logfile_List=self.access_list)
    except Exception as err:
    raise err
    finally:
    queue.put(self.df_cisco)
    self.df_cisco.to_csv(config.default_config_dict["default"].cisco_csv_Name, sep=',',
    header=config.default_config_dict["default"].df_format,
    index=True)
    

    Call the startup function :

    cisco_instance = cisco_function.Cisco_Function(filename_dict["cisco_filename"])
    cisco_process = Process(target=cisco_instance.Start_Processing, args=(cisco_instance, queue_cisco,))
    

    You can see , It has to be for Start_processing Functional self Assign a class instance , To start the function properly ;

版权声明
本文为[DoHerasYang]所创,转载请带上原文链接,感谢
https://pythonmana.com/2021/04/20210406115549585F.html

  1. 利用Python爬虫获取招聘网站职位信息
  2. Using Python crawler to obtain job information of recruitment website
  3. Several highly rated Python libraries arrow, jsonpath, psutil and tenacity are recommended
  4. Python装饰器
  5. Python实现LDAP认证
  6. Python decorator
  7. Implementing LDAP authentication with Python
  8. Vscode configures Python development environment!
  9. In Python, how dare you say you can't log module? ️
  10. 我收藏的有关Python的电子书和资料
  11. python 中 lambda的一些tips
  12. python中字典的一些tips
  13. python 用生成器生成斐波那契数列
  14. python脚本转pyc踩了个坑。。。
  15. My collection of e-books and materials about Python
  16. Some tips of lambda in Python
  17. Some tips of dictionary in Python
  18. Using Python generator to generate Fibonacci sequence
  19. The conversion of Python script to PyC stepped on a pit...
  20. Python游戏开发,pygame模块,Python实现扫雷小游戏
  21. Python game development, pyGame module, python implementation of minesweeping games
  22. Python实用工具,email模块,Python实现邮件远程控制自己电脑
  23. Python utility, email module, python realizes mail remote control of its own computer
  24. 毫无头绪的自学Python,你可能连门槛都摸不到!【最佳学习路线】
  25. Python读取二进制文件代码方法解析
  26. Python字典的实现原理
  27. Without a clue, you may not even touch the threshold【 Best learning route]
  28. Parsing method of Python reading binary file code
  29. Implementation principle of Python dictionary
  30. You must know the function of pandas to parse JSON data - JSON_ normalize()
  31. Python实用案例,私人定制,Python自动化生成爱豆专属2021日历
  32. Python practical case, private customization, python automatic generation of Adu exclusive 2021 calendar
  33. 《Python实例》震惊了,用Python这么简单实现了聊天系统的脏话,广告检测
  34. "Python instance" was shocked and realized the dirty words and advertisement detection of the chat system in Python
  35. Convolutional neural network processing sequence for Python deep learning
  36. Python data structure and algorithm (1) -- enum type enum
  37. 超全大厂算法岗百问百答(推荐系统/机器学习/深度学习/C++/Spark/python)
  38. 【Python进阶】你真的明白NumPy中的ndarray吗?
  39. All questions and answers for algorithm posts of super large factories (recommended system / machine learning / deep learning / C + + / spark / Python)
  40. [advanced Python] do you really understand ndarray in numpy?
  41. 【Python进阶】Python进阶专栏栏主自述:不忘初心,砥砺前行
  42. [advanced Python] Python advanced column main readme: never forget the original intention and forge ahead
  43. python垃圾回收和缓存管理
  44. java调用Python程序
  45. java调用Python程序
  46. Python常用函数有哪些?Python基础入门课程
  47. Python garbage collection and cache management
  48. Java calling Python program
  49. Java calling Python program
  50. What functions are commonly used in Python? Introduction to Python Basics
  51. Python basic knowledge
  52. Anaconda5.2 安装 Python 库(MySQLdb)的方法
  53. Python实现对脑电数据情绪分析
  54. Anaconda 5.2 method of installing Python Library (mysqldb)
  55. Python implements emotion analysis of EEG data
  56. Master some advanced usage of Python in 30 seconds, which makes others envy it
  57. python爬取百度图片并对图片做一系列处理
  58. Python crawls Baidu pictures and does a series of processing on them
  59. python链接mysql数据库
  60. Python link MySQL database