python In the standard library logging Modules are often used when logging , But in the actual use, we found that its own class for local log rollback

logging.handlers.RotatingFileHandler In a multi process environment, different processes write to different files , The reason is at present

When the log file is full and rolled back, the concurrency problem is not handled properly ( Or it can be said that basically no treatment ), So I implemented a class with similar functions ,

Originally intended to use multi process lock , I found it useless after I finished writing it ..., So there's a new one .lock Files act as locks to handle multiple processes . With documents

Modify the time to control that only one process can access .

The code is as follows :

import os
import sys
import json
import time
import logging
import traceback
import logging.handlers
from multiprocessing import Lock class SpiderRotatingFileHandler(logging.handlers.RotatingFileHandler):
File rollback log processor
characteristic :
1. Use the modification time of backup file to judge Fixed the problem that multiple log files were written simultaneously in multiple processes bug
2. optional Use json Format log file '''
def __init__(self, filename, mode='a', maxBytes=0,
backupCount=0, encoding=None, delay=0, is_json=False):
filename, mode, maxBytes, backupCount, encoding, delay)
# Format processor
self.Formatter = logging.Formatter()
# Process lock
self.my_lock = Lock() self.is_json = is_json
if self.is_json:
self.format = self.json_format def json_format(self, record):
json Format log
@record: Logging object
type: logging.LogRecord
# increase asctime attribute
record.asctime = self.Formatter.formatTime(record)
message = record.getMessage()
log_data = {}
# Check if it is json Format And it's in dictionary form
log_data = json.loads(message)
if not isinstance(log_data, dict):
log_data = {}
except Exception as e:
exc_info = traceback.format_exc()
#sys.stderr.write(exc_info) # Get the basic information of the log
log_record_basic_fields = [
"levelname", "filename", "lineno",
"name", "created", "asctime",
] if not log_data:
"_message": message,
}) for attr in log_record_basic_fields:
value = getattr(record, attr, "")
"_{}".format(attr): value,
result = json.dumps(log_data, ensure_ascii=False)
result = json.dumps(log_data)
return result def doRollover(self):
Do a rollover, as described in __init__().
with self.my_lock:
if = None
lock_file = "%s.lock"%self.baseFilename
max_modify_interval = 3 # seconds
do_flag = 0 # utilize Lock The modification time of files ensures that multiple files will not be written at the same time
if not os.path.exists(lock_file):
with open(lock_file, "w"):
do_flag = 1
elif time.time() - os.stat(lock_file).st_mtime > max_modify_interval:
do_flag = 1
if do_flag:
for i in range(self.backupCount - 1, 0, -1):
sfn = "%s.%d" % (self.baseFilename, i)
dfn = "%s.%d" % (self.baseFilename, i + 1)
if os.path.exists(sfn):
# Delete the largest backup file
if os.path.exists(dfn):
os.rename(sfn, dfn) dfn = self.baseFilename + ".1"
if os.path.exists(dfn):
os.remove(dfn) if os.path.exists(self.baseFilename):
os.rename(self.baseFilename, dfn)
# Refresh Lock File modification time
with open(lock_file, "w"):
pass if not self.delay: = self._open()

After testing, we found that , Log files are no longer messy to write ( But I always feel 3 There seems to be something wrong with seconds , In case of 3 Seconds full of days

Log files may cause the log file size to exceed the limit .)

json Format log output is an additional function

ok, Welcome to find fault

python logging Implementation of the process security file rollback log class more related articles

  1. Log4net introduction ( Roll back the log file )

    In the last article Log4net( Log files ) in , We use "log4net.Appender.FileAppender" Output log information to a single file , As applications continue to be used , The log file will ...

  2. git Local file rollback operation

    Today, several documents have been changed to other branches . Need to roll back . Refer to the following two articles : Link    Link In short , There are many different stages : 1. use git status Order to see , Found to be unstaged, So it's just work ...

  3. git File rollback

    scene 1: When you change the content of a file in the workspace , When you want to discard changes to the workspace directly , Command git checkout -- file. scene 2: When you not only change the content of a file in the workspace , Also added to the staging area , Want to discard changes , A two-step , The first ...

  4. python logging Log module

    One .logging Module introduction logging The module is Python Built in standard modules , It is mainly used to output the operation log , You can set the level of output log . Log save path . Log file rollback, etc : comparison print, It has the following advantages : It can be done by setting No ...

  5. ( turn )python logging modular

    python logging modular original text : 1 logging Module introduction logging The module is Python ...

  6. 13 python logging modular

    original text : 1 logging Module introduction logging The module is Python Built in standard modules , It is mainly used to output running days ...

  7. python logging modular 【 Reprint 】

    from : Reference resources : Old urchin log modular , It's very detailed , Basically, you can use it when you get it , nice 1 logging modular ...

  8. python logging modular

    1.logging The module provides four components logger: Log class , There are two functions 1) Configure the log level , processor handler, filter filterlogger.setLevel(logging.INFO)logger. ...

  9. utilize lsof Files occupied by the recovery process

    explain : This is often the case , Not cleaning up the files occupied by the process in the right way , Like logs . Causing the space not to be released . Sometimes it is necessary to recover the files occupied by the process . Solution lsof |grep del #  Find out the name of the file you want to recover . ...

Random recommendation

  1. All these years Android - mother

    Remember when I was a kid , After I finish writing a composition , After finishing the task assigned by my mother , I'll move the big chair my mother is using , Facing the road in front of the door , Just sitting with such longing , Listen, when the mother gives the baby an injection , The cry of a child , Hard to imagine that I can be a man of indomitable spirit in the future ...

  2. godaddy Spatial sql server The database can't insert chinese

    Original code : use string007 from sysobjects where id = object_id('K_V_TEST') and type = 'U') drop table K_V ...

  3. Node The way to debug -----JSHint

    Node The way to debug -----JSHint Node I don't flatter you any more , It makes javascript Unified web It's possible to have a front and back stage . But for the novice ,server Terminal JS The code may not look like client So ...

  4. ASP.NET MVC The filter of learning (2)

    Now let's continue with the previous ASP.NET MVC The filter of learning (1) To study . 3. Action filters seeing the name of a thing one thinks of its function , This filter responds before and after the action method is called . We can change the action of the actual call before the call , It can also be done after the action call ...

  5. UITableView Detailed use of

    UITableView Detailed use of   UITableView yes app Commonly used controls in development , It's very powerful , Mostly used for data display . Here is a simple example to introduce tableview The basic usage of .( Suitable for novice , The master floats by ) @ ...

  6. Cocoa Middle level (layer) Minimalist understanding of coordinate systems

    Giant panda pig · Hou Pei's original or translated works . Welcome to reprint , Reprint please indicate the source . If you think it's not well written, please give more comments , If you think it's good, please support it . thank you ! hopy ;) Cocoa The coordinate system of the layer is not clear all the time , Now let's sort it out and sum it up : ...

  7. C++ Polymorphism and virtual function of

    Polymorphism in object-oriented programming is to send the same message to different objects , Different objects have different behaviors corresponding to the same message . In a program, a message is a call to a function , Different behaviors refer to different implementation methods , That is, executing different function bodies . It can also be said that it has been realized &quo ...

  8. [CodeForce 450A] Jzzhu and Children

    Topic link : /* * Count how many times a person has to line up to get enough to leave , The person who has more choices , If two people line up the same number of times , ...

  9. Build local yum Source

    Local yum The source is actually very easy to build First of all to enter /etc/yum.repos.d/ The original yum Remove after source backup , Then build dvd.repo: The contents are as follows : [base] name=base baseurl=file ...

  10. python Study note 3 - Calculate run time

    Method 1 import datetime starttime = #long running endtime = ...