Log all interaction with user to the DB

Due to compliance requirements in the financials industry we needed to log every request a user made to our system, the action taken (view function) and response from the server.

I found a lot of other logging solution bit most revolved around debugging and DB query logging. I needed to be able to tell what a user did while being logged in as much detail as I could with out tracking the mouse pointer position on screen.

So I created this (, my first ,) middleware. Its very simple really. keeping track of a request, view_func and response object in a single model called Record (models.py file included in the code).

The fields I used are optimized to what I intend to show in the UI I am planning for this model. Depending on how you use the doc string of your views they can be tapped to explain to the user what each request/func/responce group in a session is meant to do.

There were a few gotcha's:
1. I only care about authenticated requests. So I added the 'quest.user.is_authenticated()' test.
2. I did not care about the favicon request so I skipped them.
2. The actual login request is not authenticated while the response is. This caused the process_response/view to look for a record that is not there. So I added the 'except ObjectDoesNotExist' to skip this case.

I added one bell: Logging a full HTML reply is wasteful and mostly useless. I added two values in the setting files. LOGALL_LOG_HTML_RESPONSE to toggle if we want to log them or not. And LOGALL_HTML_START to describe what a full HTML starts with. Personally I use the first few characters of my base.html template that all the rest of my templates expend. I simplified the code to the left for readability.

#settings.py#Should we log full HTML responses?LOGALL_LOG_HTML_RESPONSE=True# If we how do we recognized a full HTML response LOGALL_HTML_START="<!DOCTYPE html"#logAll.models.pyfromdjango.dbimportmodelsfromdjango.contrib.auth.modelsimportUserclassRecord(models.Model):""" Basic log record describing all user interaction with the UI. Will be propagated by a middle ware. This will be one BIG DB table! """created_at=models.DateTimeField(auto_now_add=True)sessionId=models.CharField(max_length=256)requestUser=models.ForeignKey(User)requestPath=models.TextField()requestQueryString=models.TextField()requestVars=models.TextField()requestMethod=models.CharField(max_length=4)requestSecure=models.BooleanField(default=False)requestAjax=models.BooleanField(default=False)requestMETA=models.TextField(null=True,blank=True)requestAddress=models.IPAddressField()viewFunction=models.CharField(max_length=256)viewDocString=models.TextField(null=True,blank=True)viewArgs=models.TextField()responseCode=models.CharField(max_length=3)responseContent=models.TextField()#logAll.middleware.pyimporttimeimportsimplejsonasjsonfromdjango.core.exceptionsimportObjectDoesNotExistfromlogAll.modelsimportRecordfromsettingsimportLOGALL_LOG_HTML_RESPONSE,LOGALL_HTML_STARTclassLogAllMiddleware(object):defprocess_request(self,request):# Only log requests of authinticate users try:ifnotrequest.user.is_authenticated():returnNoneexceptAttributeError:returnNone# Skip favicon requests cause I do not care about themifrequest.path=="/favicon.ico":returnNonenewRecord=Record(created_at=str(time.time()),sessionId=request.session.session_key,requestUser=request.user,requestPath=request.path,requestQueryString=request.META["QUERY_STRING"],requestVars=json.dumps(request.REQUEST.__dict__),requestMethod=request.method,requestSecure=request.is_secure(),requestAjax=request.is_ajax(),requestMETA=request.META.__str__(),requestAddress=request.META["REMOTE_ADDR"],)newRecord.save()returnNonedefprocess_view(self,request,view_func,view_args,view_kwargs):try:ifnotrequest.user.is_authenticated():returnNoneexceptAttributeError:returnNone# Fix the issue with the authrization request try:theRecord=Record.objects.get(sessionId=request.session.session_key,requestUser=request.user,requestPath=request.path,requestMethod=request.method,requestSecure=request.is_secure(),requestAjax=request.is_ajax(),requestMETA=request.META.__str__())theRecord.viewFunction=view_func.func_nametheRecord.viewDocString=view_func.func_doctheRecord.viewArgs=json.dumps(view_kwargs)theRecord.save()exceptObjectDoesNotExist:passreturnNonedefprocess_response(self,request,response):# Only log autherized requeststry:ifnotrequest.user.is_authenticated():returnresponseexceptAttributeError:returnresponse# Skip favicon requests cause I do not care about themifrequest.path=="/favicon.ico":returnresponse# Fix the issue with the authorization request try:theRecord=Record.objects.get(sessionId=request.session.session_key,requestUser=request.user,requestPath=request.path,requestMethod=request.method,requestSecure=request.is_secure(),requestAjax=request.is_ajax(),requestMETA=request.META.__str__())theRecord.responseCode=response.status_code# Decidce wether we want to log the a full html response# as this will probabaly will take a LOT of space.## In my case most of the replies I want to catch happen# to be plain text ajax repliesifLOGALL_LOG_HTML_RESPONSE:# IF set to true then log the respoce regardless theRecord.responseContent=response.contentelifresponse.content.startswith(LOGALL_HTML_START):theRecord.responseContent="FULL HTML RESPONSE"else:theRecord.responseContent=response.contenttheRecord.save()exceptObjectDoesNotExist:passreturnresponse