Tales from Django Middleware

Apr 24, 2018 • Mark Eschbach

Tags:

Over the past two days I’ve been working on modifying the django-request-logger
to honor the annotations of not logging the request entity nor the response entity. The two particular business cases
which support this charge are security and throughput issues. If you are accepting a request or response entity which
validate a password then the user password really shouldn’t be logged. Throughput is an issue with extremely large
request or response entities. For some requests I’ve seen the response time take up to 92% in logging.

Yesterday I learned the particular test fixture used in the request logger does not report exceptions risen in the
settings module. This was quiet unfortunate as I spent most of my non-meeting times yesterday chugging away at this.
It was only after JG mentioned it was really strange the module is acting like something was not imported I realized
all errors were being consumed.

Prior to the PR which will push 0.6.1 out we did not directly test the Django Rest Framework. Until I got 0.6.0
integrated into our main application there was not a reason to. Luckily a negative test pointed out the
Django Request Logger needed to deal with nonexistent route gracefully. Getting DRF integrated was interesting and
this was my first experience integrating it into an existing project.

The following was added to the libraries requirements-dev.txt file: djangorestframework==3.8.2. In the future
I would like to expand the test matrix to test against multiple versions of DRF & Django but we just have pinned
versions right now. Next up was creating a new DRF viewset and wiring it in.

Last up was probably the most annoying. This was my first time really modifying the URLs for a Django application.
From best I can tell a URL in Django is a pair of (regex, func) group to consume a give URI from within the service.
Nice and easy to hook up new route handlers. However this became a serious stumbling block in regards on how to
integrate the router object. Initially I desired to add the the whole router under /drf to make the tests clear on
what was being tested. However after working on the tests for a bit I eventually resigned to just having DRF work off
the root URI. Perhaps with more experience in the future I will be able to clean that up and collapse it. For now
this is what I had settled on:

Once I got a few tests which exhibited the issue && I fixed then BAM pull request. Then Travis indicated a failure
while I was cleaning up the PR text. Turns out Python 3.7-dev is upset.

Armed with pyenv life has been a bit easier. Just pyenv install 3.7-dev and wait. For a while. Not bad in terms
of getting a software environment done. Until it spins for a while then fails. Fails because of zipimport.ZipImportError: can't decompress data; zlib not available.
Looking back at Getting multiple version of Python installed on OSXpyenv needs CFLAGS="-I$(xcrun --show-sdk-path)/usr/include" prefixed on OSX. Nope, no luck there. On to using
a Linux machine.

Alright, so I setup a Linux machine with a Fuse SSH mount. Time to start debugging.

After some time, I’m not sure why the routes are being sucked in. We currently do not use Python3.7-dev and I am
not so certain we should be supporting pre-release builds. Unforunately I’m out of time and will have to return to the
future comparability at another time.

The Knowledge Base

This is a collection of publicly facing notes. I am hoping someone finds the notes I post here helpful.

Over the years I have found many notes, articles, and mediatations published on the internet helpful. This is my attempt to provide back to the community.