defense – l.avala.mp's placehttps://l.avala.mp
Fri, 15 Dec 2017 14:56:29 +0000en-UShourly1https://wordpress.org/?v=4.8.5https://i1.wp.com/l.avala.mp/wp-content/uploads/2017/06/cropped-lavalamp-logo-gray-2.png?fit=32%2C32&ssl=1defense – l.avala.mp's placehttps://l.avala.mp
3232130076178Addressing Security Regression Through Unit Testing – Part 1https://l.avala.mp/?p=169
Tue, 04 Jul 2017 23:05:05 +0000https://l.avala.mp/?p=169Continue Reading Addressing Security Regression Through Unit Testing – Part 1]]>EHLO again! I had the pleasure of speaking at QCon NYC last week and I must say it was a pretty damn good conference. Unlike most of the conferences I’ve spoken at, this one was a developer conference. For anyone that likes speaking on security-related topics, I can’t recommend speaking at developer conferences strongly enough. It’s great to speak with one another in the security industry about all the problems plaguing the state of security in the world, but nine times out of 10 we are not the ones with boots on the ground responsible for fixing the myriad holes that we find. This responsibility quite often falls on the shoulders of developers, and as such we should see it as our responsibility to work closely with the software development community to equip them with the knowledge required to improve the general security posture of software.

But I digress – the talk that I gave was entitled Addressing Security Regression Through Unit Testing. This post is a write-up of the topic and a run-through of the software that I wrote to support the talk. I hope this content serves to inspire some of you to implement similar methods in your own codebases!!

Security Regression

Regression in codebases is a known and largely addressed subject. The problem of regression is, simply put, that there is no guarantee that the integrity of code is maintained as new code is added. In order to address this problem developers commonly rely upon unit tests – they write unit tests that check whether or not small components of code are working as intended. These tests are then run prior to deployment as new functionality is added to ensure that the new functionality has not broken the older, unit tested functionality.

Through my time in penetration testing, I’ve come to the conclusion that regression with respect to security is a similarly large (if not larger) problem. The number of times that I’ve been on an engagement, found a number of issues in software, counseled the software owners through what the problems were and how to properly fix them, verified that the proper fixes were in place, and then came back six months later to find that the problems were back is far more than I would like to admit. Sometimes the vulnerabilities were back in the same place that they had been found previously. Sometimes the same type of vulnerability was present in new functionality. In all cases though, it seemed that there was little lasting improvement to application security posture as a result of the engagement I had conducted. We can have a much longer conversation around the shortcomings of offensive security testing (which I may reserve for another blog post), but regardless of the details it appeared that even when teams were able to properly address vulnerabilities in their codebases/networks/environments there was no guarantee that those fixes would have any bearing on the continued improvement of the affected organization’s security posture.

So what can we do to start addressing this problem? Well that’s the purpose of this talk and blog post. It turns out we can use the same techniques that developers use to address integrity regression to address security regression. Furthermore, by using a technique that so much infrastructure has already been built around, we get additional improvements to the security posture of the affected codebase by leveraging that infrastructure (continuous integration/deployment infrastructure for example). Not only that but we can use introspection to dynamically generate security unit tests that will provide us with guarantees around the security posture of code that hasn’t even been written yet. Put altogether, we can use unit tests to address security regression to a significant extent.

Dynamically Generating Unit Tests

The code for this blog post is written using the Django Web Framework. The techniques discussed here will certainly work with other frameworks (and perhaps even compiled languages), but one core component that we’ll be leveraging here is the presence of an explicit mapping of URL routes to the views that handle requests to those routes. For example:

The reason that this approach requires explicit mapping is that we will use introspection to look into these URL routes and use them to dynamically generate unit tests for all of the views registered in the application.

While we can use introspection to enumerate all of these views, one thing I did not particularly want to do dynamically was figure out how to invoke all of the different HTTP verb functionality (e.g. GET, POST, PUT, DELETE, etc) for all of the registered views with valid HTTP requests. For instance, sending a POST request to the CreatePostView requires title, description, and image parameters. Dynamically figuring out how to create a valid request is certainly possible but would take a significant amount of effort. And so, the approach that I’m using requires a small amount of additional code written for every view in the application in the form of a Requestor class. The base Requestor class is shown below:

The Requestor class contains all of the functionality required for our unit tests to invoke all of the functionality for a view in the application. The class contains methods for sending requests for each of the supported HTTP verbs, as well as methods for retrieving the data that should be supplied alongside a given HTTP verb request in order to invoke the functionality successfully. The class also contains a list of the HTTP verbs that the view supports as well as whether or not the view requires authentication. Creating a Requestor for a particular view is quite simple. For example, the Requestor class for the CreatePostView view is shown below:

So now that we have the ability to enumerate all of the views found within the application, and we have the Requestor classes for invoking the functionality for all of these views, the last thing we need is to establish a mapping from the views to the requestors written for them. To accomplish this I introduce the notion of the Registry, which contains a dictionary mapping views to the Requestor classes written to invoke the relevant view functionality:

@Singleton
class TestRequestorRegistry(object):
"""
This is a class that maintains mappings from views to test cases that are configured to
send HTTP requests to the view in question.
"""
# Class Members
# Instantiation
def __init__(self):
self._registry = {}
# Static Methods
# Class Methods
# Public Methods
def add_mapping(self, requestor_path=None, requested_view=None):
"""
Add a mapping from the view to the given requestor class specified by requestor_path.
:param requestor_path: The path to the requestor class configured for the given view.
:param requested_view: The view that the requestor is meant to send requests for.
:return: None
"""
try:
requestor_class = self.__import_class(requestor_path)
except (ImportError, AttributeError) as e:
raise RequestorNotFoundException(
"Unable to load requestor at %s: %s."
% (requestor_path, e.message)
)
if not issubclass(requestor_class, BaseRequestor):
raise InvalidRequestorException(
"Class of %s is not a valid requestor class."
% (requestor_class.__name__,)
)
self._registry[requested_view] = requestor_class
def does_view_have_mapping(self, view):
"""
Check to see if a mapping exists between the given view and a requestor class.
:param view: The view to check a mapping for.
:return: Whether or not a mapping exists for the given view.
"""
return view in self.registry
def get_requestor_for_view(self, view):
"""
Get the requestor configured to send requests to the given view.
:param view: The view to retrieve the requestor for.
:return: The requestor configured to send requests to the given view.
"""
return self.registry[view]
def print_mappings(self):
"""
Print all of the mappings currently stored within the registry.
:return: None
"""
for k, v in self.registry.iteritems():
print("%s --> %s" % (k, v))
# Protected Methods
# Private Methods
def __import_class(self, class_path):
"""
Import the class at the given class path and return it.
:param class_path: The class path to the class to load.
:return: The loaded class.
"""
components = class_path.split(".")
mod = __import__(components[0])
for component in components[1:]:
mod = getattr(mod, component)
return mod
# Properties
@property
def registry(self):
"""
Get the registry mapping functions and classes to the test classes that are configured to
submit HTTP requests to them.
:return: the registry mapping functions and classes to the test classes that are
configured to submit HTTP requests to them.
"""
return self._registry

In order to establish the mapping from views to their related Requestor classes, I use the following decorator:

def requested_by(requestor_path):
"""
This is a decorator for views that maps a requestor class to the view that it is configured to
submit requests to.
:param requestor_path: A string depicting the local file path to the requestor class for the given
view.
:return: A function that maps the view class to the requestor path and returns the called function
or class.
"""
def decorator(to_wrap):
registry = TestRequestorRegistry.instance()
registry.add_mapping(requestor_path=requestor_path, requested_view=to_wrap)
return to_wrap
return decorator

This decorator can then be used to decorate the view classes, and the only argument to the decorator is the import path of the Requestor related to the view. For instance, the CreatePostView class is decorated as follows:

@requested_by("streetart.tests.requestors.pages.CreatePostViewRequestor")
class CreatePostView(BaseFormView):
"""
This is a view for creating new street art posts.
"""

With this approach, now every view will automatically be mapped to its related Requestor as soon as it is imported. To demonstrate this, run the following from the sectesting directory:

Retrieve a requestor for every view that has the ability to invoke all of the view’s functionality

Great! With this framework we can now dynamically generate unit tests for all of the HTTP verbs for all of the views in the application. This dynamic generation is handled by the StreetArtTestRunner class found in tests/runner.py, the contents of which are shown below:

class StreetArtTestRunner(DiscoverRunner):
"""
This is a custom discover runner for populating unit tests for the Street Art project.
"""
# Class Members
ALL_HTTP_VERBS = [
"GET",
"HEAD",
"POST",
"PUT",
"DELETE",
"OPTIONS",
"TRACE",
"PATCH",
]
CSRF_VERBS = [
"POST",
"PUT",
"DELETE",
"PATCH",
]
# Instantiation
def __init__(self, *args, **kwargs):
self._url_patterns = None
super(StreetArtTestRunner, self).__init__(*args, **kwargs)
# Static Methods
# Class Methods
# Public Methods
def build_suite(self, test_labels=None, extra_tests=None, **kwargs):
"""
Build the test suite to run for this discover runner.
:param test_labels: A list of strings describing the tests to be run.
:param extra_tests: A list of extra TestCase instances to add to the suite that is
executed by the test runner.
:param kwargs: Additional keyword arguments.
:return: The test suite.
"""
extra_tests = extra_tests if extra_tests is not None else []
extra_tests.extend(self.__get_generated_test_cases())
return super(StreetArtTestRunner, self).build_suite(
test_labels=test_labels,
extra_tests=extra_tests,
**kwargs
)
def run_suite(self, suite, **kwargs):
"""
Override the run_suite functionality to populate the database.
:param suite: The suite to run.
:param kwargs: Keyword arguments.
:return: The rest suite result.
"""
self.__populate_database()
return super(StreetArtTestRunner, self).run_suite(suite, **kwargs)
# Protected Methods
# Private Methods
def __get_authentication_enforcement_tests(self):
"""
Get a list of test cases that will test whether or not authentication is correctly enforced
on a given view.
:return: A list of test cases that will test whether or not authentication is correctly enforced
on a given view.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
if not requestor.requires_auth:
continue
for supported_verb in requestor.supported_verbs:
class AnonTestCase(AuthenticationEnforcementTestCase):
pass
to_return.append(AnonTestCase(view=view, verb=supported_verb))
return to_return
def __get_csrf_enforcement_tests(self):
"""
Get a list of test cases that check to make sure that CSRF checks are being correctly
enforced.
:return: A list of test cases that check to make sure that CSRF checks are being correctly
enforced.
"""
to_return = []
csrf_verbs = [x.lower() for x in self.CSRF_VERBS]
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
supported_verbs = [x.lower() for x in requestor.supported_verbs]
supported_csrf_verbs = filter(lambda x: x in csrf_verbs, supported_verbs)
for supported_csrf_verb in supported_csrf_verbs:
class AnonTestCase1(CsrfEnforcementTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=supported_csrf_verb))
return to_return
def __get_dos_class_tests(self):
"""
Get a list of test cases that will test to ensure that all of the configured URL routes
return successful HTTP status codes.
:return: A list of test cases that will test to ensure that all of the configured URL routes
return successful HTTP status codes.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
for supported_verb in requestor.supported_verbs:
class AnonTestCase1(RegularViewRequestIsSuccessfulTestCase):
pass
class AnonTestCase2(AdminViewRequestIsSuccessfulTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=supported_verb))
to_return.append(AnonTestCase2(view=view, verb=supported_verb))
return to_return
def __get_generated_test_cases(self):
"""
Get a list containing the automatically generated test cases to add to the test suite
this runner is configured to run.
:return: A list containing the automatically generated test cases to add to the test suite
this runner is configured to run.
"""
# Ensure that all views are loaded
import sectesting.urls
to_return = []
if settings.TEST_FOR_REQUESTOR_CLASSES:
to_return.extend(self.__get_requestor_class_tests())
if settings.TEST_FOR_DENIAL_OF_SERVICE:
to_return.extend(self.__get_dos_class_tests())
if settings.TEST_FOR_UNKNOWN_METHODS:
to_return.extend(self.__get_unknown_methods_tests())
if settings.TEST_FOR_AUTHENTICATION_ENFORCEMENT:
to_return.extend(self.__get_authentication_enforcement_tests())
if settings.TEST_FOR_RESPONSE_HEADERS:
to_return.extend(self.__get_response_header_tests())
if settings.TEST_FOR_OPTIONS_ACCURACY:
to_return.extend(self.__get_options_accuracy_tests())
if settings.TEST_FOR_CSRF_ENFORCEMENT:
to_return.extend(self.__get_csrf_enforcement_tests())
return to_return
def __get_options_accuracy_tests(self):
"""
Get a list of test cases that will test to ensure that no verbs other than those specified
in OPTIONS responses are present on all views.
:return: A list of test cases that will test to ensure that no verbs other than those specified
in OPTIONS responses are present on all views.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
supported_verbs = [x.lower() for x in requestor.supported_verbs]
for http_verb in self.ALL_HTTP_VERBS:
if http_verb.lower() not in supported_verbs:
class AnonTestCase1(RegularVerbNotSupportedTestCase):
pass
class AnonTestCase2(AdminVerbNotSupportedTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=http_verb))
to_return.append(AnonTestCase2(view=view, verb=http_verb))
return to_return
def __get_response_header_tests(self):
"""
Get a list of test cases that will test the views associated with the Street Art project to ensure
that the expected response headers are found in all responses.
:return: A list of test cases that will test the views associated with the Street Art project to ensure
that the expected response headers are found in all responses.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
for k, v in settings.EXPECTED_RESPONSE_HEADERS["included"].iteritems():
for supported_verb in requestor.supported_verbs:
class AnonTestCase1(HeaderKeyExistsTestCase):
pass
class AnonTestCase2(HeaderValueAccurateTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=supported_verb, header_key=k))
to_return.append(AnonTestCase2(view=view, verb=supported_verb, header_key=k, header_value=v))
for excluded_header in settings.EXPECTED_RESPONSE_HEADERS["excluded"]:
for supported_verb in requestor.supported_verbs:
class AnonTestCase3(HeaderKeyNotExistsTestCase):
pass
to_return.append(AnonTestCase3(view=view, verb=supported_verb, header_key=excluded_header))
return to_return
def __get_requestor_class_tests(self):
"""
Get a list of test cases that will test the views associated with the Street Art project to ensure
that the view has a requestor class associated with it.
:return: A list of test cases that will test the views associated with the Street Art project to ensure
that the view has a requestor class associated with it.
"""
to_return = []
for _, _, callback in self.url_patterns:
class AnonTestCase(ViewHasRequestorTestCase):
pass
to_return.append(AnonTestCase(self.__get_view_from_callback(callback)))
return to_return
def __get_unknown_methods_tests(self):
"""
Get a list of test cases that will test whether or not views return the expected HTTP verbs
through OPTIONS requests.
:return: A list of test cases that will test whether or not views return the expected HTTP verbs
through OPTIONS requests.
"""
to_return = []
for _, _, callback in self.url_patterns:
view = self.__get_view_from_callback(callback)
class AnonTestCase1(RegularUnknownMethodsTestCase):
pass
class AnonTestCase2(AdminUnknownMethodsTestCase):
pass
to_return.append(AnonTestCase1(view))
to_return.append(AnonTestCase2(view))
return to_return
def __get_view_from_callback(self, callback):
"""
Get the view associated with the given callback.
:param callback: The callback to get the view from.
:return: The view associated with the given callback.
"""
if hasattr(callback, "view_class"):
return callback.view_class
else:
return callback
def __get_view_and_requestor_from_callback(self, callback):
"""
Get a tuple containing (1) the view and (2) the requestor associated with the given URL
pattern callback.
:param callback: The URL pattern callback to process.
:return: A tuple containing (1) the view and (2) the requestor associated with the given URL
pattern callback.
"""
registry = TestRequestorRegistry.instance()
view = self.__get_view_from_callback(callback)
requestor = registry.get_requestor_for_view(view)
return view, requestor
def __populate_database(self):
"""
Populate the database with dummy database models.
:return: None
"""
print("Now populating test database...")
SaFaker.create_users()
# Properties
@property
def url_patterns(self):
"""
Get a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the
callback function for the views that this runner should generate automated tests for.
:return: a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the
callback function for the views that this runner should generate automated tests for.
"""
if self._url_patterns is None:
self._url_patterns = UrlPatternHelper.get_all_streetart_views(
include_admin_views=settings.INCLUDE_ADMIN_VIEWS_IN_TESTS,
include_auth_views=settings.INCLUDE_AUTH_VIEWS_IN_TESTS,
include_generic_views=settings.INCLUDE_GENERIC_VIEWS_IN_TESTS,
include_contenttype_views=settings.INCLUDE_CONTENTTYPE_VIEWS_IN_TESTS,
)
return self._url_patterns
# Representation and Comparison
def __repr__(self):
return "<%s>" % (self.__class__.__name__,)

To demonstrate how we dynamically generate tests, let’s take the RegularViewRequestIsSuccessfulTestCase as an example:

class RegularViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase):
"""
This is a test case for testing whether or not a view returns a HTTP response indicating
that the request was successful for a regular user.
"""
def runTest(self):
"""
Tests that the given view returns a successful HTTP response for the given verb.
:return: None
"""
requestor = self._get_requestor_for_view(self.view)
response = requestor.send_request_by_verb(self.verb, user_string="user_1")
self._assert_response_successful(
response,
"%s did not return a successful response for %s verb (%s status, regular user)"
% (self.view, self.verb, response.status_code)
)

This test case inherits from the BaseViewVerbTestCase class, which takes a view and a verb argument in the constructor. In order to generate instances of this test case for all of the verbs and views in the application, let’s take a look at the __get_dos_class_tests method in the StreetArtTestRunner class:

Iterate over all of the supported verbs listed in the Requestor (line 11)

For each of the supported verbs, we create an anonymous subclass of RegularViewRequestIsSuccessfulTestCase (line 13)

We add an instance of the anonymous subclass instantiated with the verb and view that we are iterating over (line 19)

This may look a bit odd – why are we creating an anonymous subclass? Well the way that the Python unit testing framework works we cannot have two instances of the same test case class in a test suite (it will only run one of them). As such, we create unique classes for each of the test cases that we need to run. It’s a bit of a quirk, but nothing we can’t handle!

And with that, we have the ability to dynamically generate unit tests for all of the functionality in our application. Let’s now take a look at how we can make good use of this capability.

Testing For Adherence To The Requestor Architecture

Since we are relying on our developers to add a little bit of extra functionality for all of the views that they author, a logical first step to test is that all of the code within our codebase does, in fact, follow our architecture. This is tested by the ViewHasRequestorTestCase:

class ViewHasRequestorTestCase(BaseViewTestCase):
"""
This is a test case for testing whether or not a view has a corresponding requestor
mapped to it.
"""
def runTest(self):
"""
Tests that the given view has a requestor mapped to it.
:return: None
"""
registry = TestRequestorRegistry.instance()
self.assertTrue(
registry.does_view_have_mapping(self.view),
"No requestor found for view %s." % self.view,
)

This test is rather simple – it takes the view that the test case was instantiated with and checks the Registry to make sure that a requestor for the view exists. To see the results of this test, check out the v0.1 tag:

git checkout tags/v0.1

Once checked out, modify the settings.py file so that only the Requestor check tests are enabled:

We can now run the following to verify that all of the views have a Requestor mapped to them:

python manage.py test

The result of running this command is shown below:

Requestor unit tests passing

That’s great and all that all of our unit tests are passing, but let’s make sure that they’re testing what we intend them to. To do so, we remove the requested_by decorator from the MyPostsListView view as shown below:

#@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor")
class MyPostsListView(BaseListView):
"""
This is a page for displaying all of the posts associated with the logged-in user.
"""
template_name = "pages/streetart_post_list.html"
model = StreetArtPost
paginate_by = 2
def get_queryset(self):
"""
Get all of the posts that are associated with the requesting user.
:return: All of the posts that are associated with the requesting user.
"""
return self.request.user.posts.all()

After commenting out requested_by as shown in line 1 above, we run the tests again:

Requestor unit tests failing

Sure enough, as expected one of our unit tests fails indicating that the MyPostsListView does not have a Requestor mapped to it. Great! Let’s go ahead and uncomment the requested_by decorator and continue.

Testing For Denial Of Service

Now that we know all of our views have the necessary Requestor class mapped to them, let’s test to make sure that all of the functionality within our application is working properly (ie: returns an HTTP status code indicating a successful request). Testing this is handled by the RegularViewRequestIsSuccessfulTestCase and AdminViewRequestIsSuccessfulTestCase test cases:

class RegularViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase):
"""
This is a test case for testing whether or not a view returns a HTTP response indicating
that the request was successful for a regular user.
"""
def runTest(self):
"""
Tests that the given view returns a successful HTTP response for the given verb.
:return: None
"""
requestor = self._get_requestor_for_view(self.view)
response = requestor.send_request_by_verb(self.verb, user_string="user_1")
self._assert_response_successful(
response,
"%s did not return a successful response for %s verb (%s status, regular user)"
% (self.view, self.verb, response.status_code)
)
class AdminViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase):
"""
This is a test case for testing whether or not a view returns a HTTP response indicating
that the request was successful for an admin user.
"""
def runTest(self):
"""
Tests that the given view returns a successful HTTP response for the given verb.
:return: None
"""
requestor = self._get_requestor_for_view(self.view)
response = requestor.send_request_by_verb(self.verb, user_string="admin_1")
self._assert_response_successful(
response,
"%s did not return a successful response for %s verb (%s status, admin user)"
% (self.view, self.verb, response.status_code)
)

These test cases simply send a request to the related view using the configured verb and check to see that the HTTP status code of the response indicates that the request was successful. To run these tests, configure the application to only run the denial of service test cases in settings.py:

As we did before, let’s modify our code to introduce a failing test case just to make sure that our unit tests are working as intended. To do this, raise a PermissionDenied exception in the EditPostView view’s get method:

As expected, we see two failed unit tests indicating that the EditPostView view is returning an HTTP status code indicating a request error. Great! As before, let’s modify the EditPostView back to what it was beforehand and continue.

Testing For Unknown HTTP Verbs

We’ve now got some guarantees that our code is following the Requestor framework and that all of the tested functionality is indicating success, but what if our Requestor classes aren’t testing all of the HTTP verbs supported by our views? It is incredibly common for frameworks (especially ones like Django and Rails) to have all sorts of crazy functionality under the hood, and I’ve abused functionality that developers didn’t know about to nefarious ends on more than one occasion. To ensure that we are testing all of the functionality within our application, let’s make sure that the supported verbs reported by our views match the verbs that our Requestor classes are configured to invoke. Testing this is handled by the RegularUnknownMethodsTestCase:

As shown above, we issue an HTTP OPTIONS request to the view, parse the contents of the Allow HTTP response header (which contains a comma-separated list of the verbs supported by the endpoint), and then check those verbs against the verbs listed in the related Requestor. To run these tests, configure settings.py to only enable the unknown methods test cases:

As shown above, we have multiple Requestor classes that are not testing all of the HTTP verbs associated with their views! Funnily enough – this output was exactly what I saw when I was first authoring the test codebase for this talk. I had no idea that views that subclassed DeleteView supported both the GET and DELETE HTTP verbs!

Taking a look at the DeletePostViewRequestor class, we see that the class only tests for the POST verb:

Let’s run the tests again and check to make sure we are now testing all of the HTTP verbs supported by our application’s views:

Unknown HTTP methods test cases passing

Boom! We now know that all of our Requestor classes are properly configured to test all of the HTTP verbs associated with all of our views!

Testing For Authentication Enforcement

Darn near every application contains functionality that is only available to users that are authenticated to the application. Wouldn’t it be great to ensure that all of our post-auth functionality is properly enforcing authentication checks? Well this case is tested for by the AuthenticationEnforcementTestCase. To see this test case, first checkout the v0.3 tag:

git checkout tags/v0.3

The contents of the AuthenticationEnforcementTestCase are shown below:

When generating instances of this unit test, we generate them only for endpoints that require authentication as shown in the __get_authentication_enforcement_tests method in the StreetArtTestRunner class:

def __get_authentication_enforcement_tests(self):
"""
Get a list of test cases that will test whether or not authentication is correctly enforced
on a given view.
:return: A list of test cases that will test whether or not authentication is correctly enforced
on a given view.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
if not requestor.requires_auth:
continue
for supported_verb in requestor.supported_verbs:
class AnonTestCase(AuthenticationEnforcementTestCase):
pass
to_return.append(AnonTestCase(view=view, verb=supported_verb))
return to_return

To run these tests we configure settings.py to enable only the authentication check test cases:

We see that an error is being thrown indicating that an attribute is being accessed on a user object when that user is not authenticated. The offending code can be found in the MyPostsListView view on line 16:

@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor")
class MyPostsListView(BaseListView):
"""
This is a page for displaying all of the posts associated with the logged-in user.
"""
template_name = "pages/streetart_post_list.html"
model = StreetArtPost
paginate_by = 2
def get_queryset(self):
"""
Get all of the posts that are associated with the requesting user.
:return: All of the posts that are associated with the requesting user.
"""
return self.request.user.posts.all()

Let’s check out the next tag to see what has changed in the MyPostsListView:

git checkout tags/v0.4

The new contents of MyPostsListView are shown below:

@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor")
class MyPostsListView(BaseListView):
"""
This is a page for displaying all of the posts associated with the logged-in user.
"""
template_name = "pages/streetart_post_list.html"
model = StreetArtPost
paginate_by = 2
def get(self, request, *args, **kwargs):
"""
Handle the processing of an HTTP GET request to this endpoint to ensure that the
requesting user has sufficient permissions.
:param request: The request to process.
:param args: Positional arguments.
:param kwargs: Keyword arguments.
:return: super.get.
"""
if not request.user.is_authenticated:
raise PermissionDenied
return super(MyPostsListView, self).get(request, *args, **kwargs)
def get_queryset(self):
"""
Get all of the posts that are associated with the requesting user.
:return: All of the posts that are associated with the requesting user.
"""
return self.request.user.posts.all()

Running the tests again, we see that all of our authentication checks are now being properly enforced:

Authentication check tests passing

Great! We now know that all of our post-auth endpoints are properly enforcing authentication checks.

Testing For HTTP Response Headers

For the uninitiated, there are a number of HTTP response headers that can greatly improve the security posture of the browsers used by your application’s clients. Even better, many of these headers require little-to-no configuration in your application to just work! To test our application for the presence of these headers, lets first check out the next tag:

git checkout tags/v0.5

Once checked out, we can find the test functionality for checking response headers in the HeaderKeyExistsTestCase and the HeaderValueAccurateTestCase test cases found in tests/cases/headers.py

These two test cases test (1) that a header key exists in every one of the HTTP responses and (2) that the value associated with each header key has the expected content. To generate these unit tests, we first have a list of the headers that we want to see in the application configured in settings.py:

When generating the unit tests, we iterate over the contents of the expected HTTP response headers and the views/verbs in our application as shown in the __get_response_header_tests method in the StreetArtTestRunner class:

def __get_response_header_tests(self):
"""
Get a list of test cases that will test the views associated with the Street Art project to ensure
that the expected response headers are found in all responses.
:return: A list of test cases that will test the views associated with the Street Art project to ensure
that the expected response headers are found in all responses.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
for k, v in settings.EXPECTED_RESPONSE_HEADERS["included"].iteritems():
for supported_verb in requestor.supported_verbs:
class AnonTestCase1(HeaderKeyExistsTestCase):
pass
class AnonTestCase2(HeaderValueAccurateTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=supported_verb, header_key=k))
to_return.append(AnonTestCase2(view=view, verb=supported_verb, header_key=k, header_value=v))
return to_return

In order to properly run these tests, configure the middleware in settings.py to exclude the middleware used for populating the headers:

As shown above, we have failing and erroring unit tests indicating which views are lacking which HTTP response headers. It turns out that the only security-related response header that we currently have in the application is the X-Frame-Options header! Let’s go ahead and fix this by checking out the next tag:

git checkout tags/v0.6

This tag introduces the SecurityHeadersMiddleware middleware which populates the relevant security headers in all responses that pass through it:

Fantastic – we now know that all of the HTTP verbs supported by all of our views are returning the HTTP response security headers that we want!

Testing For OPTIONS Accuracy

Sure we know that all of the functionality in our application is currently working, and that we are testing all of the HTTP verbs reported by OPTIONS responses returned by our views, but what if the OPTIONS responses weren’t being entirely honest? What if there were HTTP verbs that could be invoked but were not included in the Allow headers returned via OPTIONS requests? Well let’s test for it! First, let’s check out the next tag:

git checkout tags/v0.9

Once checked out, we can find the test cases that test for this functionality in the RegularVerbNotSupportedTestCase and the AdminVerbNotSupportedTestCase test cases:

class RegularVerbNotSupportedTestCase(BaseViewVerbTestCase):
"""
This is a test case for testing whether or not a given HTTP verb is denied when submitted against
a view by a regular user.
"""
def runTest(self):
"""
Tests that self.verb does not work against self.view.
:return: None
"""
requestor = self._get_requestor_for_view(self.view)
response = requestor.send_request_by_verb(self.verb, user_string="user_1")
self._assert_response_not_allowed(
response,
"HTTP verb %s returned %s status code when it should have been 405 (regular user)."
% (self.verb, response.status_code)
)
class AdminVerbNotSupportedTestCase(BaseViewVerbTestCase):
"""
This is a test case for testing whether or not a given HTTP verb is denied when submitted against
a view by an admin user.
"""
def runTest(self):
"""
Tests that self.verb does not work against self.view.
:return: None
"""
requestor = self._get_requestor_for_view(self.view)
response = requestor.send_request_by_verb(self.verb, user_string="admin_1")
self._assert_response_not_allowed(
response,
"HTTP verb %s returned %s status code when it should have been 405 (admin user)."
% (self.verb, response.status_code)
)

These tests check to make sure that the relevant view and HTTP verb return a 405 HTTP status indicating that the verb is not supported. To generate these tests, we iterate over all the views in the application and create test cases for every HTTP verb that is supposedly not supported as found in the __get_options_accuracy_tests method in the StreetArtTestRunner class:

def __get_options_accuracy_tests(self):
"""
Get a list of test cases that will test to ensure that no verbs other than those specified
in OPTIONS responses are present on all views.
:return: A list of test cases that will test to ensure that no verbs other than those specified
in OPTIONS responses are present on all views.
"""
to_return = []
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
supported_verbs = [x.lower() for x in requestor.supported_verbs]
for http_verb in self.ALL_HTTP_VERBS:
if http_verb.lower() not in supported_verbs:
class AnonTestCase1(RegularVerbNotSupportedTestCase):
pass
class AnonTestCase2(AdminVerbNotSupportedTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=http_verb))
to_return.append(AnonTestCase2(view=view, verb=http_verb))
return to_return

It looks like some sneaky devil has put an (albeit completely useless) backdoor into our code! Let’s check out the next tag and verify that the backdoor has been removed:

git checkout tags/v0.10

We take a look at the MyPostsListView view to see that the backdoor has been removed:

@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor")
class MyPostsListView(LoginRequiredMixin, BaseListView):
"""
This is a page for displaying all of the posts associated with the logged-in user.
"""
template_name = "pages/streetart_post_list.html"
model = StreetArtPost
paginate_by = 2
def get_queryset(self):
"""
Get all of the posts that are associated with the requesting user.
:return: All of the posts that are associated with the requesting user.
"""
return self.request.user.posts.all()

Sure enough, the backdoor has been removed, and when we run the tests now we see that there is no hidden functionality within our application:

HTTP OPTIONS tests passing

Testing For CSRF Protection

Cross-site request forgery (CSRF) is a rather tricky vulnerability to describe, so I’ll let the good folks over at OWASP do the heavy lifting here. Long story short, for any non-idempotent HTTP verbs handled by our application we should have something called a CSRF token required in the request. If this token is either not present or not accurate, the request should be considered invalid and dropped immediately. To test that our application is properly guarded against CSRF, let’s check out the next tag:

git checkout tags/v0.11

The test case that will be checking for CSRF enforcement is entitled CsrfEnforcementTestCase:

class CsrfEnforcementTestCase(BaseViewVerbTestCase):
"""
This is a test case for testing that a CSRF token is properly enforced on a view with
a given verb via a request that is submitted by a regular user.
"""
def runTest(self):
"""
Test to ensure that that the CSRF token is properly enforced on the referenced view.
:return: None
"""
requestor = self._get_requestor_for_view(self.view)
response = requestor.send_request_by_verb(
self.verb,
user_string="user_1",
enforce_csrf_checks=True,
)
self._assert_response_permission_denied(
response,
"Response from %s indicated that CSRF protection was not enabled (verb was %s, status %s)."
% (self.view, self.verb, response.status_code)
)

This test case sends a request to the view and tells the Django unit testing framework to enforce CSRF checks. If the HTTP response indicates that the request was successful, the test case fails (as the request does not have a CSRF token)! To generate these unit tests, we iterate over all of the views in our application and create a test case for every non-idempotent HTTP verb as shown in the __get_csrf_enforcement_tests method in the StreetArtTestRunner class:

def __get_csrf_enforcement_tests(self):
"""
Get a list of test cases that check to make sure that CSRF checks are being correctly
enforced.
:return: A list of test cases that check to make sure that CSRF checks are being correctly
enforced.
"""
to_return = []
csrf_verbs = [x.lower() for x in self.CSRF_VERBS]
for _, _, callback in self.url_patterns:
view, requestor = self.__get_view_and_requestor_from_callback(callback)
supported_verbs = [x.lower() for x in requestor.supported_verbs]
supported_csrf_verbs = filter(lambda x: x in csrf_verbs, supported_verbs)
for supported_csrf_verb in supported_csrf_verbs:
class AnonTestCase1(CsrfEnforcementTestCase):
pass
to_return.append(AnonTestCase1(view=view, verb=supported_csrf_verb))
return to_return

Whoa! It looks like there are a handful of endpoints in our application that are not properly enforcing CSRF protections. Taking a look at the CreatePostView view, we can see exactly where the protections are being disabled:

On line 1 above we see that a csrf_exempt method is wrapping the dispatch method in the view. This is effectively disabling CSRF protections for the view. Let’s go ahead and check out the next tag to fix this issue:

git checkout tags/v0.12

Once checked out, we take another look at the CreatePostView and confirm that the decorator is no longer present:

We can then run the unit tests again to confirm that CSRF protections are properly enabled in our application:

CSRF enforcement tests passing

The Benefits Of Dynamic Generation

At this point, our application now has the following security guarantees:

All of our views have requestors mapped to them

All of the verbs supported by our views are working properly (or at least they are returning HTTP status codes indicating success)

All endpoints that require authentication are properly enforcing authentication checks

None of our views support any HTTP verbs other than what they report in the OPTIONS response

CSRF tokens are properly being enforced for non-idempotent requests

HTTP response security headers are present in all responses from all of the verbs from all of our views

That’s great and all, but perhaps you’re wondering why we should use dynamic generation to test for any of this. We could just write traditional unit tests to check for all of this functionality, right?

That’s right! We definitely could. However, this would require developers to write those unit tests for all of the new views that they author. As we discussed at the beginning of this post, just because we have passing unit tests for the functionality that we’ve written already doesn’t mean anything about code that we have yet to write/bring into our codebase. This is where dynamic generation truly shines – all of our dynamically-generated unit tests will automatically be applied to all of the views that are added to our application (even the ones that haven’t been written yet). As such, the same guarantees that we have about our application right now will hold true for all of the functionality that we implement moving forward. Let’s take a look at what I mean here by checking out the next tag:

git checkout tags/v0.15

In this version, some devious developer has introduced a new view into our application that has quite a few problems with it. This view, entitled NewCreatePostView, is shown below:

We immediately see an error indicating that a Requestor class does not exist for this troublesome view! Ok, so we go and talk to the developer and get them to add a Requestor and they commit to the next tag:

git checkout tags/v0.16

We check the contents of the NewCreatePostView and confirm that a Requestor is now configured for the view:

This view has hidden functionality in the TRACE verb when an admin user requests it

Without writing a single additional unit test, we have the same security guarantees around this new view that we’ve had for all of the other views in our application. This is powerful.

In addition to providing protections around code that hasn’t even been authored yet, we have a significant return on investment for the amount of effort we put into writing unit tests. In exchange for writing the plumbing for the test generation as well as writing the test case itself, we get a large number of tests for all of our views and verbs. Specifically in this demo application, in exchange for writing 12 unit tests we have a total of 693 data points indicating that the security posture of our application has not regressed to a prior state:

All dynamically generated unit tests passing

693 data points in exchange for 12 unit tests seems like a pretty great trade off to me.

Where To From Here

So we now have a pretty solid security foundation for our codebase through the use of dynamic unit test generation, but does this cover everything?

Absolutely not! We can really only check for some basic security controls through dynamic generation. What about instances where someone discovers a specific vulnerability in a specific view? Well that’s something that we can write traditional unit tests for, and then we can use those traditional unit tests in conjunction with these dynamic tests to provide not only a solid foundation of security posture but also guarantees that discovered vulnerabilities do not re-emerge in our codebase. This will be the topic of Addressing Security Regression Through Unit Testing – Part 2, coming to my blog in the near future!

I hope you all enjoyed this, and keep your eyes peeled for part two soon!