Testing and Debugging in Grok 1.0: Part 1

Grok offers some tools for testing, and in fact, a project created by grokproject (as the one we have been extending) includes a functional test suite. In this article, we are going to discuss testing a bit and then write some tests for the functionality that our application has so far.

Testing helps us avoid bugs, but it does not eliminate them completely, of course. There are times when we will have to dive into the code to find out what's going wrong. A good set of debugging aids becomes very valuable in this situation. We'll see that there are several ways of debugging a Grok application and also try out a couple of them.

Testing

It's important to understand that testing should not be treated as an afterthought.

As mentioned earlier, agile methodologies place a lot of emphasis on testing. In fact, there's even a methodology called Test Driven Development (TDD), which not only encourages writing tests for our code, but also writing tests before any other line of code.

There are various kinds of testing, but here we'll briefly describe only two:

Unit testing

Integration or functional tests

Unit testing

The idea of unit testing is to break a program into its constituent parts and test each one of them in isolation. Every method or function call can be tested separately to make sure that it returns the expected results and handles all of the possible inputs correctly.

An application which has unit tests that cover the majority of its lines of code, allows its developers to constantly run the tests after a change, and makes sure that modifications to the code do not break the existing functionality.

Functional tests

Functional tests are concerned with how the application behaves as a whole. In a web application, this means how it responds to a browser request and whether it returns the expected HTML for a given call.

Ideally, the customer himself has a hand in defining these tests, usually through explicit functionality requirements or acceptance criteria. The more formal the requirements from the customer are, the easier it is to define appropriate functional tests.

Testing in Grok

Grok highly encourages the use of both kinds of tests, and in fact, includes a powerful testing tool that is automatically configured with every project. In the Zope world—from where Grok originated—a lot of value is placed in a kind of tests known as "doctests", so Grok comes with a sample test suite of this kind.

Doctests

A doctest is a test that's written as a text file, with lines of code mixed with explanations of what the code is doing. The code is written in a way that simulates a Python interpreter session. As tests exercise large portions of the code (ideally 100%), they usually offer a good way of finding out of what an application does and how. So, if an application has no written documentation, its tests would be the next obvious way of finding out what it does. Doctests take this idea further by allowing the developer to explain in the text file exactly what each test is doing.

Doctests are especially useful for functional testing, because it makes more sense to document the high-level operations of a program. Unit tests, on the other hand, are expected to evaluate the program bit by bit and it can be cumbersome to write a text explanation for every little piece of code.

A possible drawback of doctests is that they can make the developer think that he needs no other documentation for his project. In almost all of the cases, this is not true. Documenting an application or package makes it immediately more accessible and useful, so it is strongly recommended that doctests should not be used as a replacement for good documentation. We'll show an example of using doctests in the Looking at the test code section of this article.

Default test setup for Grok projects

As mentioned above, Grok projects that are started with the grokproject tool already include a simple functional test suite by default. Let's examine it in detail.

Test configuration

The default test configuration looks for packages or modules that have the word 'tests' in their name and tries to run the tests inside. For functional tests, any files ending with .txt or .rst are considered.

For functional tests that need to simulate a browser, a special configuration is needed to tell Grok which packages to initialize in addition to the Grok infrastructure (usually the ones that are being worked on). The ftesting.zcml file in the package directory has this configuration. This also includes a couple of user definitions that are used by certain tests to examine functionality specific to a certain role, such as manager.

Test files

Besides the already mentioned ftesting.zcml file, in the same directory, there is a tests.py file added by grokproject, which basically loads the ZCML declarations and registers all of the tests in the package.

The actual tests that are included with the default project files are contained in the app.txt file. These are doctests that do a functional test run by loading the entire Grok environment and imitating a browser. We'll take a look at the contents of the file soon, but first let's run the tests.

Running the tests

As part of the project's build process, a script named test is included in the bin directory when you create a new project. This is the test runner and calling it without arguments, finds and executes all of the tests in the packages that are included in the configuration.

We haven't added a single test so far, so if we type bin/test in our project directory, we'll see more or less the same thing that doing that on a new project would show:

Ran 3 tests with 0 failures and 0 errors in 0.465 seconds. Tearing down left over layers: Tear down todo.FunctionalLayer ... not supported

The only difference between our output to that of a newly created Grok package is in the sqlalchemy lines. Of course, the most important part of the output is the "penultimate" line, which shows the number of tests that were run and whether there were any failures or errors. A failure means that some test didn't pass, which means that the code is not doing what it's supposed to do and needs to be checked. An error signifies that the code crashed unexpectedly at some point, and the test couldn't even be executed, so it's necessary to find the error and correct it before worrying about the tests.

The test runner

The test runner program looks for modules that contain tests. The test can be of three different types: Python tests, simple doctests, and full functionality doctests. To let the test runner know, which test file includes which kind of tests, a comment similar to the following is placed at the top of the file:

Do a Python test on the app.

:unittest:

In this case, the Python unit test layer will be used to run the tests. The other value that we are going to use is "doctest" when we learn how to write doctests.

The test runner then finds all of the test modules and runs them in the corresponding layer. Although unit tests are considered very important in regular development, we may find functional tests more necessary for a Grok web application, as we will usually be testing views and forms, which require the full Zope/Grok stack to be loaded to work. That's the reason why we find only functional doctests in the default setup.

Test layers

A test layer is a specific test setup which is used to differentiate the tests that are executed. By default, there is a test layer for each of the three types of tests handled by the test runner. It's possible to run a test layer without running the others and also to name new test layers to be able to cluster together tests that require a specific setup.

Invoking the test runner

As shown above, running bin/test will start the test runner with the default options. It's also possible to specify a number of options, and the most important ones are summarized below. In the following table, command-line options are shown to the left. Most options can be expressed with a short form (one dash) or a long form (two dashes). Arguments for the option in question are shown in uppercase.

-s PACKAGE,

--package=PACKAGE,

--dir=PACKAGE

Search the given package's directories for tests. This can be specified more than once, to run tests in multiple parts of the source tree. For example, when refactoring interfaces, you don't want to see the way you have broken setups for tests in other packages. You just want to run the interface tests. Packages are supplied as dotted names. For compatibility with the old test runner, forward and backward slashes in package names are converted to dots. (In the special case of packages, which are spread over multiple directories, only directories within the test search path are searched.)

-m MODULE,

--module=MODULE

Specify a test-module filter as a regular expression. This is a case sensitive regular expression, which is used in search (not match) mode, to limit which test modules are searched for tests. The regular expressions are checked against dotted module names. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can specy multiple test-module filters. Test modules matching any of the test filters are searched. If no test-module filter is specified, then all of the test modules are used.

-t TEST, --test=TEST

Specify a test filter as a regular expression. This is a case sensitive regular expression, which is used in search (not match) mode, to limit which tests are run. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can specify multiple test filters. Tests matching any of the test filters are included. If no test filter is specified, then all of the tests are executed.

--layer=LAYER

Specify a test layer to run. The option can be given multiple times to specify more than one layer. If not specified, all of the layers are executed. It is common for the running script to provide default values for this option. Layers are specified regular expressions that are used in search mode, for dotted names of objects that define a layer. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The layer named 'unit' is reserved for unit tests, however, take note of the -unit and non-unit options.

-u, --unit

Executes only unit tests, ignoring any layer options.

-f, --non-unit

Executes tests other than unit tests.

-v, --verbose

Makes output more verbose. Increment the verbosity level.

-q, --quiet

Makes the output minimal by overriding any verbosity options.

Looking at the test code

Let's take a look at the three default test files of a Grok project, to see what each one does.

ftesting.zcml

As we explained earlier, ftesting.zcml is a configuration file for the test runner. Its main objective is to help us set up the test instance with users, so that we can test different roles according to our needs.

As shown in the preceding code, the configuration simply includes a security policy, complete with users and roles and the packages that should be loaded by the instance, in addition to the regular Grok infrastructure. If we run any tests that require an authenticated user to work, we'll use these special users.

The includes at the top of the file just make sure that all of the Zope Component Architecture setup needed by our application is performed prior to running the tests.

tests.py

The default test module is very simple. It defines the functional layer and registers the tests for our package:

After the imports, the first line gets the path for the ftesting.zcml file, which then is passed to the layer definition method ZCMLLayer. The final line in the module tells the test runner to find and register all of the tests in the package.

This will be enough for our testing needs in this article, but if we needed to create another non-Grok package for our application, we would need to add a line like the last one to it, so that all of its tests are found by the test runner. This is pretty much boilerplate code, as only the package name has to be changed.

app.txt

We finally come to the reason for this entire configuration—the actual tests that will be executed by the test runner. By default, the tests are included inside the app.txt file:

Do a functional doctest test on the app. ========================================

The zope.testbrowser.browser module exposes a Browser class that simulates a web browser similar to Mozilla Firefox or IE. We use that to test how our application behaves in a browser. For more information, see http://pypi.python.org/pypi/zope.testbrowser.

The text file has a title and immediately after that a :doctest: declaration—a declaration which tells the test runner that these tests need a functional layer to be loaded for their execution. Then comes a :layer: declaration, which is a path that points to the layer that we defined earlier in tests.py. After that, comes the test code. Lines starting with three brackets represent the Python code that is tested. Anything else is commentary.

When using the Python interpreter, a line of code may return a value, in which case, the expected return value must be written immediately below that line. This expected value will be compared with the real return value of the tested code and a failure will be reported, if the values don't match. Similarly, a line which is followed by an empty line will produce a failure, when the code is executed and a result is returned, because it is assumed that the expected return value in that case is None.

For example, in the last line of the Python doctest, the expression browser.headers.get('Status').upper() is expected to return the value 200 OK. If anything else is returned, the test will fail, even if there's just a slight difference.

Adding our own tests

Now, let's add a few functional tests that are specific to our application. We will need to emulate a browser for that. The zope.testbrowser package includes a browser emulator. We can pass any valid URL to this browser by using browser.open, and it will send a request to our application exactly like a browser would. The response from our application will be then available as browser.contents, so that we can perform our testing comparisons on it.

The Browser class

Before writing our tests, it will be useful to see what exactly our testbrowser can do. Of course, anything that depends on JavaScript will not work here, but other than that, we can interact with links and even forms in a very straightforward manner. Here's a look at the main functionality offered by the Browser class:

Now that we know what we can do, let's try our hand at writing some tests.

Our first to-do application tests

Ideally, we should have been adding a couple of doctests to the app.txt file every time we add a new functionality to our application. We have gone through the reasons why we didn't do so, but let's recover some lost ground. At the very least, we'll get a feeling of how doctests work.

We'll add our new tests to the existing app.txt file. The last test there that we saw left us at the to-do instance URL. We are not logged in, so if we print the browser contents, we will get the login page. Let's add a test for this:

Since we haven't logged in, we can't see the application. The loginpage appears:

As we mentioned earlier, when visiting a URL with the testbrowser, the entire HTML content of the page is stored in browser.contents. Now we know that our login page has a username and a password field, so we simply use a couple of in expressions and check if these fields evaluate to True. If they do, it would mean that the browser is effectively looking at the login page.

Let's add a test for logging in. When we start the application in the tests, the user database is empty, therefore, the most economical way of logging in is to use basic authentication. This can be easily done by changing the request headers:

First, we find the link on the home page that will take us to the 'add form' project. This is done easily with the help of the getLink method and the text of the link. We click on the link and now should have the form ready to fill in. We then use getControl to find each field by its name and change its value. Finally, we submit the form by getting the submit button control and clicking on it. The result is that the project is created and we are redirected to its main view. We can confirm this by comparing the browser url with the URL that we would expect in this case.

Adding a list to the project is just as easy. We get the form controls, assign them some values, and click on the submit button. The list and the link for adding new items to it should appear in the browser contents:

We have added a project. Now, we'll add a list to it. If we are successful, we will seea link for adding a new item for the list:

That's the default behavior because this is how real browsers work, but sometimes, when we are debugging, it's better to take a look at the original exception caused by our application. In such a case, we can make the browser stop handling errors automatically and throw the original exceptions, so that we can handle them. This is done by setting the browser.handleErrors property to False:

Alerts & Offers

Series & Level

We understand your time is important. Uniquely amongst the major publishers, we seek to develop and publish the broadest range of learning and information products on each technology. Every Packt product delivers a specific learning pathway, broadly defined by the Series type. This structured approach enables you to select the pathway which best suits your knowledge level, learning style and task objectives.

Learning

As a new user, these step-by-step tutorial guides will give you all the practical skills necessary to become competent and efficient.

Beginner's Guide

Friendly, informal tutorials that provide a practical introduction using examples, activities, and challenges.

Essentials

Fast paced, concentrated introductions showing the quickest way to put the tool to work in the real world.

Cookbook

A collection of practical self-contained recipes that all users of the technology will find useful for building more powerful and reliable systems.

Blueprints

Guides you through the most common types of project you'll encounter, giving you end-to-end guidance on how to build your specific solution quickly and reliably.

Mastering

Take your skills to the next level with advanced tutorials that will give you confidence to master the tool's most powerful features.

Starting

Accessible to readers adopting the topic, these titles get you into the tool or technology so that you can become an effective user.

Progressing

Building on core skills you already have, these titles share solutions and expertise so you become a highly productive power user.