Test Case, Test Suite, Test Run, what’s the difference?

If you're a newcomer in the world of testing, you easily become overwhelmed by all the testing terminology getting thrown around. In this article, we'll try to explain some of the jargon you might encounter in your daily testing life, just to make it all a little bit more understandable.

Test Cases

So, we’ll start with test cases. Most importantly, test cases provide a series of instructions that validates some part of the software is doing it job as expected. Say, you require a contact form on your website that sends an e-mail when a visitor leaves a message, you’ll probably create a test case verifying the e-mail actually arrives when the form is filled and submitted. A test case can help you determine whether or not the software meets your requirements.

In some situations, a single test case can verify a simple requirement. However, it is more common practice to think of multiple situations that tests your feature more thoroughly. When your contact form offers an optional text field, try both filling and clearing the field in two separate test cases. Even better: try to go beyond the “happy path” and think of things the software shouldn’t do. If your contact form accepts an e-mail address, why not check if a non-valid address is accepted too?

Test cases do not merely consist out of instructions – or test steps, as we’d like to call them. For example: simply performing a few steps without explicitly telling the excepted outcome could cause misunderstandings. Should our contact form rejected an invalid e-mail address? Perhaps – or perhaps, we really don’t care. A proper test case usually comes with a predefined expected result, telling you what to expect when completing a test case. Similarly, it is common good practice to define the initial expected state when writing a test case. You don’t want to cover every piece of preparation in your test steps, like accessing a test environment, setting up a browser, enabling mail notifications, etc. These pre-conditions are usually defined as a separate property of the test case. When a condition isn’t met, there’s no point in testing the test case.

So, that’s it then? Well, there’s more. Like writing a blog post, creating test cases require certain writing skills, like imagining things from the end-user’s perspective. If you want to read more, be sure to read 7 tips for great test cases.

Test Suites

When you know about test cases, the concept of test suites should be really simple. With a growing amount of test cases, the need for categorizing them increases as well. You don’t want to throw all your books on a pile, you’ll organize them on shelves, for better accessibility. In a way, test suites are very similar to shelves, as they group test cases together as well.

In the process of planning and running tests, keeping track of hundreds of test cases really becomes cumbersome. Thinking in terms of “We need to run a performance test tomorrow” or “Next week we should schedule a test for our website” forces you to know the purpose and domain of each test case. A test suite allows you categorize test cases in such a way that they match your planning and analysis needs. Do you run functional and performance tests? Create two suites and label them accordingly. Do you test different applications and want to monitor them? Create a suite for every application. Are you still able to manage without test suites? Don’t use them! Remember, they are only there to help you organize test cases when needed.

Test Runs

Finally, we end up with test runs. They are little crossroads in your testing project, where several aspects come together. Simply put, test runs determine which test cases are tested by which user(s) at what particular time. Quite a mouthful!

Image you have created your test cases for your contact form and its about to become available to your organization for testing. Are you going to test it for yourself, or do you pick someone else? When are you going to start testing – and, when do you want to gather the results? Which test cases are you going to select – all of them, or just the ones marked “happy flow”? These questions should be answered when planning a test run.

When you have created a test run, you have reached an important milestone: people are able to run a test, using your test cases, grouped by your test suites. In one of our next posts, we’ll continue with some useful tips running tests and gathering test results.