Summary

Software testing is essential for compliance with DO-178C standards and the software requirements, and ultimately for eliminating software errors. Choosing both the right approach and tools will impact cost, schedule, quality and risk. Although Unit testing is commonly used as an informal means of testing in the development phase, the purpose of this paper is to discuss when unit testing should be used for formal credit in DO-178C projects and what advantages this approach offers.

What is Unit Testing?

First a few basic definitions; A software unit is an individual program, often referred to as a function, method, procedure or subroutine, which performs a specific operation. Unit testing is the method of isolating and testing individual software units. Under a unit testing approach, each software unit is exercised using a test driver that provides different combinations of inputs.

The test driver replaces any dependencies on other units and external data objects with test stubs. These stubs provide the data that would have normally been returned from a call or read from a data variable or structure and are necessary to test each test case and each code path.

A test driver is a software routine created by the test engineer to exercise the unit under test, providing specific input values and collecting actual results after each invocation to be compared to the expected results. In the C programming language, a test driver can be seen as a main() routine that takes care of setting input conditions, calling the function under test and storing the results in a buffer.

The test driver together with the test stubs and other related testing objects is referred to as the test harness. The test harness is developed by the test engineer, then compiled and linked together with the code under test to create an executable version of the test that can be run in the target hardware or simulator.

Unit Testing and DO-178C

Because developing software involves multiple levels of abstraction from requirements–from design descriptions and models, to source code and finally to the executable object code --there are multiple opportunities for errors to be introduced along the way. A designer could induce an error in a model by misinterpreting a requirement. A programmer could induce an error in the code by misinterpreting a model or compiler, or linker that could generate erroneous object code from the source code.

Of course we could do extensive testing at each and every level to attempt to catch the errors, but that would have a severe impact on cost and schedule and is it really necessary? Why not focus instead on testing the final output (the executable code) against the initial input (the software requirements) and save a lot of time and money.

Actually, that is exactly what the DO-178C standard suggests that we try to do. And, testing works great for high level requirements that describe external behaviors in a black-box fashion on the target. However, the problem is that there may be just as many low level requirements that describe the internal workings of the software and the inputs, outputs and conditions are not specified in terms that can be described or tested externally. In order to test these low level requirements, we need to understand the code, and we may need to isolate, instrument and manipulate it in order to force it to execute each test case and attain sufficient structural coverage. Unit testing gives us the ability to do just that.

The Advantages of Unit Testing

Although other approaches could be employed to test low level requirements, Unit testing offers some definite advantages described below:

1) Reduce Ramp Time

Since a unit represents only a small amount of functionality, the ramp time required to understand it is minimal and the tester can begin producing tests immediately. Likewise, since testing a unit doesn’t require interfacing with a multitude of system-specific interfaces, simulators and other test equipment, the tool used is relatively simple and can be used from project to project, regardless of what type of system is being tested. This results in more time savings for the unit test team.

2) Reduce Risk

Without a means to measure work progress, there is no way to track progress and therefore no way to control the risk of schedule and budget overruns. And if the means of tracking is tedious and time consuming, it may not be effective, particularly with a large or complex software system. Because the testing of a unit is generally a function of the number of SLOC decisions, inputs and outputs, these metrics can be tabulated for all functions and used to budget and track both team performance and individual performance easily and accurately.

3) Test Flexibility

Unlike an emulator debugger/script that is specific to a family of processors, a unit test tool can be used for any type of system or processor. And unlike an emulator debugger/script, a single change in the code doesn’t affect other unit tests since they are all isolated for testing. Conversely, an approach using breakpoints on line numbers can be a maintenance nightmare because a single code change in one unit can invalidate many if not all of the breakpoints in all the unit tests, not just the one that the code changed in. As with emulator/debugger testing, unit testing allows injecting errors or invalid inputs to test robustness that typically cannot be done at the black box level.

4) Attain Structural Coverage

Depending on the design assurance level, DO-178C requires that tests generate certain levels of structural coverage. However, some sections of the code may not execute under normal operation, and so a fault or other abnormal condition must be injected to attain sufficient structural coverage, which can be easily accomplished in a unit test harness.

5) Reduced Cost Through Test Automation

Since unit testing involves the development of more software that can be executed just like the software being tested, execution of the unit tests are fully automated and can be run repeatedly with minimal human interaction required, and thus virtually eliminating the chance of introducing human error during running of the tests. An emulator or debugger without scripting capabilities will not provide this advantage.

Additionally, because the method for passing inputs and outputs to the UUT and stubbing external units that are called is consistent from one unit to the other, the generation of the test harness can also be automated so that the test engineer only needs to define the test cases for each unit (inputs, initial conditions, expected outputs and calls to be stubbed). The rest is automatic.

Need Help?

Avionyx, the leading provider of embedded software engineering solutions, develops multiple unit test tools to meet specific customer needs, in some cases generating the test harnesses automatically from the test cases. If we can assist you with your testing approach on your next DO-178C project, please contact us.