Guidance

Test Vector Generation Failures

If test vector failures occur there are some general guidelines that apply to any type of model (TTM or Simulink). See VGS Issues and Guidelines.

S-Functions

S-Functions allow users to specify the semantics of new blocks for simulation and code generation through several programming languages, including MATLAB, C, C++, Ada, and Fortran. While the blocks’ semantics are necessary to support test generation for subsystems that reference the S-Function, these semantics are not accessible by the translator through the standard approach.

To support translation of S-Functions, the translator allows for the substitution of a Simulink subsystem in place of a S-Function. The subsystem that replaces the S-Function describes the functionality of the S-Function in terms of primitive Simulink blocks. The Simulink subsystem to be substituted for the S-Function is specified by entering a specially formated tag into the comment of the S-Function block.

Instructions for formating the S-Function substitution tag are provided in the Simulink Tester User's Guide.

Limitations

Simulation Test Drivers: State Data Initialization

T-VEC produces test vectors that may contain state data for feedback constructs such as the unit delay. This results in state data, but there is currently no control mechanisms in the Matlab simulator to set (initialize) state data.

Issues

Inlining and Coverage

Use of inlining Simulink subsystems can impact test and model coverage.
If a model includes inlined subsystems, then the missing coverage
may be related to situations where the model doesn't support taking one of
the paths through the utilities (lower-level subsystem) based simply on the model's design.

It is the very nature of design models vs requirements models. Designs include
re-usable/generic components whose logic won't necessarily be needed for
all applications.

Here's an example:
Subsystem X is a utility that has an input A with two paths:

A >= 0 then do something

A < 0 then do something

Subsystem Y inlines subsystem X, but has references to Subsystem X only when the input associated with the signal A is >= 0. This means that there cannot be model or test coverage for the path when A < 0.

The only solution is to make sure they are treated as stand-alone atomic functions so that they may be tested in their own right.

Test Vector Generation Failures

T-VEC VGS analysis is exhaustive in the sense that for every DCP throughout the hierarchy of systems is model-checked, and a test vector is then produced. Most test vector generation failures result from model defects where the constraints of the DCP are not satisfiable throughout the hierarchy of DCPs of the subsystems.

However, there are possible situations where test vector generation fails when it is possible to find vectors. For example, when signal domains are left to their defaults. There are default settings for ranges values, but these may not be adequate for the application.

T-VEC does cover a large percentage of models that use standard blocks (see View | Library... from Simulink Tester GUI for the supported blocks), however, there are other situations where T-VEC may not produce vectors, for example

The Auto Rotate Convergence Fix Mode Test Vector Generation property is one of the mechanisms that can address some of these issues, and other internal mechanisms are applied automatically, independent of the specific convergence mode. The T-VEC test generation algorithms continue to evolve to deal with classes of these kind of situations, but there is no guarantee that such situations will not occur. For such situation contact support@t-vec.com for additional assistance.

Signal Range Management

Changes in architectural decomposition may cause a signal name to be removed. For example, if a signal is defined as signal A in the hierarchical scope of subsystems x\y\z and subsystem y is removed, then that signal is undefined and is removed from the signal range file (with extension .srf).

The translator creates a backup copy of the signal range file (.srf) each time the translator is run. It creates a log file that details every change to the .srf file. It should be possible to recover from an unexpected change to the .srf file.

Model Coverage vs. Code Coverage

Test vector generation attempts to find test vectors for all of the paths (DCPs) in the model. Complete coverage means complete model coverage, however, complete model coverage does not guarantee coverage of code (e.g., MC/DC coverage), for example: code generators can put in code for checking and handling exceptional conditions (e.g., safety code around square root function). The Simulink Testers provides mechanisms such as assertions and coverage predicates that can be used to produce additional tests to cover many possible situations. Tools such as LDRA TBrun is integrated with the Simulink Tester and can provide test coverage measures to support organizations that may need evidence to support FAA certification.

Test Sequence Vectors: State Variable Initialization

Test Sequence Vectors (TSVs) are created by referencing the target subsystem multiple times.
Special T-VEC subsystems are created based on a configuration file supplied to the translator. These TSV subsystems include one or more test sequences. Each test sequence
includes one or more references to the target subsystem (i.e., the number of sample times
being tested). It includes also a set of the target subsystem’s inputs for each time that the
target subsystem is referenced. These input values have their respective subsystem reference
appended to their name. Therefore, if a subsystem has two inputs, inputA and inputB,
then a TSV subsystem that includes two references would include the inputs inputA_1,
inputB_1, inputA_2 and inputB_2. Input values to each subsystem reference can be
controlled through settings at the reference level, test sequence level or subsystem level.

Many blocks in Simulink have specific initial condition values, unit delays, integrators, etc. (and stateflow in terms of things like initial states of state machines). T-VEC has a mechanism for using these initial values during vector generation, and they include:

T=0 - defines a point in time where all test vectors are generated as if the system is in the first cycle of execution

T>=0 - test vectors are generated for things that take place after the first cycle and then continue for multiple cycles of the TSV

T>0 - test vectors where the initial condition setting action is disabled and this forces the state variables to be solved for through convergence rather than be initilized or assigned

Ignore

The default is to generate vectors for both cases, 2 vectors per DCP when no state variables are available, and 4 test vectors when state variables are available. This mechanism should be adequate to test control and logic associated with state data (See State Variable Inits Example).

Test Driver Generation

There are some situations where the object mapping information that relates a signal name to an implementation name must be defined manually to support test driver generation. This is applicable to a few types of Simulink blocks and some Stateflow modeling constructs. There are cases where it is simply not possible to predict variable names, function parameter orders, etc. There is no API for determining the choices made by the RTW to fully automate all aspects of test driver map file creation.

There is a work around with the current schemas - all mapping descriptors are passed through a Perl function before going into the target file. The perl script parses in an external file that contains new replacement mapping information. The mapping can be defined with the descriptor or a mapping key. If the perl script finds a match then it uses the replacement from the file. This function can be extended directly through the Perl script.