Using Structure vs Test Repository/Test Plan Board

Since Xray v3.0, you can use the Test Repository to organize Tests hierarchically in folders at the project level; you may also organize Tests, in an independent way, at Test Plan level, for each Test Plan.

The question arises: which approach should you use? Organize entities using Structure add-on or using Xray built-in capabilities?

Let's see the different and summarize the characteristics of each approach, focused on the related entities.

Installation

Note: If you already had Xray before you installed Structure, you need to restart your Jira server.

For more information about the Structure plugin, refer to its documentation here.

Adding Xray Extender to the Structure:

1. Set your structure to Automation

2. Choose the correct extender

3. Choose the options to enable, including Xray specific actions, and click "Apply" on the dialog

You should see a new entry in the structure, similar to this one:

Settings

Whenever adding the Xray Extender or whenever editing an already existing one (by double clicking on it), you may have several Xray Actions that are in fact options that you can enable:

Manage Test Executions under Test Plans: allows you to see related Test Executions underneath Test Plans, at first level, and manage their association by adding/removing Test Executions to/from the Test Plan

Synchronize Tests between Test Execution and related Test Plan(s): whenever adding Tests from Test Executions, it will automatically add Tests to the related Test Plans; on the other hand, whenever removing Tests from Test Executions, if those Tests are not part of other Test Executions linked to the same Test Plan(s) (i.e. if the Tests are not "already being used"), they will be removed from the Test Plan(s)

Manage Tests under Test Plans: allows you to see related Tests underneath Test Plans, at first level, and manage their association by adding/removing Tests to/from the Test Plan

Recommended Setups

The way you use Structure will vary depending on your team needs.

Within this section you may see some possible setup scenarios.

Scenario 1

The following example focus on these needs:

quickly showing, on a per version, the planned testing (i.e. Test Plans and related Test Executions)

track the overall progress of Test Plans and Test Executions and also the coverage status of "requirements"

Actions

Xray provides specific actions that will interact and integrate with changes made on Structure. These actions must be invoked explicitly.

Note

When moving a Test to different issue types, the Test will not be removed from source issue.

Example: When moving Test A from Test Set B into Test Execution C, the Test will be added to Test Execution C, but not removed from Test Set B.

Please note

Whenever doing action using Structure, you may be prompted to perform one of multiple available actions. Please choose the proper action or else you may not obtain what you were expecting for.

For example, if you add a Test Execution underneath a Test Plan, or if you add some Tests to a Test Plan, you may be prompted with a dialog similar to the following one. If you are aiming to perform Xray related actions, please choose the one containing the Xray icon. If you choose "Move within ...", then the item will be moved visually within the structure but no interaction will be made with Xray.

Pre-Condition

With the Extender added, you can now do the following actions:

Manage Tests of the Pre-Condition:

Drag a Pre-Condition to a structure and have its associated Tests dynamically appear;

Drag a Test to a Pre-Condition and have it automatically associated to the Pre-Condition. When you drag a Test to a Pre-Condition, a message will appear as displayed in the image below. You should always choose the Xray "Add" action since it's the option that will use the Xray Extender;

Drag a Test Set to a Pre-Condition and have all the Tests in the Test Set automatically associated to the Pre-Condition. You will get the same message as above;

Remove a Test from a Pre-Condition and have the association automatically removed from the Pre-Condition. When you click on the X and you have a Test selected, the Test will be removed from the Pre-Condition;

Test Set

With the Extender added, you can now do the following actions:

Drag a Test Set to a structure and have its Tests dynamically appear.

Drag a Test to a Test Set and have it automatically associated to the Test Set.When you drag a Test to a Test Set, a message will appear. You should always choose the "Add" Xray action since it's the option that will use the Xray Extender.

Drag a Test Set to a Test Set and have all the Tests in the Test Set automatically associated to the other Test Set. When you drag a Test Set to another Test Set, a message will appear. You should always choose the "Add" Xray action since it's the option that will use the Xray Extender..

Remove a Test from a Test Set and have the association automatically removed from the Test Set.When you click on the X and you have a Test selected, the Test will be removed from the Test Set. If you remove multiple Tests, an additional message may be shown that you should accept.

Test Execution

With the Extender added, you can now do the following actions:

Manage Tests of the Test Execution:

Drag a Test Execution to a structure and have its Tests dynamically appear. You may also choose to make the related Test Executions appear; these behaviours depend on the options chosen in the Extender;

Drag a Test to a Test Execution and have it automatically associated to the Test Execution.When you drag a Test to a Test Execution, a message will appear. You should always choose the Xray "Add" action since it's the option that will use the Xray Extender;

Drag a Test Set to a Test Execution and have all the Tests in the Test Set automatically associated to the other Test Execution. You will get the same message as above;

Remove a Test from a Test Execution and have the association automatically removed from the Test Execution. When you click on the X and you have a Test selected, the Test will be removed from the Test Execution.

Indirectly manage the Tests of the related Test Plans (you must enable a a specific option in the Extender of this):

Drag a Test/Test Set to a Test Execution and have the Test, or related Tests, automatically associated to all Test Plans linked to that Test Execution;

Remove a Test from a Test Execution and have the association automatically removed also from the Test Plan, if that Test is not part of any other Test Execution linked to the same Test Plan.

Sub-Test Execution

You may perform similar actions to the Test Execution. Please see above.

Test Plan

With the Extender added, you can now do the following actions:

Manage Tests of the Test Plan:

Drag a Test Plan to a structure and have its Tests dynamically appear;

Drag a Test to a Test Plan and have it automatically associated to the Test Plan. When you drag a Test to a Test Plan, a message will appear. You should always choose the Xray "Add" action since it's the option that will use the Xray Extender;

Drag a Test Set to a Test Plan and have all the Tests in the Test Set automatically associated to the other Test Plan. You will get the same message as above;

Remove a Test from a Test Plan and have the association automatically removed from the Test Plan. When you click on the X and you have a Test selected, the Test will be removed from the Test Plan.

Manage the Test Executions under the Test Plan (you need to enable a specific option in the Extender for this):

Drag a Test Execution to a Test Plan and have it automatically associated to the Test Plan;

Remove a Test Execution from a Test Plan and have the association automatically removed from the Test Plan.

Other features

Association between Tests and requirements

Since the association between Tests and requirements is done using the "Tests" link, you can take advantage of the built-in Structure extender for dealing with linked issues and therefore, manage this association.

For this, you must have "Automation" enabled. Choose "Extend with..." and then "Linked issues...".

Then, choose the "Tests" link type.

As soon as you do this, you're able to see the association between Tests and requirements, as shown below. You can see the story being validated by the manual Test.

You can also link your Test with other requirements just by dragging them under the Test and choose the action that will create the proper issue link between the Test and the requirement.

Note

If you want, you can also manage the association from the opposite end, by looking from the requirement's perspective.

This will allow you to see and manage the Tests that validate a given requirement.

For this, you just need to create an extender for the "Tested by" link type.

Association between Sub-Test Executions and requirements

Sub-Test Executions are created as sub-tasks for requirement issues. You can take advantage of the built-in Structure extender for dealing with sub-tasks and therefore, see this association.

For this, you must have "Automation" enabled. Choose "Extend with..." and then "Sub-tasks...", and then the Sub-Test Execution issue type.

Afterwards, you're able to see all the Sub-Test Executions created in the context of a requirement issue that is part of the structure.

Show requirement and test statuses

You can include information about the status of your "requirements" and its current state, along with the latest status for the corresponding Tests, in the issues being shown in the structure.

For this, you need to add the "Requirement Status" and the "TestRunStatus" columns, which correspond to the "Requirement Status" custom field of requirements and the "TestRunStatus" custom field that is available for Test issues, respectively.

Show progress information of Test Executions and Test Plans

You can include information about the progress (i.e. the "overall execution status") of (Sub) Test Executions or Test Plans. For this, you need to include the proper custom fields: "Test Execution Satus" and "Test Plan Status", respectively. More information on these fields can be found here.

Note: you need to have the latest version of Structure installed.

Show status overview of Tests contained within Test Sets

You can include information about the status of the Tests belonging to a Test Set, by including the "Test Set Status" custom field as a column. Although Test Sets are not related with execution aspects, sometimes it may be quite useful to have an idea how certain groups of Tests (i.e. the ones that you have in some Test Set) are.

Please make sure this custom field is configured properly in Xray (under "Custom Field Preferences" admin settings), to show the status of Tests for the version you want to.