Xray JSON results

When importing execution results using Xray JSON result format in a Continuous Integration environment, you can specify which Test Execution issue to import the results on using the "testExecutionKey" property. Alternatively, you can create a new Test Execution for the execution results and specify the Test Execution issue fields in the "info" object.

Xray JSON results Multipart

Xray provides another endpoint if you want to create or update a Test Executions and have control over all Test Execution fields. It allows you to send two JSON files, the normal Xray JSON result and a JSON similar to the one Jira uses to create/update issues. For more information about that second format, check the Jira documentation here. Note that in this endpoint the info property in the Xray Json result part will be ignored.

Cucumber JSON results Multipart

Xray provides another endpoint if you want to create new Test Executions and have control over newly-created Test Execution fields. It allows you to send two JSON files, the normal Cucumber result JSON and a JSON similar to the one Jira uses to create new issues. For more information about that second format, check the Jira documentation here.

It's possible to assign Test Environment(s) to the newly-created Test Execution. For that, you need to pass the ID of the custom field corresponding to the "Test Environments" custom field. In the JSON example below, it is 10030 for the info object.

Note: Currently, if you specify the Test Plan custom field, the Tests of the Test Execution will not be added automatically to the Test Plan.

Behave JSON results Multipart

Xray provides another endpoint if you want to create new Test Executions and have control over newly-created Test Execution fields. It allows you to send two JSON files, the normal Behave's result JSON and a JSON similar to the one Jira uses to create new issues. For more information about that second format, check the Jira documentation here.

POST /rest/raven/1.0/import/execution/behave/multipart

Import the execution results created with the Behave JSON output formatter.

Note: Currently, if you specify the Test Plan custom field, the Tests of the Test Execution will not be added automatically to the Test Plan.

JUnit XML results Multipart

Xray provides another endpoint if you want to create new Test Executions and have control over newly-created Test Execution fields. It allows you to send one XML file (the JUnit report) and a JSON similar to the one Jira uses to create new issues. For more information about that second format, check the Jira documentation here.

POST /rest/raven/1.0/import/execution/junit/multipart

Import the execution results created with the JUnit XML output formatter. For more information, please check the documentation about JUnit integration.

Note: Currently, if you specify the Test Plan custom field, the Tests of the Test Execution will not be added automatically to the Test Plan.

TestNG XML results Multipart

Xray provides another endpoint if you want to create new Test Executions and have control over newly-created Test Execution fields. It allows you to send one XML file (the TestNG report) and a JSON similar to the one Jira uses to create new issues. For more information about that second format, check the Jira documentation here.

POST /rest/raven/1.0/import/execution/testng/multipart

Import the execution results created with the NUnit XML output formatter. For more information please check the documentation about TestNG integration.

Note: Currently, if you specify the Test Plan custom field, the Tests of the Test Execution will not be added automatically to the Test Plan.

NUnit XML results Multipart

Xray provides another endpoint if you want to create new Test Executions and have control over newly-created Test Execution fields. It allows you to send one XML file (the NUnit report) and a JSON similar to the one Jira uses to create new issues. For more information about that second format, check the Jira documentation here.

POST /rest/raven/1.0/import/execution/nunit/multipart

Import the execution results created with the NUnit XML output formatter. For more information please check the documentation about NUnit integration.

Note: Currently, if you specify the Test Plan custom field, the Tests of the Test Execution will not be added automatically to the Test Plan.

Robot Framework XML results Multipart

Xray provides another endpoint if you want to create new Test Executions and have control over newly-created Test Execution fields. It allows you to send one XML file (the Robot report) and a JSON similar to the one Jira uses to create new issues. For more information about that second format, check the Jira documentation here.

POST /rest/raven/1.0/import/execution/robot/multipart

Imports the execution results from Robot Framework XML output format. For more information please check the documentation about

Note: Currently, if you specify the Test Plan custom field, the Tests of the Test Execution will not be added automatically to the Test Plan.

Multiple Execution Results

In order to import multiple execution results (e.g., outputted from Calabash or Xamarin Test Cloud), you must import the bundled compressed file with multiple execution results to Jira using the following endpoint:

How results are mapped to Test entities

Whenever importing results from some frameworks (i.e. JUnit, TestNG, NUnit, Robot framework), Xray can identify the automated test from the report/results file, based on an hardcoded criteria (such as the class name plus the class method corresponding to the automated test). However, depending on the test automation framework, it's possible to specify the Test issue key to which report the results in the test's code.

Independently of whether the test is identified implicitly (based on some attributes present in the test result file) or explicitly (based on the provided Test key), related Test Runs are always reported against the correct Test issue. As a consequence, if you report results multiple times there won't be duplicated Test entities.

When the identification is implicit, Xray is able to create (Generic) Test entities, if needed, per each automated test; these will be reused afterwards in similar cases.

Whenever processing results from a automation framework, for each automated test result,

If the Test key is provided and...

it exists, then create a Test Run for that Test

it doesn't exist, then don't create any Test Run (since for some reason the explicitly identified Test does not exist)

if no Test key is provided...

try to find a Test in the identified project with the same Generic Test Definition (e.g. with the same class name+class method for example)

if it exists, then create a Test Run for that Test

if it doesn't exist, then search for a Test with the same Generic Test Definition in all JIRA projects

if it exists, then create a Test Run for that Test

it doesn't exist, then create a Test in the identified project (based on the project's key or the project associated with the provided test execution key)

For some frameworks, including Cucumber and Behave, Tests must exist previously to the submission of results related to them.

The reason for it,resides mainly in the fact that is not possible to create the complete Test specification from the results file.