Details

Description

We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner.

With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported as failures anymore. Even worse, they are reported as success!

Activity

Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly

Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)

Torben Knerr
added a comment - 2018-08-22 11:34 Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly
Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)

Nikolas Falco
added a comment - 2018-08-23 09:50 Anyway let me have a look if result Error is a valid kind in NUnit2 and in case handle properly or it is a status in NUnit3 that is handled in wrong manner in the converter.
I suggest if you have NUni3 to do not convert and use the tool type set to NUnit 3.x

Nikolas Falco
added a comment - 2018-08-23 10:01 "my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted"
This is a bug of adapter https://github.com/nunit/nunit-v2-result-writer/issues/11

Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3.

Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.

Torben Knerr
added a comment - 2018-08-23 10:44 - edited Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3.
Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.