But according to NCrunch result HTML, it has 5 "fixtures" failing (incl. the one with clipped text), all on the S1 grid node (log states "Server used: S1").:(The "tests passed 14 920" is referring to NCrunch cache since since this test run only executed a few tests that were previously failing).And when looking at the "Show test output text" link, it shows output from an old test run (looking at the timestamp):

Code:

18:49:38.628 [7] DEBUG UnitTestHelper (null) - <Some old text referring to an old environment variable value no longer part of the builds>.

I.e. Is there a bug somewhere causing the "fixture failure state" not being cleared , if a test no longer is failing for whatever reason, if I use the NCrunch cache to only run failed or impacted tests?

So I see two problems here. The first is that we're reporting old cached data inside the HTML run report.

Normally, the run report should include old results under 'Show tests not run'. This is because when the engine loads up the solution, it immediately flags all tests as 'Out Of Date', so they show up with a question mark in the Tests Window. The HTML report considers the 'Out Of Date' flag when trying to decide which tests it should show, since old results don't really make much sense in this view. However, the use of the 'AlignOutOfDateStatusWithImpactStatus' configuration setting changes this behaviour. When this setting is enabled, the 'Out Of Date' flag will only get set when the test is actually impacted via code change. So I guess we shouldn't be relying on this flag when determining when to show a test result in the HTML report for the console tool. Maybe we need to use a different flag, or otherwise prevent the AlignOutOfDateStatusWithImpactStatus setting from working during the console tool run.

The second problem is related to the failure state of the fixtures. I think this is due to the combination of different pass/fail states that can exist at both fixture and test level, and probably the HTML reports this differently to the Tests Window because it doesn't aggregate child results in the same way. I'll need to look into this in more detail, but I don't think it's a major issue, because if we exclude the old cached results from the report, this issue likely won't have any impact on the actual results or UX.

This bite me again I think, but in a more trivial scenario:1) in commit A a test on TC failed due to timeout.2) I set category "long running" for the test in commit B (that category is excluded to run on TC).3) NCrunch properly reused cache from previous run (commit A), but since that test was failed in cache it seems to report that as failure even though it is now excluded due to engine mode (I assume).So NCrunch output says TestFailure but in order to get build green I must manually re-run the the build without NCrunch cache.

There's a fix coming for this in the next build. The change under the hood just forces the engine to set the AlignOutOfDateStatusWithImpactStatus setting to 'False' for all console runs, regardless of injected config settings. I can't think of any way this setting would be useful in the console run and it only causes problems with results reporting.

You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.