We have some UI tests which we are running on grid nodes. The build runs a particular engine mode which runs tests in a specific category. This is working ok, but sometimes the tests fail for some random reasons, mainly due to selenium/UI testing infrastructure intermittent issues.

I'd like to be able to rerun the build but specify a list of tests that should be run, and not run them all in the category. What is the best way of handling this? Am I going to have to have a powersheel script which modifies the definition of the engine mode dynamically before the tests are run? Or does NCrunch support a way to do this out of the box?

Hey Remco, thanks for the quick reply as ever. I had considered this after playing with the Engine mode options a bit, but its not 100% clear to me which cache file will be in play here. The Ncrunch console tool gets invoked on the TC build agent. This is done with 0 threads, so the tests can't be run on that machine itself. These are then farmed out to the grid nodes to be run. When we run the test build again it may not happen on the same TC agent, but will be farmed out to the same collection of grid nodes. Obviously the individual tests may end up on different grid nodes.

So which cache file will be used to determine if the tests failed or not? it seems to me like there could be problems either way. if its the one on the agent then different agent = diffferent cache files, and so wrong tests might be run. If its on the grid nodes themselves then what happens if tests end up on different nodes?

The cache file is usually stored on the machine running NCrunch.exe. This file won't be stored on the grid nodes, although they do have cached data of a different kind (not relevant in this scenario).

The contents of the cache file can be quite important when using the NCrunch console tool in the build system, because NCrunch will use the data here to form a picture of what has changed in the solution since the last time the tests were run. This means that if you're running NCrunch.exe on different TC build agents, the cache file won't be consistent between runs and your build will function less efficiently. It does also mean that my suggestion wouldn't work, because if the runs were being handled by different build agents then the list of failed tests might not be available on the second run.

So you probably have a couple of options here. Either you can restrict the running of NCrunch.exe so it only happens on one of your build agents, or you can modify the storage directory of the NCrunch .cache file so that it lands on a shared network drive that is used by all the build agents. Note that sharing the cache file between build agents does give a slight chance of concurrency issues.

I solved this problem by collecting the failed tests via the TC HTTP API in a powershell script and then parsing out the names and creating a piece of text which matches NCrunchs definition for running specific tests like so:

Then trigger the build that failed again passing this string in via the HTTP API as a build parameter. Then in the build I check if this parametger has a value and if it does then I edit the ncrunch.v3.solutionconfig file to replace the <Settings> <TestsToExecuteAutomatically> to have this text, and then NCrunch only executes those tests.

I solved this problem by collecting the failed tests via the TC HTTP API in a powershell script and then parsing out the names and creating a piece of text which matches NCrunchs definition for running specific tests like so:

Then trigger the build that failed again passing this string in via the HTTP API as a build parameter. Then in the build I check if this parametger has a value and if it does then I edit the ncrunch.v3.solutionconfig file to replace the <Settings> <TestsToExecuteAutomatically> to have this text, and then NCrunch only executes those tests.

bit hacky, but it works. Or seems to superficially :)

Brilliant! If I were to give an award for the most creative solution to an NCrunch related problem, you've definitely taken it for this year.

FWIW, since you use TeamCity. I store the NCrunch cache directory as an artifact for the build. In the following build the first step will download that artifact (for same branch, if available) via TC API in order to utilize "impacted or failed tests only" engine mode upon next build.

You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.