The Browser/OS Combination Problem

The Problem

You have N number of tests, O number of browsers and P number of operating systems. If you don’t think too much, it doesn’t sound that bad. Besides your using grid or Sauce Labs or something similar; how bad can it be. Then you do the math and start to realize the scale of the problem.

Number of tests

Execution time

Solutions

There are a couple of solution to this problem. The one that I've seen the most is to reduce the scope. Just test against fewer combinations. Second most common is to get a bigger grid. Both these solutions can work, but they both have drawbacks. The first solution obviously sacrifices your OS/browser coverage. The second is more costly, as it requires more nodes, and is much more likely to result in some serious bottleneck problems.

My preferred solution is to steal one of the strategies leveraged by "High-Volume Combination Testing". Combination testing often results in an impossibly large number of tests. This means doing some kind of paring down. The strategy we will be stealing is random sampling. This strategy is especially useful when you run your test frequently, such as part of CI. Over time you will get practically the same coverage as running all the combinations without nearly the infrastructure cost of greatly extending your grid.