Contributors

Blog's

Technorati

spotplex

MadKast

Google Analytics

Friday, December 5, 2008

Testing in Layers

My old boss and mentor was chatting with me over Facebook recently about a concept that he termed, "testing in layers". It is the basic idea of slowly testing out a new release with people who are close to the code, then providing the release to people a little further away (friends/family/investors), then a little further (active members on your forums), while eventually moving out toward your normal everyday users. This idea struck me because it provided a simple mental model for actions that I think many in the testing community are already doing at a basic level, and provided much room for complexity if need be. It's nice to provide a concept around an existing process, so that it can be better codified and thought out.

In my world, I use this idea in the following context:

Test internally with engineering (QA)

Test internally with non-engineering (Marketing, etc)

Test by releasing to trusted users on our forums

Test by sending the release to particular people who email into Support, whose problems may be fixed with this release

Test by releasing to everyone on our forums

Test by releasing to a small number of people who download through the website

One of the issues with this idea is that a feedback loop needs to be in place at each layer. If we should release a build to the forums and don't listen to the complaints and issues that this layer is having, then this testing is pointless. However, the further out you get from the center, the harder it is to get solid feedback. When we release to a small number of people through the website, they probably don't know that they're getting a brand-spanking new release. In this case, how do we get their feedback? As of right now, we simply watch to see if they uninstall, and if the stars align, they might even leave us feedback. It's not perfect, and can use improvement, however so far it's a process that is working adequately to help us determine how the release is doing.

The idea of quality feedback loops for your product will be saved for another blog post. Lets just say that twitter has been proving quite useful in getting product feedback lately.

2 comments:

There is a lot of competition in outsourcing software development, as there are many firms across the globe catering to clients looking for outsourcing their work. What is good is that the takers can choose the best from the lot. http://www.infysolutions.com.