"A/B Testing" is putting up two scenarios ("Scenario A" and "Scenario B"), and then measuring the impact of the changes by testing against a control. This is quite common in other areas of science, but less so in software development.

Rowan Simpson and Sam Morgan from Trade Me both show examples of the successes Trade Me has had when doing A/B testing, and they have developed systems to perform A/B tests, such as running one version of feature for one half of their users, and the new feature for the other half.

Learning from your users is very important. They are the best people to ask about your product. Unfortunately, when you ask them, you get different results than when you observe them.