Tests in production can be risky. How do you run a cost/benefit analysis to decide
whether it’s worth the risk? How do you then lower the risk? Thoughts from
TurbineLabs based on Twitter’s testing experience:

Netflix largely tests in production, also with fault injection. The impact on end users is
“member pain” when they happen to be in the path of a failure test; Netflix aims
to minimize the pain, but considers it acceptable for a small percentage of users.
So next time when your show won’t play, consider that it might be not due to your
modem or even a bug, but the result of an intentional test!

If you want to avoid “member pain” and serve the known correct output to your users,
while simultaneously sending production requests down the new path, so you can
compare outputs of old (“control”) vs new (“experiment”) and alert yourself on mismatch,
try GitHub’s Scientist. The framework is a year old now and ported to multiple languages.

This recent post from The Guardian describes their production testing system which
triggers post-commit tests in prod and shows results as a badge on the GitHub pull request.
The prod tests cheat a bit by targeting the production deployment of the app, but
against a staging backend with test accounts.

Fun

If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at http://testersdigest.mehras.net