Monday, November 11, 2013

Some years back I attended a talk on testing practices where
the presenter asked the audience how much test automation have they achieved in
their projects. I was the only one who answered almost 100%. He advised me to
keep this a secret from my management if I did not want my team to be downsized :). Good
advice, but that got me thinking whether test automation actually makes testers
redundant.

Having spent a good part of my experience in testing, I have
seen this discipline mature over the years.My opinion is that automation does not take away tester jobs; on the
contrary it makes the job more interesting and effective.

In my mind I classify automation progression in an
organization into 4 stages as shown below.

Stage 1

In stage 1, testing is a completely manual activity. Tester
responsibilities are writing test case documents based on the requirements and
functional specification, executing tests manually following the documents, and
creating bug reports for any deviations. As the product grows in size and
functionality, more test cases are added and more resources will be needed to
manually execute them. Sounds like a good recipe for growing your test team?
Well, not exactly. This model will not make either your test team or management
happy. Here are a few reasons why:

Manual test execution is
error prone: Human beings are prone to make mistakes especially when they
are doing a job that is repetitive and boring. Some invalid bugs may be
raised or even worse, some valid bugs may be missed due to errors in the
manual execution.

Impossible to test
everything manually: Testing is all about simulating conditions that are as
close as possible to real production behavior. For e.g. consider testing a website for ticket
booking. Some of the typical uses cases will be hundreds of users
accessing the website at the same time, 2 or more users trying to book the
same flight etc. How do you test these manually? You cannot possibly line
up hundreds of testers and ask them to access the website at the sound of
a bell.

Manual testing cannot
scale: With each release of the product new features are added and number
of test cases will grow exponentially. It is not possible to add testers
at the same pace and as a result the time taken for test execution grows
out of control.

Unhappy testers: A good
tester is a creative person who loves to explore new features and new
methods to break existing and new functionality. In stage 1 they get
trapped in endless cycles of manual repetitive regression test runs. This
makes good testers unhappy and they will not stick to the job for long.

Unhappy Management: With a
large test suite regression testing will take several days or weeks. It
almost becomes impossible to deliver a product on time. This will make the
management unhappy with their test team.

These problems will force all organizations to move to stage
2 sooner or later.

Stage 2

Automation can be introduced when the product has stabilized
and does not have frequent changes in functionality. The first step in automation
is the identification of an appropriate automation tool. There are several
tools available in the market and an appropriate one has to be chosen based on
the needs of the project. Very often none of them meet all the requirements and
you may have to build an in-house automation tool. For most of the projects that
I have worked on we have developed our own tools.

In the initial phase the cost of automation will be more
than manual test execution. The organization would have built up a huge backlog
of test cases over some time and these cannot be automated overnight.
Automation itself will be slow until the testers gain sufficient experience in
the tool and framework. More over until a good percentage of tests are automated
manual test execution has to continue in parallel with the automation activity.
The real benefits start showing only in the long run when a good percentage of
tests are automated. It is important that the management support the testing
organization in this ramp up phase.

As the experience with tools and techniques increases
automation starts getting pushed into earlier stages of the development process.
Today we have reached a state where it is possible to develop automated tests
in parallel with the product so that testing can start as soon as the product
is ready. In most projects I have worked on in the recent years we have
achieved close to 100% test automation.

Stage 3

In this stage the organization starts thinking about how to
bring in more automation. The tests are automated, but the test runs still had
to be triggered manually for each build. Many days passed between test
execution cycles and lots of new code was added in this time. As a result with
each test run new problems are discovered. This is a very inefficient way of
finding and fixing problems.

Solution to this comes in the form of continuous
integration testing (CIT) tools. These can trigger test runs automatically
at a frequency that is configurable. There are many CIT tools in the market. We
are using Hudson which is a very popular
open source tool.

With the availability of CIT tools, there will be a
temptation to run all the tests for every check in. Remember that this also
comes at a cost. You need hardware to run tests and running everything
everywhere always might be overkill. In our team we have followed a tiered
approach. A small suite that can finish
execution in less than an hour is run for every check-in, a larger suite that
finishes in 8 hours is run every night and the complete suite is run every
week. The weekly test can run for 24 hours all 7 days of the week. The outcome is
that bugs are discovered as soon as they are introduced and it became easy to
isolate the root cause and fix. As a result we reduced not only the testing
time, but also the time taken to find and fix bugs.

Stage 4

Now all tests are automated and they are running without any
manual intervention. Is it time to fire the testers? No, we still need someone
to analyze the test results. This takes us to the next level, which is
automated test failure analysis. A signature is identified for each failure and
is associated with the corresponding bug in the bug database. Scripts are put
in place to compare the failure with the known signatures and tag with the
matching bug. Now we have reached the state where no manual intervention is
needed for test execution or failure analysis.

Conclusion

What do the testers do if everything is automated?They do what they should really do. Spend
their time on creative tasks like designing test cases for new features,
exploring new ways of testing the product, improving test coverage etc. The
repetitive task of manual test execution is best left to the machines :).

To conclude, automation does not make testers redundant,
rather it frees up their time to be spent on more interesting and useful tasks.
A good management will identify the value provided by the testing team and will
never consider reducing resources just because tests are automated. Testers are
happy in a completely automated regime because their time is spent on test
automation and not in repetitive manual test execution. I would say it is a
win-win situation for testers and management.

Tuesday, September 10, 2013

We had an absolutely rocking July this year when
we inducted new college graduates (NCGs) into the MySQL team at Oracle,
Bangalore.

There were fun themes and games that went
hand-in-hand with the training program bringing about a vibrant atmosphere
that gave the NCGs a chance to prove that they are indeed a great addition to this high energy
MySQL team here in Bangalore :).

The fun part of the induction spanned five
Tuesdays and culminated with the grand finale in which we were bowled over by some
brilliant performances by the NCGs. What I loved about these Tuesdays was that it jump started the induction programme bringing about
some great team building and a very informal introduction to our MySQL team.

On the first Tuesday, the NCGs who were expecting
a conventional welcome party were in for a surprise as they were asked to put
up an impromptu talent show. After the initial inhibitions and reluctance, they
came out with an entertaining show that included rounds of songs, dances, an
artistic sketch and even enactment of scenes from Bollywood movies. We also had
a dress code: wrap-around skirts for women and shorts for the men. While some
folks shied away from the dress code, many enthusiastic members including MySQL
India Director, Sanjay, came dressed for the occasion.

The remaining Tuesdays made for very interesting
dress themes such as color-coordinated theme, masquerade party, and scarf/hat
for women and tie/cap for men. The MySQL team was split into six groups, each
comprising of an equal number of new recruits and existing members that were
pitted against each other. We played games like "Whose Line Is It Anyway," "Human
Knot," "Guess Who,"
and more. This brought in some good natured rivalry and competition between the
teams that served as a great ice-breaker.

The final Tuesday was marked by an Indian
traditional attire theme that was accompanied by a pizza party and a super show
by the NCGs as they gyrated to a medley of Bollywood songs.

Apart from the fun side, the
NCGs also went through the MySQL training boot camp where they were trained on
the product, tools and everything else required for excelling in MySQL. Some of
the topics covered were MySQL overview, introduction to different modules in
MySQL server, storage engines, connectors, test tools like MySQL-Test-Run and
Random Query Generator, Configuration Management Tool - Bazaar, MySQL processes,
open source etiquette, and more. Trainings were supported by an assignment to
ensure that all the concepts were well understood. They were also given some projects
that helped them brush up some of the skills most required for
the job at MySQL.

It's been two months since the induction is over, but the fun and
learning never ends at MySQL :). I think that it's an amazing opportunity to
start a career in software engineering with such an awesome product and a
wonderful team like the one we have here at Bangalore.

Signing off by wishing all
the new recruits a long and bright future with MySQL!

As always, do post your
comments, thoughts, recollections...I'd love to hear from you :)