Thursday, December 28, 2006

As a Quality Architect, I have been tasked with evaluating test automation tools for QA/QE and development more often than I can remember. There is always a fight between homegrown and commercial tools, i.e. a build vs. buy decision. I have always been an advocate of homegrown solutions, until recently. I have architected several sophisticated frameworks myself. In this blog, I uncover some of the very basic requirements of a test automation framework, which should help you with your evaluation or defining the requirements for your homegrown venture.

The complete requirement of a test automation framework can be captured in one line:

A tool that facilitates automating test scenarios and allows anyone to run them from anywhere and at anytime.

This means that automating tests should be easy and intuitive. Tests, once automated, should be able to run on any supported platform or operating system. And most importantly, anyone (QA, Development, Sustaining Team, or even customers, if required) should be able to run these automated tests in an unattended mode.

No tool can generate positive results if it does not take people and processes into account. Apart from the core test automation needs, a framework must also integrate with other existing tools in the ALM domain. For example, a test automation tool must integrate with test management system, which should integrate seamlessly with requirements management, defect tracking and other top-level dashboards. There is no one tool that can serve all our requirements and that is why it is very important to have open integration APIs available for customization. Continuous Integration and Agile testing is the new buzz these days. A framework must mesh well with cruise control, ant, maven and build repositories.

SCENARIO:

A test engineer or a developer automates a test and checks it into a version control system. Cruise control kicks of the nightly build and executes all the pre-deployment test cases. A provisioning system deploys the latest bits and kicks of the post-deployment automated tests. Test results are automatically pushed over to a central server, where they get mapped to the requirements. An email notification is generated with up-to-date report.

Next morning, the manager checks the email, clicks on the link, logs into the reporting system and gets the latest release readiness matrix with detailed drill-down test coverage and code coverage reports.

The company decides to ship automated tests with its product to its customers. Even in absence of the build workspace and central reporting server, customers are able to execute the automated tests and get the local report!

Above scenario captures majority of the requirements of a test automation framework. Some may think it is too extreme and for others some pieces of this scenario may not be applicable at all. But if you really think about it, this is the kind of infrastructure that is required to build high quality software applications. It is required for continuous integration and agile development & testing.

RECAP:

A test automation framework should (choose the ones applicable to you):

Is there such a framework available in the market? I have worked with over 40 different commercial and open source tools and have not found even a single one that delivers even 25% of these requirements. That is why most of the companies revert to home grown solutions. I was also one of the advocates of building a home grown solution, until recently!

BREAKTHROUGH:

Recently, I came across LISA from iTKO. To my surprise, the tool is very impressive - much better than any other in the industry. LISA seems to deliver over 80% of above requirements - as if the company read my mind and captured all the requirements!! There are minor quirks (like all others), but the tool is built on pure java and XML, runs everywhere, provide open APIs for expansion and integration into anything! Amazing data driven capability and provides mechanism to automate complicated end-to-end scenarios. It is a dream tool for developers and test engineers! It allows you to plugin your own java code and mash-up with other technologies. The developers don't have to maintain a separate workspace for test cases - the test cases can be kicked off from the same build.xml file. XML reports and custom report generator can be used to integrate test results into anything.

Monday, September 25, 2006

We know project management is all about the juggling the three balls of time, cost and quality. A project is successfull if it meets the functional and non-functional requirements within predetermined time, cost and quality constraints.

The traditional project management approach (and hence 99% of the tools) focus on completing the defined work within given time constraints and cost limits. However, the recent focus has been shifting more to the quality of the final output!!

Let's look at some examples:

Google. Didn't google missed the time to market long before it released its search engine?

Apple iPOD. Had it made a difference if iPOD was delayed another 6 months?

Toyota Prius in 2000. Missed the TTM by three years! (Audi released its first hybrid in 1997 and Honda released its hybrid in 1999)

More and more companies are realising the fact that quality rocks!! If your product is high quality, it doesn't really matter if you are a year or two late to the market. Every product has its life, but if it is of high quality it tends to live longer - which changes the whole Net Present Value (NPV) calculation, in case you are using it to calculate the validity of your projects and releases.

Your product may have over thousand functionalities, but just pick a handful of core ones (maybe 3 to 5) and all of the non-funtional requirements for your first release. A high quality product markets itself: word-of-mouth is the most effective marketing tool. Once a customer is hooked-in, slowly roll-out new features. That way you'll have the relationship going and you can get a continuous inflow of money - easy from SEC's perspective and no hassle of accounting manipulations either! That's what is driving software as a service (SaaS) market today.

SOA is the SaaS enabler and it is changing the way software is released. SOA brings business agility. However, our project management tools are still old-fashioned. Project managers are still focused on TTM and CTM concepts. They are still chasing deadlines and pennies. Quality awareness is forcing ALM companies to come up with more sophesticated tools that stitches the SOA fabric.

Wednesday, September 20, 2006

Interesting article and discussion on the definition of quality on StickyMinds.com! Article is dated back to 2001, but it is still very much relevent. Robert L. Glass has done a good job in defining what quality is and what it is not. As you can read through the comments, not everyone agree with his definition - as one would expect. Quality is a FAT word and can be interpreted in zillion of ways. It is therefore important that the project team agrees to one definition of quality and stick to it. Consistency is far more important than the definition itself.

ISO definition of Quality: The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. (ISO 8402: 1986, 3.1)

This definition captures both funcional and non-functional requirements. And BTW, the official name of all "illities" is Systemic Qualities. And there are a lot more Systemic qualities than what Robert has mentioned - for instance - interoperabiliy, availability, scalability, etc.

Another point I disagree with Robert is that "customers/users must participate" in prioritizing and selecting "illities". Some of these systemic qualities are customer facing and other are company facing. For example, it is in companies best interest to make sure there is flexibilty in the code for future expansion and understandability is important to the company for maintenance purpose! Customer doesn't care if your code is moduler and your architecture is flexible. All he cares is the feature set he wants, when he wants it. Customer cares less about the business requirements. But when we talk about quality, all requirements come into picture:

I don't think we need to be in accordance with the customer on all these requirements!!

Another interesting topic that was raised in the article is whether quality can be quantified, given the definition by Robert Glass. I find it rather amusing because, I think, Quality can be defined and can even by quantified. Of cosurse, not everyone would agree with your definition and your way of quantifying it, but you can definetely do it. And as I said, consistency is far more important than the definition itself.

Friday, September 15, 2006

Setting up a goal is one thing, but how do we know that we have achieved our goal? Software engineering is becoming more of an art than science. Success is a relative term! A project manager with exceptional artistic and articulative skills can sell a project, which is on road to failure, as a successful investment to the executives. In absence of real numbers, the darkness prevails. And under this darkness, all decisions lead to the path of failure.

Snapshot:

"We have a GA date approaching. PPM calls a Projet-Team meeting and takes a vote of confidence, which decides the fate of the software!!"

"A P1 bug is not a show-stopper if it already exists in production. The release will not degrade the production quality!!"

"QA gives a conditional GO with list of risks. By the time decision propogates to executives, the attachment is dropped and the Conditional-Go turns into a Sure-shot-Go!!"

"PM: The problem is not in our piece of the code. The issue is because of the other component that we are dependent on!!"

Sounds Familiar? Interesting, isn't it?

Let's face it, we need sophisticated tools that can generate real time metrics for anyone to make informative decisions. People often mix product quality with process quality. Even though a high quality process generates a high qualiy product (TQM principle), I believe the metrics for the two should be different. For example, higher percentage of test automation improves QA process quality and doesn't directly improve underlying product quality! Note: the automation of processes in the early ALM cycle would have a more direct impact on the product quality.

Here are the list of questions, that metrics should be able to answer:

Q1. What is the overall Quality Index (QI) of the product. QI for a particular feature or requirements? What is the QI of different components?

Consistency of the processes and measures is the key here.

It is easy to fabricate a QI model that concentrates on intrinsic product quality!

When files A, B and C change, which features get impacted. What test cases and configurations should a test lead plan for next build?

Q7. Process Quality. How productive is my team?

Measures of test automation.

Comparisons with baseline (and manual testing)

With above metrics in hand, I can easily make statements like:

We are ready for the release!! Our product QI crossed 85% in the last build.

Because of TTM (time to market) pressures, we have decided to release our product with 65% QI. To mitigate the risk, we have also decided to increase our customer support resources.

Our product is not ready for GA because we have a dependency on products B, C and D, and product B has a QI of only 30%. Since B is tightly coupled with our core, we are not in a position to release our product.

I can effecively utilized my QA resources to concentrate on only the impacted features in a build. We don't have to regress every build every time. We can validate a build with handfull of fixes in less than two hours, and that too with over 95% confidence!!

We can now sell SLAs and QLAs around certan metrics because we have a consistent (and automated) way of capturing them.

I can trace a customer escalation all the way back to requirements, because we have end-to-end integration of ALM tools with excellent search facilities.

Friday, September 08, 2006

I found a really interesting article on Application Quality by Allen Stoker. Make sure you read both part 1 and part 2. Best part is that it discusses the need of a Quality Architect:

"Quality begins in the team - not the application. Proper planning, communication and processes are essential to any successful project. Projects that lack these fundamentals will likely produce problematic applications. I'm a firm believer that large teams with diverse skill sets need a Quality Architect - a highly skilled technical person on your team who has no assignment but to support or ‘enable’ the other team resources. Such a resource can mean the difference between project success and failure."

This is even more interesting to me because I spent some time last year just to understand the role. I would agree 101% with Allen that this role can make a world of difference and can be responsible for a project's success or failure, especially in light of the fact that quality is the measure of success! (assuming quality is part of the defined business goals)

The Role of a Quality Architect:

Get the business, engineering, and QA teams to agree on common quality goals (i.e. define quality!)

The role touches almost every aspect of ALM , i.e from requirements to requirements. Horizontally, the Quality Architect is responsible for coordination and collaboration across cross-functional teams (from marketing to design & development to QA to operations & customer care) Vertically, the person is responsible to boost team's productivity and at the same time explain quality metric data to executives in layman terms.

The role requires a fine balance between extraordinary people skills and hands-on technical skills!

Thursday, September 07, 2006

I like the bold questions raised by jason in his blog "Development vs. QA - Why disagree?". For the last half decade (especially after dot-bust), quality has gained an overwhelming visibility in the software industry and the awareness is growing day-by-day. Numberous studies have proven the exponential relationship between the life of bug and associated cost. Sustaining costs are increasing far beyond original development costs. Therefore, companies are trying to crush new bugs, as soon as they find their way into the code. And hence, the pressure is on developers to test their own code!

I don't see the testing goals between development and QA as conflicting. I see the conflict more because of differences in role, availability of test beds, and more importantly the will! Developers generally don't want to do testing - they always write the perfect code!

The QA is way too on the other side of the wall. 99% of QA teams are involved in black-box testing of features as customer sees them. So, Quality organization is always more close to the end-customer as compared to the development.

To me - both practices are inefficient. We all know that by testing in the end, QA cannot build quality in, it's the development team that needs to write a quality code to start with. I am a firm believer of TQM principles and Deming 14 points. To improve quality, all processes must be standardized, engineering principles must be put in place, and there should be tools that ease the adoption of all these processes. Processes without tools create too much work and chaos!.

The solution is to have a QE team, a team that is more close to development (report to the same director, or even the same manager!) , responsible for all the functional testing. QE team can catch bugs early in development cycle and QA team can focus only on non-functional requirements as part of the system testing.

Developers don't want to be QA!! They restrict themselves to Unit testing and some basic functional unit testing. Another complecated issue is the deployment that is generally not automated. Lack of automated deployment breaks down the Continuous Integration cycle and developers' motivation to automate post deployment test scenarios. Using tools like MockStrutsTestCase and CactusStrutsTestCase, developers have started to look into some level of pre-deployment functional unit testing (including in-container testing) - but again, that's streatching the limits of Unit testing, as jason said.

QA is black box. But there is a limit to what QA can test - with limited resources, especially time. Once feature freeze is done and all the code is checked-in, nobody wants to give QA couple of months just to make sure they can complete the test cycle. The complicated nature of software, with all the reuseable code and interdependencies, fixing one bug late in the game has a huge potential to give birth to two or more bugs (it's like the Samuel monster from Helloboy!)

Watch Watts S. Humphrey's video for more info on why testing in the end is a bad bad thing to do.

There are different layers at which we can test for security - Physical, Hardware, OS, Network, and Application. In this blog, I am only addressing application layer security testing. Therefore, you'll not find items like testing of firewall policy rules, hardened OS, checking for all open ports on every system in the data center, testing of dialup & VPN access to systems, system interconnection vulnerabilities , or Intrusion Detection System (IDS). This blog is just a starting point and does not gaurantee end-to-end security test plan.

Authorization: Act of identifying an individual, i.e. it is determining whether they are who they claim to be. This testing includes:

Password based authentication

Checking against Denied Parties Restriction List (DRPL)

Test for unauthorized countries using Reverse DNS (rDNS)

Test for Login leakage: Test to make sure that user is not revealed whether the userID was wrong or password was incorrect, in case of authentication failures.

Authentication: Act of determining whether a given user is allowed to access a given resource under given circumstances (Role Based Access Privilege).

Test that only authorized administrators with the appropriate privilege are allowed to access each administrative function.

Spoof testing by logging with one role and trying to access non-privileged administrative function (use URL bookmarking)

Test by accessing restricted URLs without logging in.

Password Strength. Test for password length and strength, password history, rollover and expiry. Make sure dictionary words are not allowed.

Passwords in clear text.

Check for hard-coded passwords into the software bits or scripts. Run strings on binary code and look for password tags and strings

Check for password in log files (at all log levels),

Check for password in client side cookies and hidden form fields.

Encryption. Tests to make sure that all form submissions use encryption to ensure that information such as passwords do not transit on network in clear text form.

Use snoop to capture network packets and make sure no data is transmitted in clear text

Check for SSL Certificates - HTTPS and TLS (for LDAP)

Session Management. Act of maintaining a transaction or a set of transactions from a given user. This involves maintaining the context(some sort of GUID) of an original authentication so that a user does not have to provide a password for every submission.

Test for automatic password protected locking feature on time out.

Logout action must terminate the active session

If multiple servers are used, make sure session transfers are secure and work as designed.

Make sure when a session is destroyed, it is destroyed across all systems.

Test for maximum session limit per user (if there is any limit imposed).

User Profile and Privacy. Make sure that company's privacy policy is communicated to the end-users. Any forms which collect personal information must include a privacy purpose statement explaining why the information is being collected and how it will be used.

Cookies: Cookies are stored in the browser cachegenerally to manage session state. These can be permanent or session specific, with the difference that session cookies get destroyed when browser is closed. Since these are plain files, they can be edited by any hacker.

Tests for permanent cookies to make sure no user specific information (ID or username or password!!) is saved.

Auditing and Logging: Act of checking a set of actions to ensure that they comply with a given set of expectations.