8 May 2017

How powerful Salesforce test automation can help you win

Most customers who have engaged us for Salesforce test automation can identify with at least one of these issues with manual Salesforce testing:

A bad release that has cost lots of revenue, profits or reputations;

Poor quality practices aren't an issue right now, but will become a huge issue if the status quo isn't changed;

Apps built by different developers are causing conflicts in production; or

Your or your team are copping heat because of poor performance.

In fact most of our Salesforce automated testing customers can also identify with this situation:

This situation is usually the result of stressed testers who are inundated with testing requests that they simply cannot complete with any reasonable level of accuracy or integrity. One consolation point is that this symptom is not unique to your Salesforce testing program alone.

The problem with manual-only Salesforce testing...

...comes down to three primary factors that are common across most manual-only Salesforce testing programs that we have analysed:

Time - comprehensive tests take days if not weeks. Who has that much time?

Coverage - even the most thorough manual testers can't test every nook and cranny in the time available.

Accuracy - with so many moving parts to every customised system, not even the most thorough manual testing program can adequately check for every possible regression that may creep back as the code base grows.

The folks that test the core Salesforce platform internally before it is released to their customers have this to say about their testing practices:

Here at salesforce.com, we use the open-source Selenium project to execute over 40,000 test cases against the features of the Salesforce application.

Salesforce Engineering Team

What's good for the goose has to be good for the gander, right? But what that statement above does not tell you is that Salesforce has built its own testing tool for which it uses Selenium as an automation engine. This allows it to remedy the three problems I described above when running its automated regression tests. I'll tell you how you can get access to a similar tool a bit later.

No matter how much you try a manual-only testing program for Salesforce will result in errors, inefficiencies, bugs creeping through to production and ultimately loss of reputations, for the entire Salesforce delivery team. A steady stream of such occurrences cause end-users to lose faith in the systems, leading to a vicious cycle of inefficiencies, ineffectiveness and the departures of otherwise good and talented professionals.

Size and complexity

Although they are often related, the reason manual-only Salesforce testing programs often end in tears is because even a stock-standard Salesforce implementation is a complex beast. However, most organisations customise their Salesforce environment with third-party apps. While this extended ecosystem helps to deliver a more workable solution for the organisation, this situation becomes a headache for Salesforce testers because the number and interdependence of elements that they have to potentially test goes well beyond the small matter of the 40,000 that the Salesforce engineering team described above.

The interdependence problem is a significant one. It is also one for which manual Salesforce testing techniques have no answer. A study of Salesforce administrators came up with a ranked list of their Salesforce headaches:

You will notice that points 3 to 11 in the graphic above lead to point 12. The complexity of third party integrations, the need for data-driven quality assurance, managing user roles all combine to make Salesforce one incredibly difficult application to test manually.

If you haven't yet understood my point, this additional graphic from the same study should help hammer it home:

Now you might look at this and be forgiven for thinking that testing Salesforce on mobile devices isn't really a problem here because it's such a small fraction of the overall access points. If you came to this conclusion then you couldn't be further from the truth!

Why? Because the people who access your Salesforce environment on mobile devices are usually your colleagues in the field. Salespeople, field workers, managers approving orders, etc. These people are the ones who rely on your Salesforce system to stay to connected with each other and more importantly, stay connected with their stakeholders, who are most likely to be your customers.

Also, that particular study was conducted in 2015. Current access rates of Salesforce systems from mobile devices will likely be many multiples of the figure above. Now tell me that it's ok to not test your Salesforce environment for mobile devices and tablets!

If you belong to the 36% above, then the complexity of your Salesforce system just skyrocketed. Just think about the interdependence of data and processes that now needs to be tested. If you continue to do this manually then two things WILL happen:

The costs of testing and time it takes to test your Salesforce ecosystem will blow out as more manual testers will need to be hired; and

You will not be able to keep track of all the bugs that need to be fixed or have been fixed or have resurfaced after a fix, leading to very frustrated and angry users who just can't get Salesforce to work for them.

I could go on, as I'm sure could you, but I am confident that you get the drift of why a manual-only Salesforce testing program is ineffective and inadequate.

Want to see how Salesforce test automation could save you weeks every time you test?

So if not manual testing, then what?

Let's be clear, I'm not suggesting that you should cease and desist from manual testing of your Salesforce environment entirely. In fact anyone who tells you to do this is either a pretender or high on something. Manual testing must be used smartly and in concert with a robust and flexible test automation program to really improve the quality of your Salesforce environment.

But be careful, because many of the products that purport to offer Salesforce test automation are really only good at cycling through screens. They don't actually do real regression testing. Many of these products are often considered "industry-standard", which is a reason why many IT decision makers harbour the view that Salesforce test automation is an expensive and ineffective investment. In fact, start to run in the opposite direction if you are told that your Salesforce test automation solution will utilise "record-n-play" technology to build your automated regression suite.

Read this guide to 10x your testing effectiveness to understand how these three factors can be combined to deliver a truly best-practice Salesforce test automation program.

Salesforce, like other complex ERP and CRM systems, has nuances that make test automation difficult. This is why you need to be sure that the people or organisations you select to build and execute your Sales automated regression testing suite have experience in overcoming these nuances. While none of the challenges posed by Salesforce are show-stoppers for test automation, only the best test designers and automation engineers have the know-how to build Salesforce test automation that will not be bogged down by an automated mess.

At Qsome we have built a specific framework that enables faster builds of automated regression tests for Salesforce. It was while building this Salesforce-specific framework that we created functionality to take care of asynchronously loaded elements, particularly within Salesforce's console app. This intimate know-how allows us to deliver value fast and often during our engagements.

Want to see how a framework built for Salesforce test automation could save your bacon?

OK, but how can Salesforce test automation "help me win?"

Winning is a very relative term. The context that helps you decide you need or want to win is just as important as the destination. For some people it's a job well done. For others it is the accomplishment of goals that were previously thought to be impossible within their organisation or among their peers. For other still, it might save their job or win them awards.

Whatever your motivation for implementing Salesforce test automation there are three outcomes that you must demand of your team or your service providers:

Cut testing time from days or weeks to mere hours

Increase test coverage and the robustness of each test, particularly with negative testing

Provide test-related analytics that determines when you should release to production

While most Salesforce users are satisfied with even just one of the above outcomes, you will be settling for a sub-standard outcome and literally leaving money on the table if your Salesforce test automation program does not deliver all three of these outcomes. Without achieving all three of these outcomes you are not maximising the Green Zone:

Read page 4 of our Ultimate Guide to Test Automation for a brief business case of the ROI that's possible from an investment in test automation. The example used in the Ultimate Guide is based on a sizeable, but simple application. As we learned above, the size and complexity of a customised Salesforce environment makes it an application ripe for reaping huge dividends from test automation.

Implementing Salesforce test automation is by no means an easy task. If you need help in figuring out the right plan of action or how to go about implementing your plan, speak to us understand how we might be able to help you: