Article

How to Avoid the Most Common Load Testing Mistakes: Secrets to Success from EDF’s Massive Load Test Project

by Susanne Janssen | SAPinsider

April 1, 2010

Load tests are critical for any company undergoing an implementation or upgrade. Yet too many companies overlook the most crucial load testing considerations. This article, which includes tips from Electricité de France’s hugely successful and productive load test project, explores a two-pronged approach to ensuring that your load tests — and your go-lives — are mistake-free.

For any organization that’s planning a new system implementation or upgrade, load tests are vital. These tests validate whether or not a system can handle the expected future load required to support the company’s evolving business needs. While there are many resources that detail how to perform load tests,1 we’ve found that certain aspects of load testing are too often overlooked, causing project delays, uncertain results, and increased costs.

Most people think that the most important criteria for a successful load test project is whether targeted key performance indicators (KPIs) are met. But another crucial criterion is how well you handle the series of system and coding optimization recommendations that the load test yields — a factor which can also influence how well you meet your KPIs. If properly understood and implemented, these recommendations are powerful instruments to ensure solid, long-term performance within the production system.

So how can you make sure that your company meets its target KPIs during load testing — and, more importantly, once your system is in production? We recommend a two-pronged strategy: Properly prepare your test system data and conduct extensive unit or single user tests before you even begin your full load tests, and understand how to sustainably handle the performance-optimizing recommendations that your load test will generate.

To help illustrate the benefits of this dual approach, we’ll follow the example of one utilities company, Electricité de France (EDF), that has executed extremely successful and productive load tests as a result of following this approach (see sidebar). Throughout this article, you’ll find tips a nd best practices from EDF to help others follow its lead.

EDF’s Load Test Project: An Introduction to Its Size and Scope

In the context of the deregulation of the energy market in Europe, Electricité de France (EDF) decided to replace its numerous utilities’ legacy systems with SAP for Utilities, using SAP ERP 6.0, SAP CRM 5.0, and SAP NetWeaver Business Warehouse 7.0.

This project was first conceived in 2003. Its goal was (and is) to accommodate 28 million business partners from nearly 100 legacy systems on the new SAP landscape by 2013. As of December 2009, 5 million business partners have already migrated. These migrations are done in steps involving 500,000 to 1 million business partners at a time. To ensure that its system infrastructure can handle this high load, EDF decided to closely cooperate with SAP Active Global Support (which they engaged through a SAP MaxAttention contract), as well as with their application partners and performance experts from EDF’s database, operating system, and storage providers.

With the goal of reacting flexibly to any changes in technology or required business functions that might come up during the implementation, and mindful of the large load that the infrastructure would need to handle, EDF decided to perform three cycles of tests, one each year, each with a different scope. EDF has finished the first test cycle, simulating the load of 10 million business partners (roughly one third of the target volume). In 2010, EDF will simulate the load of 20 million business partners; and in 2011, EDF will simulate the load of 28 million business partners.

Set Up Your Load Tests for Success: Planning and Preparation Tips

To get the best results from your load test, planning and preparation are key. To start, you’ll need to build a team of business owners, SAP functional experts, SAP technology experts, and performance experts. This team will then be responsible for laying the groundwork for the load tests by, for example:

Determining the scope of the tests

Determining which business processes need testing

Determining the timeline of the project, the KPIs, and whether additional experts will be needed

Managing interdependence with the go-live and functional development cycle

Identifying the type of data to load into the test system

Setting up unit testing or single user testing

What follows are key considerations for each phase.

Increase Testing Efficiency Through Progressive Scoping

One precondition for successful load testing is defining what you want to test. To do this, you must first determine and set the scope of the test. This involves:

Inventorying available resources — including hardware, software, teams, and support

Defining test objectives and monitoring indicators

Assessing what is required to achieve your objectives

Documenting the decision process for determining which processes should be tested

In EDF’s case, the team needed to test many different aspects of performance and scalability to ensure that the system could handle the final volume targets. That’s why EDF decided to run a series of load tests, each with a different scope.

The goal of the first load test (conducted in 2009), which represented 10 million business partners, was to identify potential application bottlenecks for the target volume. This test focused on the company’s most critical business processes and tested the majority of custom-developed coding. In addition, the tests helped to analyze the layout of the architecture to see if it could serve as a basis for the target architecture.

The goal of the second load test (to be conducted this year), which represents 20 million business partners, is to check the infrastructure’s efficiency — this includes testing how it interacts with SAP and non-SAP systems, the height of its availability, and its fail-over limit, for example. EDF will also test a few functional modifications — including implemented recommendations from the previous test phase — to analyze their performance impact.

The goal of the third load test (to be conducted in 2011), which represents 28 million business partners, is to verify whether the production system will be able to handle the final target load.

When dealing with such large volumes of users or with such a large-scale implementation, we recommend splitting your load tests in such a way that you can more effectively address each set of challenges. This approach allows you to progressively implement improvement recommendations and then validate them in the next round of tests.

Run Your Tests on Representative Data

Because the goal of load tests is to provide solid recommendations for how to improve and optimize the production system, the load test system itself should reflect the data layout of the production system as closely as possible. In addition, matching the testing system and the production system will improve your precision when sizing the production system. To determine what kind of data needs to be loaded into your system, the project team will need to work closely with the functional experts, who will contribute valuable insight into the kind of data that the production system will contain.

EDF took great efforts to load real, representative master data and transactional data, and then “age” this data in the performance test system, so that its database would be similar in all aspects to the future production system.

Focus on Testing Critical Business Processes

Another way to limit the complexity of your load tests is to focus only on critical business processes. In some cases, these processes will be clear — a bank would likely want to focus on account balancing calculation jobs, for example. But in other cases, determining these processes might require further analysis.

For instance, dominating EDF’s business load are multiple batch processes that run at night, some of which can be run in parallel and some of which depend on the completion of a previous job — that is, they’re on the “critical path.” To determine exactly which processes are on the critical path, EDF analyzed its current production system to find the execution time of the processes in relation to the amount of data processed and the parallelization parameters (the number of parallel jobs or the time intervals, for example). This analysis helped them identify which processes needed special attention at the load testing stage, as well as which processes would need special attention in the new system.

Run Unit or Single User Tests Before Load Testing

Once you have identified which processes you want to test, prepared the test system, and set up the scripts, you can begin the testing phase. To improve the results of your load tests, SAP recommends running initial tests, either as unit tests — that is, one process step at a time — or as single user tests, in which one user or click stream is monitored for performance KPIs prior to the load tests. Why? Well, for your load test to produce meaningful optimization suggestions, you’ll need to re-run your test cases several times to identify the root cause of the performance problem, and then validate the impact of the suggested solution.

These initial tests reveal areas of improvement, many of which — such as minor changes to system settings, customizations, or coding — can be put into practice quickly. Other recommendations may reveal functionality-related problems, architectural issues, or severe changes to the coding, which the company can address and then validate during load testing.

Let’s revisit our EDF example. Before EDF started its “stage one” load tests, the individual process steps were tested as separate unit tests. The team was then able to analyze and o ptimize the performance of each process.

Note!

Some of the improvements that a unit test identifies might take weeks or even months to address, so their benefit will not be immediately visible during the load test.

Then, using the data from these unit tests, EDF could determine the optimal system settings and the precise combination of parallelization parameters — including number of jobs, processing interval, and packet size — to use in the load tests to render the highest amount of throughput for each process (see Figure 1).

Figure 1

Finding the optimal number of parallel processes to achieve maximum throughput

Thanks to unit testing, EDF identified several bottlenecks before even reaching the load testing phase and was able to optimize its coding, thus cutting the batch night process duration in half (from 30 hours down to only 13). EDF reduced the duration of the batch night process even further after the load testing phase (down to 9 hours), a feat that may not have been possible without the changes made during the unit testing phase.

EDF’s Tips for Unit Testing

Take time to present the algorithm and functionality of each tested program to the entire team of performance experts. When everyone is on the same page, the team can focus on essential activities and won’t have to worry about redundancy.

The team that performs the unit test should also do the load testing, since they best know the history and understand the interdependencies.

Involve functional experts to review the business aspects of the load test scripts. A small functional problem in the script can distort results.

In case of Z-coding, involve the development team so that they can foster coding optimization. The team can quickly estimate the effort to implement a recommendation into the coding and may point out interdependencies.

Use Your Load Test Results Wisely: Handling Recommendations

After your load test, the experts who ran the tests, monitored the system, and analyzed the processes will generate a list of recommendations to improve the system and optimize performance. A very simple recommendation could be to increase the table buffer by 500MB, a parameter modification that can be done on the fly and will only impact available memory. More complex recommendations are those that affect custom coding or modifications to the SAP standard — these require further testing once implemented.

To best manage these recommendations, you’ll need to establish an effective procedure for evaluating the impact of each recommendation, as well as its potential requirements for further coding and test efforts. This procedure must include methods for qualifying and prioritizing the recommendations, as well as a mechanism to integrate resulting improvements into the production rollout.

EDF’s Tips for Managing Load Test Recommendations

Establish a test coordinator to manage prioritization and further handling of the recommendations.

Set up a small, additional test system to deal with recommendations that will take longer to implement — those that could affect functions and application coding, for example.

Since some tests will need to be reproduced several times to determine whether the recommendations are being implemented successfully, it’s a good idea to have a system backup strategy in place so that the conditions concerning system settings and data structure for each test cycle are identical.

Allocate adequate time to implement recommendations — this includes leaving enough time for in-depth analysis of more complex recommendations, which can be done in parallel to the testing.

Prioritize Your Recommendations

Before you can implement the recommendations from the performance test team, you’ll need to qualify and prioritize them. An easy way to do this is to set up a matrix (see Figure 2). This matrix would help you easily see how your recommendations are ranked in terms of ease of implementation, pervasiveness, functional impact, additional research required, and so on.

Job #

Problem Description and Impact

Status

Suggested Solution

Timeline

Job 1

Runtime of the job too long.

Impact of data storage: very high.

Error message number: 123456

: )

Modify standard coding — responsibility of partner A.

Evaluate impacts of tests planned for calendar week X:

Runtime (-10%)

Error log files written (-90%)

Done

Job 2

Parallelization not possible.

Impact on business need: high.

Error message number: 654321

: (

SAP suggests a custom development.

Business to decide if they want it.

Contact person: Jane Doe

Within two weeks

Figure 2

Example of a decision and follow-up matrix

At EDF, the different experts (business application, system, storage, database, and SAP application experts) sat in the same room so that everyone was always on the same page. EDF also appointed an independent test coordinator, whose job was to mitigate a common agreement on the applicability of the recommendations issued by the different parties involved. This coordinator was empowered to decide how critical recommendations should be dealt with, for example, by demanding additional tests and validations.

EDF’s Tips for Qualifying and Prioritizing Recommendations

Rank each recommendation by priority, complexity, and potential gain — this can be in terms of either performance or functional aspects.

Set deadlines and keep track of the recommendations’ fulfillment status.

Decide where the recommendations should be implemented first — for example, determine whether to implement them in the load test system immediately or to test them in a separate system first to mitigate potential risks.

Figure out if implementing the recommendations will require additional validation tests.

Consolidate and Integrate Recommendations

Some of the recommendations will require additional functional development work before they can be implemented. The performance test team, therefore, must work closely with the application development team to clearly communicate what the recommendations entail and to test any completed development work. To facilitate this and stay organized, the performance test team should categorize the recommendations — as functional or performance-related, for example.

A close relationship with the application development team can also help if you come across a problem and cannot easily find a solution. In this case, you’ll need to run proof-of-concept tests to help determine viable recommendations.

EDF’s Tips for Integrating Recommendations into the Overall Development Process

Make sure that the application development team allocates sufficient time in their schedules to implement coding modifications.

Together with the person in charge of the development cycle, define a method for handing over recommendations to the performance test team and for validating improved coding.

With the functional experts, prepare a recommendation consolidation. This list should include information on the potential impact of implementing your requirements. You can then pass this list on to development.

Work closely with the application development team to conduct proof-of-concept tests for problems for which there are no readily apparent solutions.

For quality assurance, include a performance expert in the development assessment.

Conclusion

The true objective of any load test is to ensure that your production system will run smoothly at go-live. As demonstrated by EDF’s successful load testing strategy, the keys to reaching this objective are twofold: Rigorously prepare for load tests by setting up truly representative test scenarios on a production-like test system, and take great care to ensure that the recommendations obtained during the tests are not only of high quality, but are sensibly addressed.

To learn more about SAP’s services, which can help guide your company through successful load tests, visit service.sap.com/tmc.

Additional Resources

Susanne Janssen (susanne.janssen@sap.com) has been a member of the SAP Performance and Scalability team at SAP AG since 1998. She specializes in performance methodologies and procedures, as well as hardware capacity planning. In this role, she spent time at EDF during the first stage of its load test series.

Fayçal Hadjiat (faycal.hadjiat@edf.fr) is project manager at EDF, where he’s worked for 10 years. He leads mass-market performance initiatives to ensure that EDF will meet its goal of accommodating 28 million business partners by 2013. He is currently in the planning phase for this project’s next major test stage, the technical infrastructure tests.

More from SAPinsider

While the use of cloud-based solutions is on the rise, on-premise solutions remain a critical part of IT infrastructures. To help minimize the administrative burden of managing these hybrid environments,...

In today’s IT landscapes, software developers must not only support the daily operations of modern business, but also balance the performance expectations of end users demanding short response times and...