Confidence was low - during the previous two years the service had buckled under peak demand and crashed. The client was determined to avoid any repeat and asked us to load test the live service in order to prove it was ready.

The focus of the test was a back-end API service, responsible for serving data to the mobile app.

By analysing the demand patterns from previous peaks, we ascertained our testing targets of 500,000 virtual users and 15,000 transactions per second (TPS).

We used a suite of software, called Cloud Load Testing (CLT), to deliver:

Automation of Load Injection

Test Automation

Automation of Test Analysis

1. Automation of Load Injection

Performance testing with 500,000 users requires a large number of load injectors. Our CLT suite provides us with a mechanism to create numerous load injectors, and, for the purposes of controlling costs, to destroy them easily as well.

We use Amazon Web Services (AWS) to host our load injectors. This enables us to globally distribute our load (which is important to ensure we don’t overload the content delivery network that is in front of the service we test).

2. Test Automation

Test automation is required to

Upload test data and scripts

Collect results

Schedule tests

Execute tests

We've developed automation to run in parallel across multiple load injectors. Through a process of improvement we have reduced the time to perform some tasks from 75 minutes down to 5 minutes.

3. Automation of Test Analysis

In terms of time saving, we have found the most value is in the automation of the analysis of performance test results.

Load testing to 500,000 users creates enormous data sets which need to be parsed and mined in order to find performance issues.

We automated the processing of multiple results files. With reference to the load ramp profile, we use a series of algorithms to detect performance anti-patterns in the data sets. This allows us to identify and prioritise test defects.

By automating the test analysis we have been able to quickly find issues and shorten the feedback cycle to the client.