RISE-UP: Performance Testing Principles

We need to RISE-UP in advocacy of performance testing. This post outlines some principles of performance testing with the mnemonic RISE-UP.

Performance testing should be:

Repeatable

Integrated

Scientific

Early and Often

Understand

Predictable

Realistic

Ideally, we need both our load placed upon the system and our environment to be realistic.

Load on the system – Throughput – Transaction Volume Model

We need to have a realistic idea of the load placed on our systems, we then need to replicate that load. Being unaware and running a perf test at half the load level may mean we fail to identify performance problems. Running at double the load may mean we overprovision spend time and effort fixing bottlenecks which we may never reach.

We need to invest the time into finding out what the true load level in our production systems is, this may mean speaking to many different teams MI, BI, looking at logs, analysing data, but this is an essential first step for any performance test.

Environment – Is your pre-production environment treated in the same way as your production environment? The data loaded into the pre-prod the same as your prod environment, emergency fixes replicated? The infrastructure created the same way? Do everything you can to have an identical copy of production.

Load generators – Is how you are generating the load on the system as realistic as possible? Does it give you the real response times or is it from one machine within your network and does this machine constrain your results?

Integrated

We need to integrate our performance tests into the team, the product and the pipeline.

We don’t want to do performance testing in isolation, quality, as we know, is everyone’s responsibility within an agile team. So should performance testing, if the team are a java team consider using a java based tool. If the team has more python skills use a tool like Locust. Similarly for the product, if the product is written in Scala try using a Scala-based tool and keep the code and tests as close to the product as possible. The goal here is to encourage the team to adopt the tool and tests and this is more likely if its similar to the product or their current skillset.

With the increasing adoption of continuous delivery and automated pipelines having environments span up on demand performance tests can be made part of these pipelines.

If our delivery is to be continuous can our performance testing be always on, continuous? Can you replicate\mirror your production workload against your pre-production environment and have this running all the time? Then simply deploy latest builds into this environment and wait and watch for feedback.

Scientific

When performance testing it’s important to only change one thing at a time. If you are changing your scripts and scenarios don’t change the versions of the software you are testing. You need to re-baseline and establish the level of performance with the current scenario.

When you do find a performance hit or defect usually the first thing people do is to question the validity of the results and you have to justify your testing and results, hence being scientific in your testing is probably helpful.

Early & Often

Testing is about providing feedback. Agile emphasises early delivery and early feedback through working software. To support this we need early and often performance testing.

Performance testing historically is often left to the end of the project and crammed in when there is no time to fix any problems. We seek to integrate testing earlier into the lifecycle and value the feedback this gives us by working in small increments. We need to do the same with performance testing. How could you get some early feedback?

Can you performance test on a smaller scale environment, in dev for example? Some teams I have worked with in the past have found and fixed defects in very early dev environments saving lots of time. Could you monitor build execution times? Could you put timers in unit tests even and monitor trends over time.

Performance issues are not like functional defects. It is not usually just a simple small fix, a late project performance defect usually needs a lot of digging around refactoring and investigation. Early detection of a performance issue could be incredibly valuable.

Understand

When performance testing it is imperative you attempt to understand how the system is performing, what the bottlenecks are. It is not ok simply to give response time statistics. Getting resource utilisation stats from servers, what is the hardware doing? Look at correlation IDs, follow transactions through the system. Can you even influence the product team to make the system more understandable and testable by increasing logging? Using log aggregation tools such as Splunk.

Get access to the logs and metrics on production and on your test environments, attempt to understand what is happening on each and the differences. Do the metrics and logs look the same in your environment as they do in production? Are you missing something important from your test?

Understand as much as you can about the wider context and ecosystem your system exists within. Are there any up or downstream systems involved which need to be considered as part of this test?

Predictable\Repeatable

When performance testing a lot of the time we have an existing system in place. We are often looking for changes in performance. Particularly with the prevalent methods of incremental and iterative development and continuous delivery.

For performance testing, this means a lot more potential testing around changes in performance between the many builds and releases. If we are unable to establish baselines to a small enough variance we will be unable to see potential performance changes. The pics below demonstrate:

In Post It 1 (Not This) We can see a high variance anywhere between 0.5 and 1.5 sec response times and because of this variance and being unable to establish a predictable baseline we cannot see the small performance decrease say 0.1 of a second around 10% which we can see in Post It 2. We need to see a predictable repeatable baseline with as little variation as possible to be able to notice small changes in performance. To what % level of variance do you get in your tests?

Your tests should be able to be run time and again and get the same result if nothing has changed.