Creating a load test report

One of the most common questions in load testing is how to create a good test report that you would be able to present to your manager or customer. This big question actually consists of many smaller ones. What to start with? How to create the right test? What load parameters should be applied? How to interpret the results? Which numbers to look at when you browse through a long report generated by a load testing tool?

I will hardly answer all these questions and present the complete methodology in one article. This would be a long story and… it would be too boring. I know that there is a 90% chance that you really need something more simple and ready to use right away. I will simply present an example of the load testing report. A typical one, if you want.

However first of all I have to mention several steps you need to do in order to obtain the test data that you will use to create the report.

Define test goals. The main difference between functional and load testing is that a load test cannot pass or fail until you specify additional conditions. You cannot test that a web site can stand load in general, but you can test that it can stand some specific load with some specific performance parameters. These requirements should be determined by the business goals.

Describe test goals in technical terms. As soon as you have an understanding of what exactly you want to check in your test, you need to formalize these requirements and transform them to technical terms that can be applied in you load testing tool. You need to define the type of load, how many virtual users will run on each test phase, what is the acceptable response time, etc.

Create a realistic emulation of virtual users. This is probably the most complex and important task. First, you need to divide all potential real users of your web application into different types depending on their expected behavior on the site. Second, you need to create a special script, (profile, scenario, or whatever term is used in your load testing tool for this purpose) for each type of users separately. Finally, you should make sure that they work correctly and produce consistent user sessions.

Run the test. This is probably the easiest task. The only important thing to remember is that if you apply a significant load volume, you should make sure that the resources of the systems that you use for load generation are sufficient, and your load testing tool does not get overloaded itself. The same can be said about you network capability that can also limit the load volume, if you transfer significant amount of data in your test.

Analyze test results. How to get valuable information from a series of digits reported by your load testing tool? This is not as hard as it seems to be. First of all you should check that the test went as expected, the load specified for each phase was really created, and there were no errors occurring because of an incorrect work of the load testing tool or your test scripts. After that it’s time to check if the tested web site worked as expected under the load: number of errors, response times, and number of sessions/requests served by second. You should check how these parameters change depending on the load volume.

Isolate problems and suggest improvements. The web site can consist of several components, such as load balancer, web server, application server, database, etc. The performance of the whole system is usually limited by one of these components. This means that improving or fixing that component can result in significant growth of performance, while improving other components will not change anything. Such component is also called “performance bottleneck”. Note that this does not necessarily mean that you should completely change the hardware or reinstall something. In many cases improvement can be achieved by changing options, or fixing bugs in the application code.

Now I am ready to show what I promised from the very beginning: an example of the load testing report. Please note that it is not a general template that can be used for all cases. It is rather a simple, but most common example. Here it is.

1. Test objectives

We need to find out if the web site meets the performance requirements as specified below.

The site needs to be able to handle 20000 unique visitors per day providing the industry standard level of service in terms of response time and error rate. Average response time should be less than 5 seconds and error rate should be less than 1%.

We also expect that average number of simultaneous users on site will be about 150, during the peak hours it will grow to 300. We need to check that the site can handle this load.

2. General test conditions

The following test conditions are most appropriate to simulate the real life site usage.

3.1 Capacity test

3.2 Stress test

4. Test results

4.1 Capacity test

The performed capacity test has shown that the site preserves the desired quality of service only with a limited load. When the number of users simultaneously working with the site goes beyond 160, average response time starts to grow proportionally. When the number of users is about 300, the average response time increases to 19 seconds. See the graph below.

4.2 Stress test

The performed stress test has shown that when the number of users reaches 340, the server starts producing errors. The error rates increases with further load growth. See the graph and tables below.

Number of active users

Total errors %

5. Conclusion

The web site can experience slowdowns showing inacceptable response time in peak hours. When the load is at an average or below average level, the desired level of service is provided.

The web server resource utilization measured during the testing shows that the performance problems cannot be related to the web server hardware. We recommend to check if the database server resources are sufficient and/or increase the number of connections with the database.

3. When we choose ramp-up load, assuming max is 250 users, start & completion settings is to complete 250 user sessions before test end, but among the columns of Number of Active Users in test report, we only see maximum of 87 users, what could be the possible reason? Is that because of error occurs?

1. This depends on your web site. Usually you start from a very small number of users (like 1 or 10) and run to the number of users that you expect to visit your web site at the estimated peak load. If you do not have such estimation, you can simply try achieving some round number, like 100. If the site responds well to such load, you can conduct next test with load from 100 to 1000 users, and so on.

2. Because we want to know the number of users on which the site starts producing errors.

3. In WAPT each virtual user executes its sessions one by one. As soon as a session is completed, it starts the next one, and so on. So, each virtual user can execute several sessions during the test. At any test moment the number of virtual users is the number of concurrent user sessions. In your example, some users executed several sessions and they all completed total 250 sessions much sooner than the load would have reached 250 concurrent users.

Hi,
I am new to WAPT tool.
Please explain me about “Ramp-Up” load scenario. I am bit confused at fields like “From”, To”, “Users with step” and “Every”. How we can give the values for it.
For Example, If I want to test a web application with 2000 users, start with 5 users. Please give me all the configuration details clearly.

Each virtual user executes sessions one by one. As soon as it completes a session, it starts the next one, and so on until the test is finished. So, each virtual user emulates several successive real users and can execute corresponding number of sessions during the test. At any moment the number of active concurrent sessions is equal to the number of virtual users. The more users you add, the greater load will be created against the web site. So, if you use the “ramp-up” option, in the beginning of the test you have small number of users, but it can grow throughout the test. Comparing performance characteristics (such as response time) for different test phases you can check if they degrade with higher load.

It is up to you how to specify the parameters for the growing test load. For example, you can make WAPT add 5 more users every 10 seconds. So, the load will grow from 5 to 2000. Note that it will take 4000 seconds to get to the maximum specified load in such case, so you should adjust the total test duration accordingly.

First of all I would not recommend to run load tests on public web sites (unless you are asked to do this by the site owner), because most probably they will simply ban your IP address as a result. If you need to experiment with something, try running very small number of users.

If you perform correct parameterization of your profile, the result of running it will be a new registered user. If you run several virtual users, this will result in several new registrations. Each virtual user executes sessions one by one. As soon as it completes a session, it starts the next one, and so on until the test is finished. So, each virtual user emulates several successive real users and can execute corresponding number of sessions during the test. At any moment the number of active concurrent sessions is equal to the number of virtual users. The more users you add, the greater load will be created against the web site. So, if you use the “ramp-up” option, in the beginning of the test you have small number of users, but it can grow throughout the test. Comparing performance characteristics (such as response time) for different test phases you can check if they degrade with higher load.

Note however that all public web sites also have protections against robots. They require to enter capcha text, for example, when you try registering a new user. Unfortunately it is not possible to overcome this with WAPT, because it is also a robot. This is another reason why you should better practice on your own web site where you can turn off such protection for the testing time.

Such test parameters as the number of virtual users should be determined by the business goals of your web site. For example, if you expect that it will be normally visited by 1000 users concurrently (in peak hours), you can stress test it with 2000 users to make sure that even in case of 2 times bigger load it will stand it.

As for JMeter, I think that this tool is good because it is free. It is not supported and not very easy to use for a beginner, so I would definitely recommend WAPT, if you can afford it.

The “Ordered list” function takes values from a list one by one. Each new value can be applied either for each new session or for each new virtual user. In the latter case all sessions of that users will share the same value. The list can be either specified directly in the dialog where you edit function parameters, or taken from a file that you choose in that dialog. I suggest watching the following demo clip to get more familiar with the parameterization process: http://www.loadtestingtool.com/quick-start-guide.shtml

If you have any further questions related to your specific case, please send your virtual user profile (.wpp file) to support@loadtestingtool.com. Please also attach your verification log. To save it, verify the test and choose “File | Save logs” from the WAPT menu.

In fact, the terms “web site” and “web application” have the same meaning these days, because any web site that includes dynamic content has an application running behind it. WAPT can be used to test any application that has a web interface, which means any application that uses HTTP(s) protocol for communication between the client and server. If you have any doubts about a specific one, please let me know how can I take a look at it and I will further suggest on that.

Hi , first of all thank you for the good suggestions and materials.
I am new in capacity and load testing. I have told to test a system which consist of different servers, web application server, gateway server , a queue server and DB server.
The main focus is to see how increasing the load on each server would affect the CPU , memory and other factors on each server. basically although the system is a web application but the database get the data from gateway and the web application is actually just used to browse the data in the database.
right now we want to do load testing on the part that read the data, queue it and write it in to the database. I need to simulate the input data and monitor the CPU, memory …. of that particular server and also the bandwidth of its output (which goes to database) and find the capacity of the server.. how much data it can handle (read and process and send to DB) and from which point it gets slow and what is the bottleneck at that stage…
So I would like to know if you have any good suggestions for me , Is there any tool available to do and monitor these kind of stuff and how we can analysis the data out of them, for example zabbix can monitor some stuff but how I have to analyze those data and get meaningful results.,..
Thank you so much for your help

This looks like a typical load testing project for a rather large and complex web application. I would recommend using the Pro version of WAPT, because it can monitor performance parameters directly from one or several servers through WMI and SNMP interfaces. If you apply version 3.1 (which is currently in beta state, but fully functional), you will be able to monitor any server parameters available through SNMP.

Note also that in any case WAPT creates load by emulating real users working with your web application. It cannot apply load directly to a separate system component, however you can create different types of virtual users. They can perform different types of activity, which will load system components in various ways.

If you have any specific questions on WAPT usage, please don’t hesitate to contact me through this blog or through our support email: support@loadtestingtool.com.

Hi Ivan. I have a question about realistic scenario in browsers and Wapt. Browsers connect to server, starts downloading the web page,and starts downloading embedded URLs like javascript, CSS and image files – static content (for this purpose browsers create more threads, each of which opens a new connection and sends requests in multiple concurrent connections). As result, the page loading time in browser is less then in the case when http requests (for downloading css, js ,…) are sent sequentially by some load tool. Does Wapt allow to simulate such scenario? Because if a page contains many static content, time for this page to load can be very overestimated if load tool does not allow to simulate real browser behavior and such important metric as time for page to load can not be calculated

Yes, if your desktop application communicates with its server through HTTP(s) and it can do this through a proxy, you can test the server with WAPT. Note that WAPT only tests the server part in any case, because it fully emulated the client activity.

We are using load balancing mechanism, In live servers version 1 web application is running in version 1 application domain.I want to test Version 2 Web application in version 2 app domain Using WAPT tool.

If i will create 10 users & do performance test with WAPT tool Version 2 application ,is it will effect to Version 1 Live web application users and network traffic.

My question is simple I am having 2 applications one is live another one not live in same server.If i will test not live application using WAPT tool,is it effect to live server application users & network traffic ?

The answer to your question depends on the implementation of your web applications. For example, if they share the same database server, the performance of one application will be affected if you test another one.

Hello Boss,
I am new to WAPT tool, Can u please any one, how to validate the below logic.

Load test 200 concurrent users running normal business process scenarios (level 1, level 2, listing, assessments, dashboard, workflow, etc) for 1 to 2 hours elapsed time, continuous running ramping up from 0 to 200 users in 20 minutes incrementing load by 10 new users every minute, running full load for ½ hour to 1 ½ hours and then ramping down 10 users every minute. Response time between pages < 3 seconds. The system should not crash and the system resources should not be consumed more 100% for more than 1-2 minutes at any time during the entire test. The test should be repeatable with the same normal business script.

First of all you should create a profile for each type of virtual users. To do this you can simply record a typical session using a browser and WAPT recorder. After that you can specify the load parameters on the “Test Volume” page in WAPT or WAPT Pro.

Each virtual user executes sessions one by one. As soon as it completes a session, it starts the next one, and so on until the test is finished. So, each virtual user emulates several successive real users and can execute corresponding number of sessions during the test. At any moment the number of active concurrent sessions is equal to the number of virtual users. The more users you add, the greater load will be created against the web site. So, if you use the “ramp-up” option, in the beginning of the test you have small number of users, but it can grow throughout the test. Comparing performance characteristics (such as response time) for different test phases you can check if they degrade with higher load.

Each virtual user executes sessions one by one. As soon as it completes a session, it starts the next one, and so on until the test is finished. So, each virtual user emulates several successive real users and can execute corresponding number of sessions during the test. At any moment the number of active concurrent sessions is equal to the number of virtual users. The more users you add, the greater load will be created against the web site. So, if you use the “ramp-up” option, in the beginning of the test you have small number of users, but it can grow throughout the test. Comparing performance characteristics (such as response time) for different test phases you can check if they degrade with higher load.

Boopathi, the only limitation of the trial version is the number of virtual users it can generate (20). In all other respects it is fully functional, so all features are the same as in the full version.

There is no such feature as ramping down, but you can use a workaround. You can use the “Stop user” operator to reduce the number of active users. So, you can check some condition (like current time) and stop some users this way reducing their number.

I am just started using this tool, it is quite easy to
operate and replay, but i have query that while load testing does it really load data into the applications back end database or it just replay with defined virtual users.

I am recording the scenario by accessing through browser of given URL, does it really meaningful to get the proper result as i did not do any configuration just record and replay with 20 users by ramp up 1 user per 10 sec.

If you just record and replay a user session, WAPT will repeat same sequence of HTTP requests. It will handle cookies and hidden form values automatically, so in most cases this will emulate correct sessions.

However you may want each virtual user to use its own credentials. Your session may modify server data and you need to use session-specific values, because initially recorded ones cannot be reused every time. In such cases you need to parameterize the profile after recording.

WAPT does not work directly with the database. It performs the emulation of application users. If such emulation is performed correctly, your application will work with the database as it normally does. As a result you should see the corresponding load on the database.

To make your profiles work correctly you may need to parameterize them. That is what I tried to explain in my previous message. If you have any doubts regarding this requirement, please send your virtual user profile (.wpp file) and its verification log to support@loadtestingtool.com. I will take a look at the files and will suggest if you need to do any additional work to execute your test.

I am testing ADF [10g] appl’n but when recording script if i click any button image or select LOV (List of values) image then system gets hang by continuous blinking and then after i have to do “End Task” from task manager & restart tool but problem remain same.

Are you recording with the embedded browser? If so, please try to use external one. When you have many JavaScript code on a page, embedded browser may have problems executing it correctly. So, we recommend recording such sites with external browser.

You can extract the required value from the server response to a variable. This is done on the “Response processing” tab for the corresponding request. In the next request you can use that variable containing “transaction no” instead of the initially recorded value. So, in each emulated session that value will be different and will be taken from the page received after posting the first form.

Same way the parameterization can be done on subsequent steps. Unfortunately this process is application-specific, so I cannot provide more exact instructions on how to do this in your specific case. However if you want, we can work on this for additional charge and deliver ready-to-use parameterized profiles for your application. Please let me know if you are interested in this service.

Usually this means a parameterization problem. In any case you need to check logs step by step from the very beginning of the session and compare verification requests and responses with the originally recorded ones. Once you see a difference, you may conclude on what could cause it.

Hello Ivan, thank you so much for this article, it is the most useful I found about WAPT, i have WAPT 8.5 PRO installed, i am new on QA world but i love to learn.
I can record and login to my web app with no problem, but i have to login with 2 different users and i can’t , in my web the user name is not in the url, is in the parameters section, so i have created a variable to handle this,it works with the first user but when i reviewed the log to see why the secon uses cannot login, i saw the first user set a cookie and the second don’t , i don’t know why, both users has the same password,and are sending the password correct.
Please help me, i would like send you screenshots if you want.

Also, please run the 2 sessions test with 2 users with full logging enabled. After the test is completed, save the logs (“File | Save logs” on the menu). Please also send the log files to the me to the above address. This will let me further suggest on the problem.

Possibly this is something I have not mentioned exactly in the article… Further test phases do show higher numbers for pages/hits per second, however we only consider stages with acceptable response time. As mentioned earlier in the analysis, it starts to grow significantly after 30:00.

If you do not consider this, the best performance numbers would be 5.93 and 122 respectively, which are not much higher though.

I am running load test for my web application I have some clarifications..Can you answer the below questions?

1.What are the major parameters to be considered while doing performance testing for an web application?
2.What should be the min/max range of CPU usage,Memory and Response time?
3.How to conclude that a certain amount of values(measurement values) is good enough for uploading a file?
4.How to calculate CPU usage,Memory and Response time of an application(Which one should increase and which one should decrease if we upload three different volume of file say for an example:file 1 – 50 records,file 2 – 500 rec and file 3 – 1000 rec.) the above three param should increase or decrease?

1. This is a complex question. In short, first of all you need to check if the test is executed correctly. After that you can check error rate and response times for various requests. Other parameters are used to troubleshoot specific problems.

2. CPU should never go to 100% for longer than few seconds. Response time requirements depend on your business goals. In some applications a 5 second response time may be considered acceptable. In other applications this is way too long. You should also pay attention on how that time depends on the created load. Ideally it should not change significantly within the range of normal expected load.

3. When you test file uploads, you should rather pay attention to the amount of data sent per second. Make sure that you have sufficient bandwidth for such testing.

4. Greater load usually increases CPU usage and can also increase response times, however the exact dependence is application-specific. In any case your goal is to check that when you create realistic load, any parameters that affect user experience when working with the application stay within the acceptable range. To check that you can also conduct manual testing while the application is under load.