Transcription

2 Today s Agenda Explore the benefits of a properly performed performance test Understand the basic elements of the performance test Learn how to design, execute, and measure an effective performance test 2

3 Performance Testing Definition A category of testing in which the system's performance requirements are measured and/or tested against specified service objectives. Performance/stress testing is performed by an independent test team, not by developers. 3

4 Benefits of Performance Testing 4

5 Benefits of Performance Testing Performance in Perspective Just 8 hours of downtime in a year equates to 99.9% uptime. At that performance: Two planes at O Hare airport would have unsafe landings each day 12 newborns would be given to the wrong parents each day 291 pacemaker operations would be performed incorrectly 315 entries in Webster s Dictionary would be misspelled 5

6 Benefits of Performance Testing Primary Goals of Performance Testing Utilize technology to provide repeatable, consistent testing of the System Under Test (SUT) Increase quality through greater scope of testing performed Conduct testing that is very difficult, no, impossible to do with just humans Ensure application infrastructure can handle desired volume 6

15 Performance Test Basics A Variety of Different Components Performance testing can be performed on both hardware and software, below are several places where most business will conduct this type of testing Software Applications Networks Databases Mainframes Firewalls Routers 15

16 Performance Testing Basics Functional vs. Performance Test Automation Functional testing is designed to insure that the application is working as designed from a screen to screen and control to control standpoint. Performance testing is intended to insure that the software AND hardware can handle the number of users or load of data that will make it stable for day to day use. Testing these look for 2 different results Functional Defects, buttons or controls don t work Performance Defects, when 2000 users hit this page, the server crashed 16

19 Performance Test Basics Test Design Options Load Testing Attempts to model the anticipated real world usage and utilization over a short period of time, with the expected number of users and average user interaction delay times. This is looking for the typical user experience. Baseline Initial test from which remaining tests are measured, typically measured with one user. Scalability / Capacity Testing by increasing the workload in an effort to determining stress points and to measure the limits of the capacity of the system. Stress and Hot Spot Testing Testing over a short period of time when the site is hit with larger than expected loads, requiring extensive computations/data retrieval. Here you are looking for how the system breaks down under stress. A variation is hot spot testing, where you focus the stress on a specific portion of the product, looking for a weak link. 19

20 Performance Test Basics Test Design Options Spike and Bounce Testing Testing with a sudden growth in load over a very short period of time, looking to see if the system can respond to abrupt changes in the workload. A variation is to follow the spike with a bounce down to a very low load level, and then continue repeating the up and down pattern. This tests whether the system can recycle its resources properly. Endurance Testing A load or stress test that is run for an extended period of time, typically several days, with the purpose of detecting slow-to-appear defects. This measures the reliability of the system. Integrity Testing Combines functional testing with stress testing to ensure that functionality, which worked under low volumes still works. 20

21 Performance Test Basics Test Objectives 21

22 Performance Test Basics Transaction Concurrency How many transactions will need to be run per minute if a load test has to be run for two hours with 5000 users, assuming an average transaction length of five minutes? Determine how many transactions run per minute: 120 min / 5 min = 24 iterations for each user 5000 users X 24 iterations = 120,000 transactions 120,000 transactions / 120 minutes = 1000 transactions per minute Apply the transactional concurrency to the application: The test is run during the 10 AM-12 NOON time slot The test should consist of 5000 users running 24 iterations The system must be able to handle 1000 transactions per minute 22

28 Design and Execute Analyze System Under Test (SUT) Map System Architecture Data Flow Identify key components within the system architecture Verify performance testing tool compatibility with the SUT Review System Architecture with Project Team 28

29 Design and Execute Vendor Evaluation and Selection Review potential vendors and conduct an informal analysis of their capabilities Publish requirements to selected vendors Confirm their interest and ability to proceed with a tool evaluation Select vendors for the evaluation process Complete the evaluation Validate selected vendor pricing and implementation costs Finalize vendor and proceed to implementation planning 29

31 Design and Execute Implementation Planning Define implementation requirements, both personnel and technical Choose a pilot application Identify available resources Develop a work plan for implementation which includes a process development effort Review the approach with all stakeholders 31

33 Design and Execute Plan and Acquire the Test Environment Confirm the source and volume of test data Performance testing can use large amounts of data Login IDs for all virtual users Data for all types of transactions Determine how test data will be obtained and used 33

34 Design and Execute Construct a Transaction Matrix A Critical Tool to Successful Design Identify user profiles to the System Under Test (SUT) Identify the transactions within each user profile Document Transaction Matrix 34

35 Design and Execute Construct a Transaction Matrix Analyze the SUT Review the business requirements documentation Identify the different transactions that could be performed in the application during peak load Create a Distribution Matrix Identify the probable different end-users who you think will be visiting the site Identify what transactions will be performed by each of the users you have identified 35

36 Design and Execute Benefits of a Transaction Matrix Tells the probable kinds of users to the system Tells which transactions are to be performed by each of the identified users to the system Helps in scripting the performance test cases, by easily making out what transaction is performed by each of the identified users to the system. 36

39 Design and Execute Test Scenario Selection It is not feasible to load test every transaction \ Business Process Identify transactions that are: Data intensive High Volume Mission Critical Concentrate on a peak time frame to find peak load Create a transaction matrix to identify possible scenarios and user profiles Typically 20% of the users will generate 80% of the load on a system so not all user types need to be represented. 39

40 Design and Execute Benefits of Performance Test Scenarios Identifies the load test objectives Identifies the number of scripts that are needed Identifies the number of load tests that are needed Document Scenarios 40

41 Design and Execute Creation of Performance Test Scenarios Analyze and document the performance requirements Interview the Business and Development personnel Review performance test objectives Review user profiles Review transactions to be performed by the users Review metrics Identify and document the Scenarios for the test 41

42 Design and Execute Creation of Performance Scenario Document Scenario document tells the probable percentage of users identified to the system The metrics identified for each of the users will be used while constructing a work load definition before executing the scripts Scenario document helps in creating a test suite execution plan enabling the smooth flow in the communication channel before executing the scripts. 42

46 Design and Execute Test Tool Components Most Performance or Load Testing Tools will be broken down into these 3 main pieces. They may be named differently, but they will perform the same action Controller Load Generator(s) Virtual Users 46

47 Design and Execute Test Tool Components 47

48 Design and Execute Test Tool Components 48

49 Design and Execute Test Scripting Techniques When creating a Performance Testing script you will use a method called Record & Playback. While discouraged in Functional Test Automation, it is the recommended method of script creation in Functional Test Automation The script will be recorded, then modifications will be made as necessary to add user data from files or databases to simulate different users at test time 49

50 Design and Execute Test Preparation Identify critical test sets to run from the Test Execution plan Generate detailed test execution records Identify needed changes to the suite as well as to the SUT Develop records to be reviewed during postimplementation review for individual as well as team learning 50

51 Design and Execute Test Execution Run the script with one virtual user to verify a baseline for the test Verify / schedule test script execution by communicating to the team members Run the script as per the scenarios developed 51

53 Design and Execute Test Analysis During the execution of a performance test, a failure does not always mean that a defect has been found. The result will need to be analyzed to find the root cause: A test tool problem may arise A piece of hardware may be down for maintenance during the test or unavailable The data source may be unavailable or have incorrect data in it 53

55 Design and Execute Test Analysis Sample Results Graph Number of visitors increases But KB per second and transaction per second falls Because transactions start to fail 55

56 Design and Execute Test Evaluation Verify that all requirements were met Identify and evaluate any outages If there are any items that need action, decide on the best way of addressing the items. Options include: do not close the project define a follow-on project initiate a maintenance process. Consider how the Test Suite will be maintained 56

57 You ve Succeeded Now What? Ensure that the software license quantities and types meet the needs of the automated testing initiative Plan a workforce model for current and future needs Provide for ongoing process and technical reviews Help the organization understand the value of test automation 57

58 You ve Succeeded Now What? Annual review of the technology plan Will architectures change? Will application usage demand and patterns change? Do user licenses and types of automated testing software match the technology plan? 58

59 You ve Succeeded Now What? Workforce Management Understand the unique skills that are needed to support effective use of the tool Implement the organizational model that works Center of Excellence or team oriented Utilize techniques to attract and hold a skilled workforce Keep the workforce capable 59

61 Bringing Value with Performance Testing Review Explore the benefits of a properly performed performance test Understand the basic elements of the performance test Learn how to design, execute, and measure an effective performance test 61

62 Bringing Value with Performance Testing Benefits Utilize technology to provide repeatable, consistent testing of the System Under Test (SUT) Conduct testing that is very difficult, no, impossible to do with just humans Ensure application infrastructure can handle desired volume Minimize Downtime and Loss of Revenue 62

63 Bringing Value with Performance Testing Basic Elements Performance testing is intended to insure that the software AND hardware can handle the number of users or load of data that will make it stable for day to day use. Performance testing is typically performed on multiple infrastructure components There are several different design options for performance tests 63

Technical white paper A closer look at HP LoadRunner software Table of contents Sizing up the system 2 The limits of manual testing 2 A new take on testing: the HP LoadRunner solution 3 The HP LoadRunner

Application Performance Testing Basics ABSTRACT Todays the web is playing a critical role in all the business domains such as entertainment, finance, healthcare etc. It is much important to ensure hassle-free

ABSTRACT Crystal clear requirements before starting an activity are always helpful in achieving the desired goals. Achieving desired results are quite difficult when there is vague or incomplete information

Process A Whitepaper Copyright 2006. Technologies Pvt. Ltd. All Rights Reserved. is a registered trademark of, Inc. All other trademarks are owned by the respective owners. Proprietary Table of Contents

Performance Testing Definition: Performance Testing Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device. This process can involve

Welcome to this introduction to application performance testing and the LoadRunner load testing solution. This document provides a short overview of LoadRunner s features, and includes the following sections:

TRACE PERFORMANCE TESTING APPROACH Overview Approach Flow Attributes INTRODUCTION Software Testing Testing is not just finding out the defects. Testing is not just seeing the requirements are satisfied.

Rapid Bottleneck Identification TM A Better Way to Load Test WHITEPAPER You re getting ready to launch or upgrade a critical Web application. Quality is crucial, but time is short. How can you make the

White paper: Unlocking the potential of load testing to maximise ROI and reduce risk. Executive Summary Load testing can be used in a range of business scenarios to deliver numerous benefits. At its core,

A white Success The performance testing helped the client identify and resolve performance bottlenecks which otherwise crippled the business. The ability to support 500 concurrent users was a performance

Sample Exam ISTQB Expert Level Test Automation - Engineering 2014 Version 1.0 Copyright Notice This document may be copied in its entirety, or extracts made, if the source is acknowledged. Table of Contents

PorposalPPP Q: Gomez is standlone web application testing tool? Gomez provides an on demand platform that you can use for both testing and monitoring your Web applications from the outside in across your

IBM RATIONAL PERFORMANCE TESTER Today, a major portion of newly developed enterprise applications is based on Internet connectivity of a geographically distributed work force that all need on-line access

Choosing A Load Testing Strategy Why and How to Optimize Application Performance What Is Load Testing? Systematic exposure of an application to real world, expected usage conditions before deployment Analyzes

Resource Monitoring During Performance Testing Experience Report by Johann du Plessis Introduction During a recent review of performance testing projects I completed over the past 8 years, one of the goals

Web Testing Introduction Making a web site does not end with putting all the media and software together. Actually, web site work never ends. When all the design is done, you have to test the site first

Test What You ve Built About Your Presenter IBM i Professional for 16 Years. Primary Focus is IBM i Engineering / Programming Well Versed in 2E. Well Versed in RPG (All Flavors) Well Versed in CM Products

Rapid Bottleneck Identification A Better Way to do Load Testing An Oracle White Paper June 2009 Rapid Bottleneck Identification A Better Way to do Load Testing. RBI combines a comprehensive understanding

Process of Performance Testing a Banking Application The Need One of the largest banks in India offers Online and Mobile Banking application to its customers. Several online services are hassle-free services

An Oracle White Paper February 2010 Rapid Bottleneck Identification - A Better Way to do Load Testing Introduction You re ready to launch a critical Web application. Ensuring good application performance

Performance Test Summary Report Skills Development Management System December 2014 Performance Test report submitted to National Skill Development Corporation Version Date Name Summary of Changes 1.0 22/12/2014

White Paper Performance Testing Methodology by Johann du Plessis Introduction One of the main concerns with a performance testing project is how much value the testing adds. Is performance testing worth

Summer Internship 2013 Group No.4-Enhancement of JMeter Week 1-Report-1 27/5/2013 Naman Choudhary For the first week I was given two papers to study. The first one was Web Service Testing Tools: A Comparative

Using WebLOAD to Monitor Your Production Environment Your pre launch performance test scripts can be reused for post launch monitoring to verify application performance. This reuse can save time, money

Performance Testing on Production System Abstract Performance testing is conducted to check whether the target application will be able to meet the real users expectations in the production environment

Solution Offering Infosys RADIEN CASPER Capacity Assessment and Engineering Framework (CASPER) Enterprises recognize the importance of performance as a key contributor to the success of any application,

NOUS INFOSYSTEMS LEVERAGING INTELLECT Mobile Performance Testing Approaches and Challenges ABSTRACT Mobile devices are playing a key role in daily business functions as mobile devices are adopted by most

Service Virtualization: Reduce the time and cost to develop and test modern, composite applications Business white paper Table of contents Why you need service virtualization 3 The challenges of composite

Web Load Stress Testing Overview A Web load stress test is a diagnostic tool that helps predict how a website will respond to various traffic levels. This test can answer critical questions such as: How

WHITE PAPER: WEB PERFORMANCE TESTING Everyone wants more traffic to their web site, right? More web traffic surely means more revenue, more conversions and reduced costs. But what happens if your web site

Levels of Software Testing There are different levels during the process of Testing. In this chapter a brief description is provided about these levels. Levels of testing include the different methodologies

Table of contents 1. Agent Technology...3 1.1. Has the Knoa agent been tested with standard set of services on the PC?... 3 1.2. Do users need to do anything to activate the Agent?...3 1.3. Does the Knoa

Chapter 5 Regression Testing of Web-Components With emergence of services and information over the internet and intranet, Web sites have become complex. Web components and their underlying parts are evolving

What Is Specific in Load Testing? Testing of multi-user applications under realistic and stress loads is really the only way to ensure appropriate performance and reliability in production. Load testing

Load Testing an Internet Site Course Overview Course Title Load Testing and Performance Tuning Methods for Optimizing Web Site Performance Description Maximizing Web site performance is essential for those

To ensure the functioning of the site, we use cookies. We share information about your activities on the site with our partners and Google partners: social networks and companies engaged in advertising and web analytics. For more information, see the Privacy Policy and Google Privacy &amp Terms.
Your consent to our cookies if you continue to use this website.