Transcription

2 New user behaviors that businesses are introducing to their web sites are driving more volume than ever before. Product promotions and shifting business models can trigger radical spikes in user traffic and expose issues not encountered at more traditional levels. This new, unexpected workload can result in a flash or burst and spike traffic to dramatically high levels in a short period of time. Why does this happen? For instance, marketing departments often neglect to inform IT departments of aggressive campaigns. Businesses often don t take the key steps to ensure that IT can support these campaigns. Through Groupon, the Gap (one of the top 25 most visited retail web sites) offered $50 of merchandise for $25 and consumers bought more than 441,000 vouchers. Amazon (the number one Internet retailer) partnered with LivingSocial to make $20 vouchers available for $10 and shoppers scooped up more than 1.3 million of them. As a membership reward, Gilt and Rue La La schedule a limited time, private sale of high-end merchandise. Access patterns from mobile devices such as the iphone plus content aggregators such as Mint.com can drive system workloads to unexpected levels in a short period of time. These types of events can provide a cataclysmic rush to IT resources, and IT is often forced to react to a performance-engineering emergency instead of being prepared ahead of time. I. Changing online business workload Web sites that previously experienced smooth and steady traffic increases that could be accurately predicted over time are now subject to erratic day-to-day volumes and dramatic surges. IT groups are quickly trying to assess the impact to the infrastructure that these workload profiles generate. The load-testing infrastructure required to simulate these extreme burst loads can be prohibitively expensive to create and maintain. The traditional load testing approach of simulating virtual users can test many aspects of the system, but since the tests originate inside the company s network instead of on the Internet, many complicated factors are unable to be validated. This leaves a significant gap in the IT team s ability to accurately validate the web site s ability to absorb burst loads. A website is a complex system with many components. These components may not have been designed to handle sudden and extreme bursts of volume. As components in the system become stressed and begin to fail under this burst load, the web site often becomes unstable and users of the site are subjected to longer and longer response times. As this occurs, customer experience with the site will descend from satisfied to tolerable to frustrated and finally to abandonment. 2

3 This begs the question: how does your web site s response time change under sudden and increasing volume? Does it reach a tipping point and fail completely? Does it accept all requests and then do all users experience a significant slowdown? Do your best customers see the same slowdown as those just browsing or are they all treated equally? These questions can only truly be answered by validating your web site during burst levels. II. Traditional performance testing approach The traditional approach to performance and scalability testing relies upon a relatively small number of virtual users to generate the same number of transactions per second that the larger real user population is expected to generate. These virtual users are browserless, that is, the test application records the HTTP requests that a browser will make and stores them in a script that will be executed during a test. The virtual users execute inside the corporate firewall and often access a test environment that is also within the corporate network. This type of test is most effective for stressing the application and database servers but falls short of simulating the real production environment, or real world user experience. Figure 1: Flash events, and cloud performance testing measure the actual experience the end user will have while the system is under load. In most cases, all of this testing occurs in a test environment which is designed to mimic the production environment but is usually not an exact copy. In addition to load testing in a test environment, many organizations use third party monitoring services in production to characterize the real experience users have throughout the day. These services, provided by companies such as Keynote Systems or Gomez simulate users from across the Internet and record the results. A virtual user profile is designed and then executed from several locations around the country. These companies provide you daily reports of the end user experience. III. Responding to the new workload IT should consider some important questions in analyzing whether their performance strategy is on track to handle future needs, such as: Have you tested your web site for a burst situation? Have you measured what the user experience will be and can you communicate this to your business? Given the new workload to the web site, how does the performance engineering team respond? Do they have the ability to define and execute the volume of test scenarios required to simulate a burst for instance the load that 100,000 users will generate? How will the Internet between your corporate firewall and their browser impact the customer experience on your web site? Other techniques are used to simulate different aspects of the system and attempt to understand the user experience while the system is heavily loaded. Sometimes a network tool may be used to constrain the available bandwidth during the test for some of the virtual users and to simulate conditions outside the corporate firewall. Additionally, a functional testing tool may be employed during the performance test to exercise the web browser and to Can your load-balancing strategy accommodate a partial failure in one site while maintaining overall user experience? The answers to these questions may drive the performance team to assess or reassess their ability to implement a burst load scenario. The sheer scale and complexity of designing an environment capable of simulating burst loads is daunting. The virtual user often licenses traditional load testing tools. Simulating more virtual users requires additional licenses, which must be borrowed from other 3

4 parts of the organization or, in many cases, the team must work with their load testing tool vendor to buy temporary licenses. Some of the traditional vendors may offer a lease-based licensing model. This is usually a time based license for a much higher number of virtual users, though supplying the hardware to generate the increased number of users remains a challenge. To execute this burst load volume the team will also have to procure or buy or borrow enough hardware to run these virtual users. All of this requires significant coordination, overhead and leadtime. Setting up and managing this environment to support these virtual users can be prohibitively expensive. Burst load tests will likely occur less frequently than traditional performance tests, which means the licenses and servers will sit idle a majority of the time further reducing the economic attractiveness of this option. IV. Inside-in versus outside-in A vast majority of performance and stability tests are executed within the firewalls of the company or inside-in. This has the advantage of allowing the performance teams to run tests in a repeatable and controlled process in a closely managed environment. Unfortunately, the number of components between the corporate firewall and the customer s browser is large and is growing more complex every day. The types of traffic running over the Internet are growing and diverse. HTTP is no longer the only protocol on the Internet. Android and ipad/iphone apps are rapidly expanding in their diversity and proliferation and Skype and video conferencing are commonplace. Desktop sharing with WebEx and Join.me are becoming more popular. Video streaming from YouTube, Netflix, Amazon and other services are gaining market share from traditional bricks and mortar content delivery channels. Ignoring these variables leads to a false sense of security, while simulating them is nearly impossible. Web 2.0 sites typically have content sourced from multiple partners as well as from your own infrastructure. The only way to truly understand how your web site will handle burst loads across the Internet is to test from the Internet. V. Extending capability to the cloud Running large-scale volume tests from the cloud will help you uncover risks within your systems and provide you with response time measurement from outside your corporate firewall for your key business transactions. This will provide a key indicator of your customers expected web site experience during a burst load. The forward thinking performance team should incorporate cloud-testing capability into its toolkit. The cloud option will allow the performance team to quickly design and execute simulated burst tests on the web site by eliminating the lead time and overhead associated with building out the infrastructure to support such a test. The team should focus on value-added activities such as test design while cloud sourcing the overhead activities to the cloud provider. To build out cloud testing capability efficiently, several corporate processes and procedures will require change and the core performance processes, tasks and activities will need to adapt to this model. Teams must answer many questions such as how frequently will cloud testing be used? and what qualifies a web site to require cloud testing? Cloud-based burst tests against Internet-facing environments may closely resemble Denial of Service attacks to the network operations team, so before executing the burst performance test, you must coordinate with them. Burst tests from the cloud to a test environment will require firewall changes and will require review by the network security team (in order to efficiently monitor test environments). Sometimes sensitive data must be used to drive the performance test. The data formerly contained within the firewall will now be placed on the cloud. Risk and security teams must ensure that the processes are in place to protect this data. All performance tests need the ability to log in to the system undergoing the test; this necessitates having a population of test user ids for the application. The performance engineer must consider how to create the very large number of test accounts required for the cloud performance test. Creating 100,000 unique users is no simple task. A valid performance test requires realistic test data and oftentimes this is masked as production data. 4

5 Figure 2: Cloud testing All of these factors require careful planning to ensure successful cloud-based testing. There are a number of emerging options for leveraging the public cloud to extend your performance testing capability including SOASTA, CloudTest, Gomez RealityLoad, and HP LoadRunner in the cloud (currently Beta). Plus you can provision your own cloud service using Microsoft Azure, Amazon EC2 or Rackspace s public clouds and drive cloud-based loads using your own load generating software. Leveraging private cloud performance testing requires a significant financial commitment and may not be practical for your organization. Some vendors provide professional services along with their load-generating tool and actually participate in the test execution. They help design and execute the burst test with you leveraging their deep knowledge of their tool and experience with other clients. Other vendors provide only the tools and cloud infrastructure while the company designs and executes its own tests. Vendors such as SOASTA provide a dashboard along with their load test tool that reveals a real-time system level utilization and metrics in addition to the transaction response times all in one integrated console. Other vendors require you to build this capability yourself or to coordinate the system level metrics and the cloud test response times for correlation after the test. Whether you choose a fully inclusive vendor solution, grow your own using existing tools and public cloud infrastructure, or invest in a private cloud infrastructure, you will spend less time on expensive overhead activities such as managing licenses and provisioning physical servers and more time on the value-added activity of performance testing the system. VI. Conclusion Conventional performance testing plays a critical part of the ongoing process of validating the fundamental performance and stability of applications. Your current load-testing 5

6 environment allows you to test incrementally and to target key components. Testing with your current infrastructure at sub-burst levels can identify many issues. Conventional testing serves as the first gate in your comprehensive performance testing approach. Once the system has cleared the conventional testing gate, outside-in testing from the cloud at burst levels provides the best opportunity to efficiently validate your system at much higher volumes. Testing can be designed to validate performance at extreme burst levels or even to find the upper limit of a system s scalability and reliability before users see an unacceptable increase in response times and abandon your web site. The cloud performance testing market is rapidly emerging and undergoing changes every month. Many enterprises are reviewing the benefits that an internal cloud environment can provide. There will be the opportunity to leverage the internal cloud for performance testing in a shared services environment. The key to an internal performance lab is the utilization of the environment. A highly utilized (5 by 24) performance environment should be the goal of every performance team. While running load tests from the cloud cannot replace your current investment in traditional load testing, there are significant and real advantages to pursuing cloud based approaches. Both approaches work together to mitigate business risks related to system performance in a complementary way, and understanding both the risks of burst loads and the options to manage them will ultimately make the business of the web for both marketers and IT more successful. 6

7 Collaborative Consulting is a leading information technology services firm dedicated to helping our clients achieve business advantage through the use of strategy and technology. We deliver a comprehensive set of solutions across multiple industries, with a focus on business process and program management, information management, software solutions, and software performance and quality. We also have a set of offerings specific to the life sciences and financial services industries. Our unique model offers both onsite management and IT consulting as well as U.S.-based remote solution delivery. To learn more about Collaborative, please visit our website at us at or contact us at Copyright 2014 Collaborative Consulting, LLC. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. WP

Five Strategies for Performance Testing Mobile Applications Mobile applications are now mainstream business. Businesses ranging from healthcare providers to retailers and airlines are creating applications

Stop Cheating: Four Obsolete Performance Testing Work-Arounds Performance testing is often as much an art as a science. Traditionally, performance testing operates within such significant constraints that

8 Common Myths About Performance Testing in Production Don t Let These Misconceptions Keep You From Production Testing White Paper Table of Contents Myth #1: Production testing = live testing... 1 Myth

Consequences of Poorly Performing Software Systems COLLABORATIVE WHITEPAPER SERIES Poorly performing software systems can have significant consequences to an organization, well beyond the costs of fixing

NOUS INFOSYSTEMS LEVERAGING INTELLECT Mobile Performance Testing Approaches and Challenges ABSTRACT Mobile devices are playing a key role in daily business functions as mobile devices are adopted by most

PorposalPPP Q: Gomez is standlone web application testing tool? Gomez provides an on demand platform that you can use for both testing and monitoring your Web applications from the outside in across your

White Paper The Top 20 Tools Needed for Hybrid IT 1 IT today is more complex than ever. Public cloud, networks, storage systems, servers, and applications all combine to deliver today s IT services. But

Session 5. Mixing and matching Public, Private and Hybrid Clouds for maximum benefits Best of both/ Best of all regarding specific needs, based on the use of resources Hybrid cloud is simply a mix of private

Leading the Evolution WHITE PAPER TESTING FOR PEAK PERFORMANCE: How Cloud-based testing ensures that your applications can handle peak demand The business challenge of unavailable applications Thousands

Flexible business solutions move to the cloud Whitepaper Whitepaper Cloud computing is hardly the newest buzzword on the block but it has now entered business consciousness through its frequent appearances

Cloud 101: What is the cloud? What is the cloud? There have not been many phenomena in the history of IT that have received so much attention or hype as cloud computing. It seems odd, therefore, that this

Cloud computing an insight Overview IT infrastructure is changing according the fast-paced world s needs. People in the world want to stay connected with Work / Family-Friends. The data needs to be available

white paper TESTING FOR PEAK PERFORMANCE How cloud-based testing ensures that your applications can handle peak demand THE BUSINESS CHALLENGE OF UNAVAILABLE APPLICATIONS Thousands or even millions of customers,

Are You in Control of Your Cloud Data? Expanded options for keeping your enterprise in the driver s seat EXECUTIVE SUMMARY Hybrid IT is a fact of life in companies today. Increasingly, the way to deploy

Web Load Stress Testing Overview A Web load stress test is a diagnostic tool that helps predict how a website will respond to various traffic levels. This test can answer critical questions such as: How

WHITE PAPER: WEB PERFORMANCE TESTING Everyone wants more traffic to their web site, right? More web traffic surely means more revenue, more conversions and reduced costs. But what happens if your web site

Innovation: Add Predictability to an Unpredictable World Improve Visibility and Control of Your Telecom Network Judith Hurwitz President and CEO Sponsored by Hitachi Data Systems Introduction It is all

Remote Infrastructure Management Emergence of the Cloud-based Helpdesk A leadership perspectives white paper Recommended next steps for business and industry executives Issue 11 in a series Executive Summary

Cisco Wide Area Application Services Optimizes Application Delivery from the Cloud What You Will Learn The adoption of cloud-based computing and applications promises to improve the agility, efficiency,

Monitoring Best Practices for OVERVIEW Providing the right level and depth of monitoring is key to ensuring the effective operation of IT systems. This is especially true for ecommerce systems like Magento,

KEMP LoadMaster Enabling Hybrid Cloud Solutions in Microsoft Azure Introduction An increasing number of organizations are moving from traditional on-premises datacenter architecture to a public cloud platform

TecTakes Value Insight How to Turn the Promise of the Cloud into an Operational Reality By David Talbott The Lure of the Cloud In recent years, there has been a great deal of discussion about cloud computing

WHITE PAPER Four Virtual Meeting Room Deployment Strategies for Technology Companies: Which Strategy is Best for You? Firms can use the guidance offered in this paper to better understand four available

MOVING TO THE CLOUD: Understanding the Total Cost of Ownership The most significant cost of owning and managing software applications is tied up in operating and maintaining the software, along with the

datasheet Trend Micro deep security as a service Advanced Security Built for the Cloud Organizations are embracing the economic and operational benefits of cloud computing, turning to leading cloud providers

The Definitive Guide to the Cloud and Kentico CMS THOMAS ROBBINS Contents Introduction... 4 What is Cloud Computing?... 4 The Benefits of the Cloud... 6 Full Hardware Utilization... 6 Lower Power Costs...

Session 3. the Cloud Stack, SaaS, PaaS, IaaS The service models resemble a cascading architecture where services on a higher level, as identified by Weinhardt et.al. (2009); encapsulate functionality from

WHITE PAPER : WEB PERFORMANCE TESTING Why Load Test at all? The reason we load test is to ensure that people using your web site can successfully access the pages and complete whatever kind of transaction

Service Virtualization: Reduce the time and cost to develop and test modern, composite applications Business white paper Table of contents Why you need service virtualization 3 The challenges of composite

Monitoring Best Practices for COMMERCE OVERVIEW Providing the right level and depth of monitoring is key to ensuring the effective operation of IT systems. This is especially true for ecommerce systems

Any time a new technology gains rapid acceptance in the market, the hype can be overwhelming. Vendors tout technical bells and whistles. Industry publications spew numbers and reports. Analysts predict

Guide to AWS Brought to you by Welcome to Ingram Micro Cloud For more than 35 years, Ingram Micro a Fortune 100 company with $30 billion in annual revenue has been successfully connecting IT technology

Eight essential tips for a successful How a The move to the cloud is happening and it s happening now. But before you jump start your cloud migration project, be sure you understand how to adequately prepare

Insights ESG and Solvency II in the Cloud In this article we look at how the model of cloud computing can be applied to high performance computing (HPC) applications. In particular it looks at economic

Monitoring Best Practices for OVERVIEW Providing the right level and depth of monitoring is key to ensuring the effective operation of IT systems. This is especially true for ecommerce systems like Magento,

Inventory and Analytics for Browser-based Applications in the Enterprise Introduction Times are changing. Desktop and client/server business applications (collectively referred to as native applications

Migrating Within the Cloud, SaaS to SaaS A Real World Experience COLLABORATIVE WHITEPAPER SERIES COLLABORATIVE WHITE PAPER SERIES: Migrating Within the Cloud, SaaS to SaaS How do you know when a technology

Services Overview Cisco Cloud Enablement Services for Education Bringing the Cloud to the Campus In today s higher education environment, IT organizations must keep pace with a long list of competing demands:

HIGH-SPEED BRIDGE TO CLOUD STORAGE Addressing throughput bottlenecks with Signiant s SkyDrop 2 The heart of the Internet is a pulsing movement of data circulating among billions of devices worldwide between

Introduction to performance testing Silvia A Siqueira WW Sr. Product Mktg. Manager Delivering applications that perform well Think about building a fancy and huge tower and discovering later that it can

The Business Value of a Comprehensive All-in-One Data Protection Solution for Your Organization You have critical data scattered throughout your organization on back-office servers, desktops, mobile endpoints

GET CLOUD EMPOWERED. SEE HOW THE CLOUD CAN TRANSFORM YOUR BUSINESS. Cloud computing is as much a paradigm shift in data center and IT management as it is a culmination of IT s capacity to drive business

Economic Benefits of Cisco CloudVerse Key Takeaways Executive Summary Cloud architecture enables IT to be delivered as a service and delivered only when it is needed over the network from central, secure

WHITE PAPER Cloud computing: Marketing hype or sound business strategy? WHITE PAPER Cloud computing: Marketing hype or sound business strategy? Building a house is difficult and costly. That s why a family

ABSTRACT Crystal clear requirements before starting an activity are always helpful in achieving the desired goals. Achieving desired results are quite difficult when there is vague or incomplete information

Best practices for choosing and integrating a mobile payments platform A GlobalOnePay White Paper Mobile commerce (mcommerce) purchases and in-app payments made on mobile devices are rapidly becoming just

Cloud Computing for Small to Mid Size Businesses Tech66, LLC William Burleson wcb@tech66.com www.tech66.com Why Tech66 and the Cloud? You want to focus on your core business, not on running your IT infrastructure

The Business Case Migration to Windows Server 2012 R2 with Lenovo Servers New levels of integration and capabilities provide the foundation for building more successful businesses with this new infrastructure

A Unified View of Network Monitoring One Cohesive Network Monitoring View and How You Can Achieve It with NMSaaS Executive Summary In the past few years, the enterprise computing technology has changed

Economic Benefits of Cisco CloudVerse Executive Summary Cloud enables IT to be delivered as a service, when and wherever it is needed, from central, secure, public and private data centers. Enterprises

Process A Whitepaper Copyright 2006. Technologies Pvt. Ltd. All Rights Reserved. is a registered trademark of, Inc. All other trademarks are owned by the respective owners. Proprietary Table of Contents

HRG Insight: Cloud Computing Keeping apprised of terminology in today s constantly changing IT landscape can be a fulltime job for IT decisionmakers. Some terms lend themselves to a fairly educated guess

A Neotys White Paper Table of Contents Overview...3 Benefits of Load Testing with the Cloud...3 Perform Large Scale Tests...3 Perform More Realistic Tests...4 Save time and Reduce Costs with Pay-as-you-go...4

Service Virtualization A faster, more efficient and less costly way to develop and test enterprise-class applications As cloud and mobile computing gain rapid acceptance, IT departments are expected to

Putting Critical Applications in the Public Cloud The Very Latest Best Practices & Methodologies Business White Paper December 2011 Introduction Many organizations are beginning to realize that there are