Transcription

1 PorposalPPP Q: Gomez is standlone web application testing tool? Gomez provides an on demand platform that you can use for both testing and monitoring your Web applications from the outside in across your users, browsers, mobile devices and geographies using a global network of 100,000+ locations, 168+ countries and 2,500+ local ISPs. Reality Load is Gomez s web load and performance testing solution and you can reuse the same scripts for both load testing and production monitoring Q: What is the difference between the other load testing tools which enables the wan emulation, location based load testing and Gomez load testing? A few key differences are the scale of the Gomez offering, the number of locations that you can test from and the fact that it is not emulating a WAN. In terms of scale, the Gomez Last Mile network includes 100,000+ peers located in countries on every continent. (Reality Load runs on real user s machines on the local ISPs and connection speeds available to them). The Gomez network is already deployed and requires no configuration other than to choose which computers (we called them peers) you want to use for load testing or monitoring purposes.. This gives you an unprecedented view into user experience that cannot be gained by simply emulating different configurations. If you are interested in how your real users would experience your site under load you need to realistically test under the same conditions, and the Gomez Last Mile is the only way to do that. For example with Gomez Last Mile you can find geographical response time discrepancies that may surface only under load What will be the impact on the business if average response time stays under 4 seconds (your target goals), but users from key markets are seeing response time of 10+ seconds? Q: How would we use Gomez for the internal applications...not the external app? Reality Load, and the rest of the Gomez platform, is designed to test and monitor your applications across the internet, from an outside in customer perspective. But testing internal applications is a common request. At present there are a couple of options. One would be to allow the Gomez test machines access to the internal system, either by allowing requests from our IPs or by indentifying our agent string. Another option currently available for our Monitoring offerings, though not our load testing, is to deploy a Gomez private peer behind the firewall from which the performance measures could be taken. Q: Also what will be the approach of Gomez while testing any web based application running on a mobile phone? For testing mobile applications Gomez has 2 complimentary offerings, Reality Load (discussed in our webinar) and Active Mobile for monitoring mobile web applications. Users could run high volume load test using Reality Load, including device specific user agent strings and HTTP headers, while utilizing Active Mobile for the most accurate measure of end user experience. Active Mobile emulates devices across major mobile carriers in the US (Chicago, Boston, Seattle), the UK, Germany and China, giving an accurate view of the impact of the increasing load on the mobile user experience. Q: So the agent actually mimics browser behavior, is not actually running the scripts inside browsers? The browsers used by Gomez s Reality Load are real Web browsers. That means that they are actually running scripts and processing the pages, not just playing back HTTP type requests. Q: does it support Virtual private network usage? VPNs are not supported for load testing at this time. Gomez, Inc. 10 Maguire Road, Suite 330 Lexington, MA Main: Fax:

2 Q: Also does this tool monitors servers during the test run and co relate the bottle necks? Today most of our RL customers (like Jim from Dollar Thrifty mentioned during the Webinar) are using existing in house system mgmt tools to correlate performance inside and outside the firewall during a load test. The Gomez platform has an open integration philosophy, and we will continue to integrate with any EMS tools that you may be using. Now that Gomez is a Compuware Division we are looking into offering an integrated view of application performance under load by connecting the Gomez platform and Compuware Vantage behindthe firewall infrastructure monitoring capabilities, so you can quickly correlate bottlenecks and performance issues across both the enterprise and the Internet. Q: Do you use a 3rd party load test tool behind the GOMEZ Reality load tool? Something like HP LR, RPT or a combination of it? The Gomez load testing tool uses our own tool set. It uses the same script recorder and test agent as the rest of the Gomez Platform. This makes it very easy to move between load testing and 24/7 monitoring and vice versa. Q: Can the raw data be used for the CompuWare Load testing toolsets? The raw data is exported in CSV files, so it accessible to a wide range of toolsets. Q: Gomez software is opensource? Is there a trial version? it runs in any OS or web explorer? The Gomez platform is a SaaS platform accessible on demand through a web browser that means that you don t need to allocate hardware or install any software products, and relies on. The Gomez network 100k+ testing locations, 168+ countries, 2,500+ ISPs. It is not opensource, but if you are interested in a trial just let us know and we can set you up. Q: On your interface, when and why would you check the checkboxes, Exclude JS, CSS or Images? This was a feature requested by a number of our customers. Basically, it can serve two purposes, one is around test methodology and the other around cost. Methodology wise, we have customers who prefer to start their testing with simple steps, focusing very specifically on pieces of their applications. This focused approach starts with hitting just the basic pages and not any of the ancillary content like images, JS, CSS that may not be needed. Here they are focusing just on the impact of the requests for their application pages on the server under load. The second reason is cost. Many of our customers use CDNs that charge by the amount of content served. When running a lot of load tests the costs can get pretty high. So they will exclude the objects from their early tests and only include them for a final test or two. Also, just wanted to highlight that we see a lot of issues with third party providers and external components performing poorly, especially under load, and regional discrepancies as well, so don t forget to include test cases that cover external components and geographical testing into your plans We also allow users the option to exclude requests by host. Q: Are you taking care of that pages may be cached in a transaction? If not, can you be sure that the front end is not getting too much load? Our load test tool performs as a real user s browser would perform for each transaction. So if images are cached for instance, each iteration of each user run would request the object once, but not again within a transaction. 2

3 Q: If you are testing in the PRODUCTION environment during off peak hours, you still cannot know how much USER traffic is hitting your site. So you cannot isolate your results to your test. How do you account for UNKNOWN load during your test? This is definitely a concern for testing of production systems. Here it is important to monitor the utilization of the production system prior to the load test, during the load test and after the load test. This can give you a profile of the live traffic and assist in understanding the utilization on the servers. Also, by monitoring things like firewall bandwidth (at the network level), and comparing the overall total with the load reported for the test users (from the test tool) the load from live users can be extrapolated. For instance, if Reality Load was showing that the test was consuming 80 Megabits per second of bandwidth, and the firewall statistics were showing 90 Megabits per second of overall traffic, then you can get a sense of how much live traffic was on the site (in this case, 10 mbps worth of traffic). The same could be said for page request rates or other measures with the difference between the load tool measures and the basic system measures providing the sense of live traffic. The best way to ensure quality of experiences and that an application scales properly under load is to realistically test your application all the elements of the web application delivery chain from the outside in, using real users and desktops worldwide to access your production environment Q: How do you calculate transactions per second? One of the Reality Load metrics is transactions per minute, I assume this is what the question refers to. This is calculated as the sum of the transactions (complete runs of the test scripts) complete in each minute. Q: Do you help customers identifying workload profile, if yes what is your approach? Most of our customers operate in a self service model and utilize their own test methodologies. However we do offer collateral around best practices for things like workload profiling as well as offering professional services assistance where needed or requested.. In general though, the approach is to evaluate the traffic the web application is currently seeing with focus on three key areas: the most heavily hit pages/functionality, the most business critical pages/ functionality and any pages/functions known to cause problems. Q: Can you terminate a test run when things start to break? Does the initiator of the test have access to the test admin console? Can they terminate a test in a timely way to avoid crashing the application? The Gomez tool is a self service platform, so users have access to the controls to schedule a test, stop it, view real time results, etc. Tests can be stopped within a few seconds just by clicking a stop button. In addition, Reality has a Max Failure Rate settings (Reality Load checks that the % of users encountering errors at any given minute remains below the Max Failure Rate value) and it can automatically reduce the load level looking for essentially a last known good state if needed. Q: We've got a staging environment that matches production and it has a web server in the DMZ and is accessible from the Internet. We use Loadrunner to test the system. What is the advantage of using Gomez? Traditional inside the firewall testing tools, like LoadRunner, are built on the philosophy of generating load and measuring performance behind the firewall. This approach is valuable, but will only tell you part of the story, since these tools are focused on testing internal systems. However, your end users don't live in data centers they are located at the end of a long and complex web application delivery chain. When you generate load and measure performance behind the firewall you have no visibility into end user experience, or how 3rd party 3

4 components (ISPs, CDNS, etc) will scale under load. And all components and services need to perform optimally in order to deliver quality web experiences across all users and regions. Some examples of problems that you could be missing are: Inconsistent geo performance especially under load, CDN configuration issues or oversubscribed POPs, major ISPs outages, SMS routing, latency issues, etc Q: Did you support flex applications? Yes, Gomez supports flex applications. Q: Who does the scripting? Do customers write own scripts or do you write the scripts/scenarios for the customer? Specifically scripting web services The Gomez Platform, including Reality Load is a self service platform, and so most of our customers are building their own scripts for load testing and monitoring. However, there is a service desk option for free assistance with script creation if our customers run into any difficulties. Q: What language are scripts written in? The scripts for the Gomez Platform are recorder through a visual interface. Users click through their application and the Gomez recorders recorder the users interactions with the web application. When the scripts are played back, the agent builds out the page (document object model, javascript and CSS processing etc) and then replays the recorded actions. Behind the scenes, the scripts are saved as XML, and can be extended using javascript, but there is no scripting language per se. Q: How do you create the scripts (clicking through the site)? Yes, scripts are recorder by clicking through the site. Q: How easy is to script Q: How difficult is it to create testing scripts and what tools are used for that task? Scripting is relatively simple and uses the Gomez Script recording tools for script creation. It involves just clicking through the web site while the Gomez recorder captures your actions. When the scripts are played back the agent starts at the first URL and then executes the recorded actions. Q: Can we capture screens of application at predefined points in the script...? Reality Load allows users to collect screen captures on error (SCoE) in load tests. Q: Do you have any facility for testing client side rendering using javascript or ajax? The test users are executing the javascript, ajax calls, etc during the loadtesting, so those metrics are already included in the data. Q: Raw data exported is going to be cumulative of the ser responses how often it has been collected (or) for every user for every request until the end of the test? Reality Load allows users to export with the aggregated (minute by minute) data, or the full raw data set. The full raw data set includes data on every hit of every page request of every transaction run in a load test. Q: Is the summary report export format in XML? The summary report for Reality Load is exported as a PDF, though all of the data behind the reports can be exported in CSV files. 4

5 Q: How did you insure that you correctly backed out the scripted transactions without losing any real customer traffic data. Gomez transactions are easily identifiable by text in the request header. This text can be used to ensure that traffic monitoring tools do not count the load test requests in their reports. Q: what was the size of Page 0 (the megabits/s seem to grow really quickly is this representative of real life use?) Page 0 in the demo was a 250KB page. This is pretty common for a homepage, and in fact smaller than many I have seen recently. Q: Can the results point to database transaction bottlenecks? Interpreting load test results is a combination of effective test planning, collecting the right metrics and proper analysis. Looking at the user experience metrics, and combining that with data collected from the back end systems can point to a the location and likely cause of a bottleneck. For example, if only certain pages in a transaction are seeing performance degradation the first step would be to ask what they have in common. If they all make database hits then check the CPU of the database server. Is it within tolerable parameters or was it under duress. This correlating of functions exercised, user experience changes and server metrics can be highly effective in isolating performance bottlenecks. Q: Our test environment is significantly slower than our production environment. How do you load test in that scenario and insure that your production environment can actually handle the load? In scenarios where a test system and production system are not identical (and this is often the case) it is important to understand the 'factor of scalability', or the degree of difference between the two systems. This may take a little testing, or some estimation. In this way you could execute your test on your staging system and project them out to the production capacity. However, unless you actually run a test against your production system you'll never know for sure. I've seen many situations over the years where tests on staging go well, and then production fails under live load because of a missed configuration, unexpected change, etc. So despite the difficulties of testing against a production environment, I believe it s critical to run at least some tests against it before the customers do... Q: General load test question: So would starting with 50 users for 15 minutes and then running 100 for 15 minutes and so forth to see where the throughput breaks down as far as number of users...would this be a viable test My preference in load testing is to bring users on slowly over time, so that when a problem is encountered, you'll know exactly the load level that can be supported. In the example presented here, if the site could handle 75 users you wouldn't know that for sure. You'd have a 'yes' for 50 users, but only a no for '100'. I would start at 1 user and ramp to 50 over 10 minutes, then hold at 50 for 5 minutes (or until some transactions can complete at this load level), then ramp to 100 over another 10 minutes and hold again at 100. That way you would see the impact of increasing load as well as the flat load. Q: Is throughput test is the same name as "Stress test" for you? There is some overlap in the terms, but the goal of a throughput test isn t necessarily to push a system to the breaking point. The goal of a throughput test is to measure the system s ability to deliver pages, hits, bandwidth, orders etc (things that are typically measured in terms of metric per second/minute/hour), independently of how many users are running. So typically in a throughput test a small number of users run through transactions very quickly (no think time delays). This can uncover bottlenecks or validate capacity 5

6 with fewer users and less time than larger tests (what we would call concurrency tests), which are focused on measuring performance in terms of the number of users the system can support. Q: Does this product only test pure http traffic or can it handle new technologies like flex, java scripts, ajax etc and be able to emulate the client side processing as well? Reality Load uses the same transaction engine as the rest of the Gomez platform. This agent can handle flex, java script, ajax, etc and client side processing Q: Do your browsers include mobile devices such as the iphone and Blackberrys (on simulated 3G connections)? Presently Gomez supports mobile simulation for Monitoring, but not for load testing. The monitoring is simulating connections through modems on a all major carriers in the US (Chicago, Boston, Seattle), and in UK, Germany and China. Stay tuned for our Reality Load 2010 plans. Q: Also does this supports Citrix testing? Citrix is not supported at this time. Q: Does support security sites like https? Yes, HTTPS is supported by all of the Gomez products including Reality Load 6

WHITE PAPER : WEB PERFORMANCE TESTING Why Load Test at all? The reason we load test is to ensure that people using your web site can successfully access the pages and complete whatever kind of transaction

Testing & Assuring Mobile End User Experience Before Production Neotys Agenda Introduction The challenges Best practices NeoLoad mobile capabilities Mobile devices are used more and more At Home In 2014,

Testing in the Cloud A live demonstration of deploying testing in the cloud 19-July Sydney, Australia 21-July Melbourne, Australia Using the Cloud: A Practical View into the Cloud world The organisations

Rapid Bottleneck Identification TM A Better Way to Load Test WHITEPAPER You re getting ready to launch or upgrade a critical Web application. Quality is crucial, but time is short. How can you make the

Rapid Bottleneck Identification A Better Way to do Load Testing An Oracle White Paper June 2009 Rapid Bottleneck Identification A Better Way to do Load Testing. RBI combines a comprehensive understanding

An Oracle White Paper February 2010 Rapid Bottleneck Identification - A Better Way to do Load Testing Introduction You re ready to launch a critical Web application. Ensuring good application performance

NOUS INFOSYSTEMS LEVERAGING INTELLECT Mobile Performance Testing Approaches and Challenges ABSTRACT Mobile devices are playing a key role in daily business functions as mobile devices are adopted by most

Web Load Stress Testing Overview A Web load stress test is a diagnostic tool that helps predict how a website will respond to various traffic levels. This test can answer critical questions such as: How

WHITE PAPER: WEB PERFORMANCE TESTING Everyone wants more traffic to their web site, right? More web traffic surely means more revenue, more conversions and reduced costs. But what happens if your web site

Performance In The Cloud Copenhagen, 25 th Nov, 2010 Richard Stone Copenhagen 11-25-10 FINAL.pptx Market Landscape The new (post-recession) world Re-connect with Customers Work Anywhere is What We Do Business

Businesses are increasingly realizing the need for wide-ranging functional testing of mobile apps, as well as per formance, load and user experience testing, although most enterprises do not test a l l

A Neotys White Paper Table of Contents Overview...3 Benefits of Load Testing with the Cloud...3 Perform Large Scale Tests...3 Perform More Realistic Tests...4 Save time and Reduce Costs with Pay-as-you-go...4

Business white paper Load factor: performance testing for Web applications Table of contents 3 A look at load testing 3 In the pursuit of predictability 4 Put your apps through the paces 5 Traits of an

Bringing Value to the Organization with Performance Testing Michael Lawler NueVista Group 1 Today s Agenda Explore the benefits of a properly performed performance test Understand the basic elements of

Process A Whitepaper Copyright 2006. Technologies Pvt. Ltd. All Rights Reserved. is a registered trademark of, Inc. All other trademarks are owned by the respective owners. Proprietary Table of Contents

Copyright www.agileload.com 1 INTRODUCTION Performance testing is a complex activity where dozens of factors contribute to its success and effective usage of all those factors is necessary to get the accurate

Connect Intelligence Performance intelligence for your application delivery chain BT Connect Networks that think Connect Intelligence is a managed service from BT based upon Compuware Technology that provides

Five Strategies for Performance Testing Mobile Applications Mobile applications are now mainstream business. Businesses ranging from healthcare providers to retailers and airlines are creating applications

Open Source and Commercial Performance Testing Tools Palla Vinod Kumar Accenture Delivery Center for Technology in India Accenture, its logo, and High Performance Delivered are trademarks of Accenture.

www.agileload.com There is no second thought about the exponential increase in importance and usage of mobile applications. Simultaneously better user experience will remain most important factor to attract

IBM RATIONAL PERFORMANCE TESTER Today, a major portion of newly developed enterprise applications is based on Internet connectivity of a geographically distributed work force that all need on-line access

Performance Testing Definition: Performance Testing Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device. This process can involve

Application Performance Testing Basics ABSTRACT Todays the web is playing a critical role in all the business domains such as entertainment, finance, healthcare etc. It is much important to ensure hassle-free

Cisco Application Networking for IBM WebSphere Faster Downloads and Site Navigation, Less Bandwidth and Server Processing, and Greater Availability for Global Deployments What You Will Learn To address

Web Application Monitoring Service The Value of Measuring End-User Experience from a Global Point of Presence White Paper from www.site24x7.com Web: Email: support@site24x7.com Table of Contents 1. Introduction

Best of Breed of an ITIL based IT Monitoring The System Management strategy of NetEye by Georg Kostner 5/11/2012 1 IT Services and IT Service Management IT Services means provisioning of added value for

NOT ALL END USER EXPERIENCE MONITORING SOLUTIONS ARE CREATED EQUAL COMPARING ATERNITY WORKFORCE APM TO FOUR OTHER MONITORING APPROACHES COMPREHENSIVE VISIBILITY INTO END USER EXPERIENCE MONITORING REQUIRES

OPTIMIZING APPLICATION MONITORING INTRODUCTION As the use of n-tier computing architectures become ubiquitous, the availability and performance of software applications is ever more dependent upon complex

Performance Test Summary Report Skills Development Management System December 2014 Performance Test report submitted to National Skill Development Corporation Version Date Name Summary of Changes 1.0 22/12/2014

Planning the Migration of Enterprise Applications to the Cloud A Guide to Your Migration Options: Private and Public Clouds, Application Evaluation Criteria, and Application Migration Best Practices Introduction

27 th March 2015 Istanbul, Turkey Performance Testing Best Practice Your Host.. Ian Molyneaux Leads the Intechnica performance team More years in IT than I care to remember Author of The Art of Application

NOT ALL END USER EXPERIENCE MONITORING SOLUTIONS ARE CREATED EQUAL COMPARING ATERNITY WORKFORCE APM TO FOUR OTHER MONITORING APPROACHES COMPREHENSIVE VISIBILITY INTO END USER EXPERIENCE MONITORING REQUIRES

ABSTRACT Crystal clear requirements before starting an activity are always helpful in achieving the desired goals. Achieving desired results are quite difficult when there is vague or incomplete information

WHITE PAPER: WEB PERFORMANCE MANAGEMENT The Top 10 Reasons Why You Need Synthetic Monitoring A complete view of the application delivery chain (ADC) is required to optimize the performance and availability

ExtraHop and AppDynamics Deployment Guide This guide describes how to use ExtraHop and AppDynamics to provide real-time, per-user transaction tracing across the entire application delivery chain. ExtraHop

RUN THE RIGHT RACE Keep pace with quickening release cycles. Discover automation with the human touch. CHOOSE A TEST TO RUN BELOW 26032015 FUNCTIONAL TESTING With Borland everyone from business analysts

1 Serving 4 million page requests an hour with Magento Enterprise Introduction In order to better understand Magento Enterprise s capacity to serve the needs of some of our larger clients, Session Digital

Welcome to this introduction to application performance testing and the LoadRunner load testing solution. This document provides a short overview of LoadRunner s features, and includes the following sections:

Leveraging the Cloud for On-Demand Performance Testing COLLABORATIVE WHITEPAPER SERIES New user behaviors that businesses are introducing to their web sites are driving more volume than ever before. Product

OBSERVEIT DEPLOYMENT SIZING GUIDE The most important number that drives the sizing of an ObserveIT deployment is the number of Concurrent Connected Users (CCUs) you plan to monitor. This document provides

Load Testing an Internet Site Course Overview Course Title Load Testing and Performance Tuning Methods for Optimizing Web Site Performance Description Maximizing Web site performance is essential for those

HIGH-SPEED BRIDGE TO CLOUD STORAGE Addressing throughput bottlenecks with Signiant s SkyDrop 2 The heart of the Internet is a pulsing movement of data circulating among billions of devices worldwide between

Tuning Tableau Server for High Performance I wanna go fast PRESENT ED BY Francois Ajenstat Alan Doerhoefer Daniel Meyer Agenda What are the things that can impact performance? Tips and tricks to improve