Disclaimer

This tutorial is designed to supplement the E2E Gaming tutorial, providing you with a learning tool surrounding the mechanics of autoscaling. (However, you can use the steps below as a general guideline for getting a load testing server in your own custom deployment.)

Prerequisites

A deployment that is ready to be load tested, including a server array for your application.

Steps

Import the ServerTemplate

Navigate to: Design -> MultiCloud Marketplace -> ServerTemplates

Search for and import the "Load Tester - httperf v2" ServerTemplate to your local collection

Add a load testing Server and launch it

Browse to your deployment

Pick the correct cloud and select the Add Server action button. Fill out the form appropriately, including the following fields:

ServerTemplate - Imported, Load Tester - httperf

Nickname - LoadTester

Select your SSH Key and Security Group

EIP - none

Associate IP at launch - Select (check) this option

When ready, select the Add action button

Before launching your server, set the following Input at the deployment level

LB_HOSTNAME - Set this to the fully qualified domain name of your front end servers. For example, www.rightscaletraining.com

Launch your server

Configure the Server Array

You need to make sure your server array is enabled, and configured properly. Specific settings for the array vary based on application, and what you want to accomplish with your load tests. If you are simply learning the mechanics of server arrays and our alert/monitoring subsystem, you may want to limit the number of servers you can scale to, and how quickly the scaling process works. In our example settings below we change the maximum count of servers from the default 20 to just 5. The amount of time it takes to scale is artificially shortened by changing the calm time.

Select your server array

Select the Edit action button. Change parameters as needed. The following changes are examples only!

Default max count: 5

Resize calm time: 2

If your server array is not already enabled, enable it.

See the "Status" setting. Select the enable action link if needed.

Select the Save action button to save your settings

Initiate the load test from the shell

This step requires to SSH into the httperf server and run the httperf command manually. This command will send 100 requests per second to get "/index.php" hosted on server with IP "50.57.80.214", port 80. After 60000 requests (~10 minutes), httperf will stop and display the load statistics. Hit Control+C at any time to abort the load and return to the shell.

Initiate the load test using an operational script

Optional: SSH into your front end servers. Get a sense for how busy they are. For example, run "uptime" and "top" to view various system resources before, during and after the load gets applied. You should observe httpd (apache) bubble up to the top, the CPU %id will drop way down (possibly to 0% idle time), free memory will drop as well. This will make the log file viewing as a Post Tutorial Step more informative.

Navigate to your load test Server -> Scripts tab

Run the "PERF httperf start" operational RightScript

There are several missing Inputs you will have to set. Suggested values for a typical load test are included below. More complicated values are considered application specific and you will need to know your application and httperf well in order to stress test in accord with your requirements.

PERF_MAX_RATE

The "rate" is the number of connections that are created at a time. The min and max rates are used to change the rate each time the test is run by setting it somewhere between the values of min and max. (PERF_MIN_RATE and PERF_MAX_RATE respectively. See the "- -rate" option on the httperf manual page for more information.)

Set to 10000

PERF_MIN_RATE

Set to 5000

PERF_PERIOD

Value is tied to a sine wave (sinusoidal period). If not clear how this works, its is best to set the value to 1 (minute).

Set to 1

PERF_SERVER

Public DNS of your frontend server(s). This could also be a publicly routable IP address of your frontend. However, if you have set up DNS records for the load balancing tier, it is best to enter the public DNS name.

For example, www.yourApplicationFE.com or ec2-50-16-4-153.compute-1.amazonaws.com (Note: Do not include http:// when specifying the DNS name.)

PERF_TEST_DURATION

Total length of time in minutes you want httperf to run

Set to 20

Note: You can run the "PERF httperf stop" RightScript to shutdown the test prior to this length of time.

PERF_TEST_NAME

Name of the test you are about to run. Can use a descriptive name. It will be used for naming logs in the /home/httperf/ directory.

For example, perf_test

PERF_TEST_RUN

Time to run each test (in minutes). This value plays a factor in the number of connections that are established.

Set to 5

PERF_URI

URI used in repeated test requests to increase load. Typically the index.php for the default application (such as the e2e Gaming deployment or other deployment you just build out). For a PHP based application, this should be a PHP page able to respond to load test queries. The value is also used in construction of part of the log file name. (See Post Tutorial Steps.)

For example, set to /index.php

Navigate to the Monitoring page for your deployment. Within about 45 seconds your front end server should be under significant load, voting to grow your array, etc.

Post Tutorial Steps

Additional Documentation and Logging

This step is considered optional, considering time, interest and previous level of understanding of httperf.

The log file will display a wealth of information, including system load (uptime numbers), errors, and connection information (rate, time, length). It also conveys each httperf command line with actual parameter values. For example:

httperf --hog --timeout=5 --client=0/1 --server=www.rightscaletraining.com --port=80 --uri=/index.php \
--rate=200 --send-buffer=4096 --recv-buffer=16384 --num-conns=30000000 --num-calls=1
#Note: Above command is a single line of text. The \ followed by a <CR>/<LF> is for display purposes only.

Cleanup

Depending on whether you are going to keep your load tester running or not, you may need to perform some cleanup tasks. For example, if you lowered the resize calm time to speed up your testing, you can change it back. Similarly, you may need to change the min or max count, or disable the server array entirely. Remember, cloud charges apply to running servers.

Other Considerations

Before you validate your array's scaling thresholds, it may help to segment array load testing from array functional testing. This allows you to isolate functional issues, for example, the joining and leaving of new application server instances to and from the load balancing pool, from load-related performance issues.

Some social networking platforms have usage restrictions in their Terms and Conditions that may prohibit load testing through their platform. In this case, it is common to send load directly to a specific application server instance, for example a php index page, to validate your array's scaling thresholds.

Your first test should be small. Watch the monitoring graphs on a single application server instance to determine which system resource becomes bound first (CPU,MEM,etc). Once determined, you will need to create an alert escalation based on a specific metric. This alert escalation will drive array grow and shrink voting. Then, you can throttle your testing volume and see the array shrink and grow.

If you are using AWS' ELB, reported metrics will be limited as compared with what you would expect to see on a specific load balancer instance. In this case, to you can watch the number of operational application server instances to tune your array thresholds.