The ssbench-masterrun-scenario command will run benchmark “scenarios”
against an
OpenStack Swift cluster, utilizing one or more distributed ssbench-worker
processes, saving statistics about the run to a file. The ssbench-masterreport-scenario command can then generate a
report from the saved statstics. By default, ssbench-masterrun-scenario
will generate a report to STDOUT immediately following a benchmark run in
addition to saving the raw results to a file.

Coordination between the ssbench-master and one or more ssbench-worker
processes is managed through a pair of PyZMQ sockets. This
allows ssbench-master to distribute the benchmark run across many, many
client servers while still coordinating the entire run (each worker can be
given a job referencing an object created by a different worker).

Installation on Ubuntu

I apologize for this stupid dependency dance with Ubuntu (tested with 12.04
LTS Precise). With the –noop benchmark, gevent-zeromq is about 25%
faster than pyzmq 2.2.0.1’s zmq.green module, so I consider the annoying
gevent-zeromq dependency worth it. The gevent-zeormq
Cython build doesn’t work with Ubuntu 12.04’s Python’s distribute, and Cython
has to be installed in a prior “pip” command to be recognized by
gevent-zeromq’s setup.py:

Installation on OS X

On the Mac, I recommend installing Homebrew and using that to install Python
2.7 and libevent. I haven’t tested a fresh install in a while, but I had far
less problems with Cython and gevent-zeormq on OS X, probably because the
Homebrew Python was newer than Ubuntu 12.04’s?

Scenarios

A “scenario” (sometimes called a “CRUD scenario”) is a utf8-encoded JSON file
defining a benchmark run. Specifically, it defines:

A name for the scenario (an arbitrary string)

A sizes list of “object size” classes. Each object size class has a
name, a size_min minimum object size, a size_max maximum object
size (in bytes), and an
optional crud_profile for just this size. If crud_profile is not
given for a size, the top-level crud_profile will be used. The
crud_profile here is just like the top-level one, an array of 4 numbers
whose relative sizes determine the percent chance of a Create, Read, Update,
or Delete operation. Objects created or updated within an object size
class will have a size (in bytes) chosen at random uniformly between the
minimum and maximum sizes.

An initial_files dictionary of initial file-counts per size class. Each
size class can have zero or
more objects uploaded prior to the benchmark run itself. The proportion of
initial files also defines the probability distribution of object sizes
during the benchmark run itself. So if a particular object size class is not
included in initial_files or has a value of 0 in initial_files, then
no objects in that size class will be used during the benchmark run.

An operation_count of operations to perform during the benchmark run.
An operation is
either a CREATE, READ, UPDATE, or DELETE of an object. This value may be
overridden for any given run with the -o COUNT flag to ssbench-masterrun-scenario.

A run_seconds number of seconds the benchmark scenario should run. This
is mutually exclusive with operation_count, so only one of those two
should be specified. Both values may be overridden with command-line
arguments to ssbench-master.

A crud_profile which determines the distribution of each kind of operation.
For instance, [3, 4, 2, 2] would mean 27% CREATE, 36% READ, 18% UPDATE,
and 18% DELETE.

A user_count which determines the maxiumum client concurrency during the
benchmark run. The user is responsible for ensuring there are enough workers
running to support the scenario’s defined user_count. (Each
ssbench-worker process uses gevent to achive very efficient
concurrency for the benchmark client requests.) This value may be overridden
for any given run with the -u COUNT flag to ssbench-masterrun-scenario.

A container_count which determines how many Swift containers are used for
the benchmark run. This key is optional in the scenario file and defaults to
100. This value may be overridden for any given run with the -c
COUNT flag to ssbench-masterrun-scenario.

A container_concurrency value which determines the level of client
concurrency used by ssbench-master to create the benchmark containers.
This value is optional and defaults to 10.

For each operation of the benchmark run, a size category is first chosen based
on the relative counts for each size category in the initial_files
dictionary. This probability for each size category appears under the “% Ops”
column in the report. Then an operation type is chosen based on that size
category’s CRUD profile (which can be individually specified or may be
inherited from the “top level” CRUD profile).

If each size category has its own CRUD profile, then the overall CRUD profile
of the benchmark run will be a weighted average between the values in the “%
Ops” column and the CRUD profile of each size category. This weighted average
CRUD profile is included in the report on the “CRUD weighted average” line.

ssbench comes with a few canned scenarios, but users are encouraged to
experiment and define their own.

The ssbench-master command requires one sub-command, which is currently
either run-scenario to actually run a benchmark scenario,
report-scenario to report on an existing scenario result data file, or
kill-workers to tell connected ssbench-worker processes not started
with --workers to kill themselves:

The kill-workers sub-command of ssbench-master kills all
ssbench-worker processes which are pointed at the ssbench-master
ZMQ sockets (this is useful for multi-server benchmark runs where the workers
were not started with ssbench-master’s --workers option):

Authentication

ssbench-master supports all the same authentication arguments, with similar
semantics, as python-swiftclient’s command-line tool, swift.

For v1.0 authentication, you just need ST_AUTH, ST_USER, and ST_KEY
defined in the environment or overridden/set on the command-line with -A,
-U, and -K, respectively.

For v2.0 authentication (Keystone), it’s more complicated and you should refer
to Keystone and/or python-swiftclient documentation for more help.

Regardless of which version of authentication is used, you may specify -S
<storage_url> on the command-line to override the Storage URL returned from
the authentication system.

Example Multi-Server Run

Start one or more ssbench-worker processes on each server (each
ssbench-worker process defaults to a maximum gevent-based concurrency
of 256, but the -c option can override that default). Use the
--zmq-host command-line parameter to specify the host on which you will run
ssbench-master.:

The above example would involve a total client concurrency of 2000, spread
evenly among the four workers on two hosts (bench-host-01 and
bench-host-02). The four workers, as started in the above example,
could support a maximum total client concurrency (-u option to
ssbench-master) up to 4000.

Example Simple Single-Server Run

If you only need workers running on the local host, you can do so with a single
command. Simply use the --workers COUNT option to ssbench-master:

Benchmark Reports

The default, textual table report may be seen in the above example output. You
can also specify --csv when running a scenario or generating a report later
to generate a CSV report instead. This feature is still pretty new so expect
the CSV report output to change over time.

Right now, the default report’s CSV version is two lines: a line of column
header names and one line of actual data. Both lines are very long and the
set of columns present in any given CSV report will depend on the scenario
which was run. Some column names have the --pctile value in them and many
columns have the object sizes in them, which are defined in the scenario file.
You can think of the two CVS lines as a linear denormalization of the contents
of the two-dimensional table output.

Scalability and Throughput

Assuming the Swift cluster being benchmarked is not the bottleneck, the
scalability of ssbench may be increased by

Running up to one ssbench-worker process per CPU core on any number of
benchmarking servers.

Increasing the default --batch-size parameter (defaults to 1) on both the
ssbench-master and ssbench-worker command-lines. Note that if you
are running everything on one server and using the --workers argument to
ssbench-master, the --batch-size parameter passed to
ssbench-master will be passed on to the automatically-started
ssbench-worker processes.

For optimal scalability, the user-count (concurrency) should be greater than
and also an even multiple of both the batch-size and number of
ssbench-worker processes.

As a simple example, on my quad-core MacBook Pro, I get around 9,800 requests
per second with --noop (see below) with this command-line (a
--batch-size of 1):

$ ssbench-master run-scenario ... -u 24 -o 30000 --workers 3 --noop

But with a --batch-size of 8, I can get around 19,500 requests per second:

HTTPS on OS X

When running ssbench-worker on a Mac, using HTTPS, I got a significant
speed-up when setting OPENSSL_X509_TEA_DISABLE=1 in the environment of my
ssbench-worker processes. I found this tip via a curl blog post after
noticing a process named trustevaluationagent chewing up a lot of CPU
during a benchmark run against a cluster using HTTPS.

The No-op Mode

To test the maximum throughput of the ssbench-master <==>
ssbench-worker infrastructure, you can add --noop to a
ssbench-masterrun-scenario command and the scenario will be “run” but
the ssbench-worker processes will not actually talk to the Swift cluster.

In this manner, you may determine your maximum requests per second if talking
to the Swift cluster were free.

The reported “Average requests per second:” value in the “TOTAL” section of
the report should be higher than you expect to get out of the Swift cluster
itself.

With an older version of ssbench which used a beanstalkd server to manage
master/worker communication, my 2012 15” Retina Macbook Pro could get ~2,700
requests per second with --noop using a local beanstalkd, one
ssbench-worker, and a user count (concurrency) of 4.

With ZeorMQ sockets (no beanstalkd involved), the same laptop can get between
7,000 and 8,000 requests per second with --noop.

Contributing to ssbench

First, please use the Github Issues for the project when submitting bug reports
or feature requests.

Code submissions should be submitted as pull requests and all code should be
PEP8 (v. 1.4.2) compliant. Current unit test line coverage is not 100%, but
code contributions should not lower the code coverage (so please include
new tests or update existing ones as part of your change). Running tests will
probably require Python 2.7 and a few additional modules like flexmock and
nose.

Regarding test tools, I started out using flexmock, but plan to mostly add
new tests using the mock library since that’s been included in the stdlib
and the Python community seems to be converging on it. So please use mock
instead of flexmock for new tests.

If contributing code which implements a feature or fixes
a bug, please ensure a Github Issue exists prior to submitting the pull request
and reference the Issue number in your commit message.

When submitting your first pull request, please also update AUTHORS to include
yourself, maintaining alphabetical ordering by last name.

If any of the file(s) you change do not yet have a copyright line with your
name, please add one at the bottom of the others, above the license text (but
never remove any existing copyright lines). Your copyright line should look
something like: