The yaml file defines an environment with three machines named machine1, machine2 and machine3. machine1 is the local machine. machine2 is the machine at 100.100.100.100 reachable by the command ssh -i ~/key_path username@100.100.100.100 and login using the passphrase key_password. machine3 is a Google Cloud Platform (GCP) virtual machine that will be created automatically in the GCP project benchmark-environment in the zone us-west1-a with name example-instance when running perf.py. perf.py will also handle logging into username@external_ip_of_vm to run the benchmarks.

The above is an example only. Generally machines should be uniform, since they are treated as such by the tests. Machines must also be accessible to each other via their default routes. Furthermore, some benchmarks will meaningless if running on the local machine, such as density.

For remote machines, hostname, key_path, and username are required and others are optional. In addition key files must be generated using the instrcutions below.

The above yaml file can be checked for correctness with the validate command in the top level perf.py script:

Writing benchmarks

To write new benchmarks, you should familiarize yourself with the structure of the repository. There are three key components.

Harness

The harness makes use of the docker py SDK. It is advisable that you familiarize yourself with that API when making changes, specifically:

clients

containers

images

In general, benchmarks need only interact with the Machine objects provided to the benchmark function, which are the machines defined in the environment. These objects allow the benchmark to define the relationships between different containers, and parse the output.

Workloads

The harness requires workloads to run. These are all available in the workloads directory.

In general, a workload consists of a Dockerfile to build it (while these are not hermetic, in general they should be as fixed and isolated as possible), some parses for output if required, parser tests and sample data. Provided the test is named after the workload package and contains a function named sample, this variable will be used to automatically mock workload output when the --mock flag is provided to the main tool.

Writing benchmarks

Benchmarks define the tests themselves. All benchmarks have the following function signature:

Each benchmark takes a variable amount of position arguments as harness.Machine objects and some set of keyword arguments. It is recommended that you accept arbitrary keyword arguments and pass them through when constructing the container under test.

To write a new benchmark, open a module in the benchmarks directory and use the above signature. You should add a descriptive doc string to describe what your benchmark is and any test centric arguments.

Generating SSH Keys

The scripts only support RSA Keys, and ssh library used in paramiko. Paramiko only supports RSA keys that look like the following (PEM format):