Overview

Right now it's fairly difficult to test clients and proxies (moxi) under controlled circumstances. Moxi does have an existing test framework that simulates both server and client, but it is not usable for large-scale tests, was designed for proxy testing only and adding new tests is difficult. The current ns_server client tests expect there to be a working memcached server running; it would be easier to test independently as well as under adverse conditions if we had a flexible simulator we could depend on for testing.

A unified simulator framework in a single project would give us a central repository for memcached server behaviors that we could use for testing everything we have that acts as a memcached client under a variety of conditions that are difficult or impossible to reproduce using vanilla memcached.

Requirements

These are in order of priority and get increasingly vague as they go to lower priority (and thus farther in the future).

Implemented or at least fully controllable from a high level language to make changing behavior easy.

Usable as a target for testing clients under "normal" conditions - this would make the simulator useful immediately.

Concurrency tests. In fact, moxi is configured to be single-threaded and downstream_max=1 for the test framework to currently work, as python threads currently don't allow for the right thing to happen.

A language (API or DSL) usable by any Membase engineer (test or otherwise) to specify (potentially abnormal) server behavior

Ability to synchronize server behavior with existing test frameworks - restart with different command line options or send Python code on a separate port to, for example, tell the server to start giving errors for set commands.

Ability to specify behavior in a centralized fashion across a "cluster" of simulators, at least on the same node - could have one node listening on more than one port with separate behavior specifications for each port, or start separate processes.

Ability to specify behavior in a centralized fashion across a "cluster" of simulators on separate nodes (use Erlang as the runner?) - for automating large scale distributed tests.

Current technology

There is a Java-based memcached server simulator implementation (JMemcached), as part of the libmembase project. It was created due to unit testing needs.

We have three Python memcached server implementations:

An asyncore-based implementation that's part of the same project mc_bin_client came from - this is the first implementation ever of the memcached binary protocol and was used to test spymemcached. It's single-threaded, and due to asyncore's lack of features, it is not very flexible. For example, making it delay responses would not be very straightforward.

The moxi client/server test suite. This is entirely thread-based. It has the interesting property of being able to very closely coordinate server and client behavior, but Python threads are hard to work and don't behave as well one might hope, and it would be difficult to use this for large-scale (many simulated concurrent clients and many simulated servers) tests.

A Twisted Python implementation that Dustin used for his S3-backed memcached server experiment. This is quite flexible and would probably be the best framework to start with. It currently has an example that shows how to make it slow down requests.

Another option would be to use the existing Erlang framework for this, but the Erlang runtime is more difficult to work with Python, and it would be easy to spawn Python processes from Erlang if in the future we decide to use Erlang to run distributed tests. Node.js is another option, but we seem to be closest to a working simulator with the existing Twisted-based code.