Observe/Persistence Testing

This document will outline proposals for a new OOB command for fine-grained control over persistence and replication settings. This will be used to test our so-called durability operations and ensure they actually report correct status.

The problem with testing durability and persistence on a real server is the fact that the actual replication and/or persisting of a specific key depends on the physical latencies which exist in the disk and network hardware (or the CPU speed, if running in an all-ram, virtual machine environment). Thus there is a lack of certainty in determining what the expected state of the key is and what the expected OBSERVE response should be.

Injecting fine-grained commands in the mock will solve this issue as the persistence and replication status of a key will now be explicitly controlled via OOB command rather than dependent on hardware speed.

Concepts

At the basic level, there are multiple concepts which need to be dealt with here. We'll list them here

Nodes and Replicas

In a normal environment, a key is first stored to the master's cache (in memory) and is then placed within a replication queue to be distributed to other replicas. Because replication is dependent on the network, and persistence dependent on disk, we may have one of the following scenarios:

Key is cached in master, persisted in master, cached in replicas and persisted in replicas. This happens if the time to write to the master's disk is less than the time to send+persist to the replicas; so e.g.:

Key is stored in master's RAM

Key is placed in both master's write queue and replication queue

Key is written to disk

Key is sent over network to replicas

Replicas persist item to disk

Key is cached in master, replicated to replicas, persisted to master, persisted to replicas. This happens if the write queue for the master is slower than the network replication queue; so e.g.

Key is stored in master's RAM

Key is placed in both master's write queue and replication queue

Key is sent over network to replicas

Key is written on master's disk

Key is written on replicas' disks.

An even more extreme condition is where the master's disk is very slow, and the key is both persisted and replicated to the replicas before it is persisted to the master

Key is stored in master's RAM

Key is placed in both master's write queue and replication queue

Key is sent over to network to replicas

Key is written on replicas' disks

Key is written on master's disk

Controlling Replication and Persistence status

The commands will feature options to:

Explicitly persist a key to master. A simple command would look like:

Command: persist

Payload (Payload for Command):

Key:"Some Key" (the key must exist)

vBucket: The vBucket ID for the key. This must be provided

CAS: (Optional) - the CAS value for the key. If not supplied, the true cas will be used