Overview

Tap provides a mechanism to observe from the outside data changes going on within a memcached server.

CautionThe TAP API is an internal Couchbase Server API, used by much of the core functionality of Couchbase Server. It is currently not intended or supported as an API for external use. The API may change between releases. TAP usage without a complete understanding of the interface and its server side implementation can have a severe impact on memory and IO on cluster nodes. The performance impact can be high enough that nodes or the cluster may not be effectively used by other client workload.

Please NoteIn current releases of Membase Server, TAP dump is only implemented for Couchbase bucket types, not for memcached buckets.

Use Cases

Tap is a building block for lots of new types of things that would like to react to changes within a memcached server without having to actually modify memcached itself.

Replication

One simple use case is replication.

Upon initial connect, a client can ask for all existing data within a server as well as to be notified as values change.

We receive all data related to each item that's being set, so just replaying that data on another node makes replication an easy exercise.

Observation

Requesting a tap stream of only future changes makes it very easy to see the types of things that are changing within your memcached instance.

Secondary Layer Cache Invalidation

If you have frontends that are performing their own cache, requesting a tap stream of future changes is useful for invalidating items stored within this cache.

Ideally, such a stream would not include the actual data that had changed. Today, all tap streams include full bodies, but specifying new features that can be implemented by engines such as requesting the omission of values is very straightforward.

External Indexing

A tap stream pointed at an index server (e.g. sphinx or solr) will send all data changes to the index allowing for an always-up-to-date full-text search index of your data.

vbucket transition

For the purposes of vbucket transfer between nodes, a new type of tap request can be created that is every item stored in a vbucket (or set of vbuckets) that is both existing and changing, but with the ability to terminate the stream and cut-over ownership of the vbucket once the last item is enqueued.

Protocol

A tap session begins by initiating a command from the client which tells the server what we're interested in receiving and then the server begins sending /client/ commands back across the connection until the connection is terminated.

Moxi does not support proxy of the TAP sessions, so you have to connect to the server you are interested in on port 11210. (Like all other traffic, you must [authenticate using SASL] to connect to the desired bucket)

Initial Base Message

A tap stream begins with a binary protocol message with the ID of 0x40 .

The packet's key may specify a unique client identifier that can be used to allow reconnects (resumable at the server's discretion).

A simple base message from a client referring to itself as "node1" would appear as follows.

Options

Additional tap options may be specified as a 64-bit flags specifying options. The flags will appear in the "extras" section of the request packet. If omitted, it is assumed that all flags are 0.

Options may or may not have values. For options that do, the values will appear in the body in the order they're defined (LSB -> MSB).

Backfill

BACKFILL ( =0x01=) contains a single 64-bit body that represents the oldest entry (from epoch) you're interested in. Specifying a time in the future (for the server you are connecting to), will cause it to start streaming only current changes.

An example tap stream request that specifies a backfill of -1 (meaning future only) would look like this:

Keys only

KEYS_ONLY ( =0x20=) contains no extra data and is to notify the tap server that we (the consumer) don't care about the values so we would prefer that the tap server didn't send them. The server may however decide to ignore your request.

An example tap stream request that specifies that we would prefer the just the keys would look like this:

Registered Client

REGISTERED_CLIENT ( =0x80=) contains no extra data and is to notify the tap server that we (the consumer) would like to create a tap connection where the tap connection will exist until we deregister the tap client. Deregistering a tap client can only be done through the deregister tap client command through the memcached front end.

*Note: A registered tap connection that is not being read from will not allow closed checkpoints in the memcached engine to be purged from memory causing out of memory errors. Use of a registered tap connection is only recommended for expert Couchbase users.

An example tap stream request that specifies that we would like a registered tap client would look like this:

Response Commands

After initiating tap, a series of responses will begin streaming commands back to the caller. These commands are similar to, but not necessarily the same as existing commands.

In particular, each command includes a section of engine-specific data as well as a TTL to avoid replication loops.

[describe extended formats]

Mutation

All mutation events arrive as TAP_MUTATION ( =0x41=) events. These are conceptualy similar to set commands. A mutation event for key "mykey" with a value of "value" (no flags or expiry) and a TTL of 255 would look like:

*In Couchbase 2.0+ the engine private field is used to specify the length of any meta data sent along with the key and value. This meta-data will be placed between the Item Expiry field and the Key field.

Delete

TAP_DELETE ( =0x42=) may contain an engine specific section. A typical packet for deletion of "mykey" with a TTL of 255 without any engine specific data would look like this:

*In Couchbase 2.0+ the engine private field is used to specify the length of any meta data sent along with the key and value. This meta-data will be placed between the Item Expiry field and the Key field.

Flush

TAP_FLUSH ( =0x43=) may contain an engine specific section. This message is used to replicate a flush_all command. A typical flush packet with a TTL of 255 without any engine specific data would look like this:

In an opaque message the engine private field contains control message that are used to signal to the receiving engine that the state of the tap stream has changed. Below is a list of engine private control values:

TAP_OPAQUE_ENABLE_AUTO_NACK (0) - When the receiving node initiates a tap stream with a sending node the receiver may specify that it wants to receive acknowledgment messages from the sender. The receiver does this by specifying an ACK flag when creating the connection. If the sending node receives this flag it sends the TAP_OPAQUE_ENABLE_AUTO_NACK in order to signal that the sender supports acknowledgement messages.TAP_OPAQUE_INITIAL_VBUCKET_STREAM (1) - Signals the beginning of the backfill operation. Backfill takes place when the newest checkpoint in the receiving node is behind the oldest checkpoint in the sending node. Backfill takes a snapshot of both disk and memory on the sending node and streams it to the receiving node in order to get the receiving node up to date.TAP_OPAQUE_ENABLE_CHECKPOINT_SYNC (2) - Signals to the receiving node that the sending node supports checkpoints.TAP_OPAQUE_OPEN_CHECKPOINT (3) - Signals that we are at the beginning of an open checkpoint. This message is only sent if the tap stream is a registered tap stream.TAP_OPAQUE_START_ONLINEUPDATE (4) - This signal can be sent by either the sender or receiver. It tells whatever engine that receives it to stop persisting data and only keep things in memory.TAP_OPAQUE_STOP_ONLINEUPDATE (5) - This message can also be sent by the sender or receiver. It signals to the receiving node that the persistence should be resumed.TAP_OPAQUE_REVERT_ONLINEUPDATE (6) - This message can also be sent by the sender or receiver. It signals to the receiving node that everything received since the TAP_OPAQUE_START_ONLINEUPDATE should be deleted from the engine. Note that this will remove everything received by the engine after TAP_OPAQUE_START_ONLINEUPDATE is received including message not send through the memcached front end as well as other tap streams.TAP_OPAQUE_CLOSE_TAP_STREAM (7) - Sent by the sending node to signal that the sending node is finished sending data and is closing it's connection to the receiver.TAP_OPAQUE_CLOSE_BACKFILL (8) - Signals the end of the backfill.

*Note: Online update is a feature that can be used to test cluster usage with the addition of new data. It is not recommended for use in normal production setting.

Set vbucket

The purpose of the TAP_VBUCKET ( =0x45=) packet is to set the state of a virtual bucket in the consumer (this is part of vbucket takeover)