More Help

This document discusses how sharding works in CouchDB along with how to
safely add, move, remove, and create placement rules for shards and
shard replicas.

A shard is a
horizontal partition of data in a database. Partitioning data into
shards and distributing copies of each shard (called “shard replicas” or
just “replicas”) to different nodes in a cluster gives the data greater
durability against node loss. CouchDB clusters automatically shard
databases and distribute the subsets of documents that compose each
shard among nodes. Modifying cluster membership and sharding behavior
must be done manually.

How many shards and replicas each database has can be set at the global
level, or on a per-database basis. The relevant parameters are q and
n.

q is the number of database shards to maintain. n is the number of
copies of each document to distribute. The default value for n is 3,
and for q is 8. With q=8, the database is split into 8 shards. With
n=3, the cluster distributes three replicas of each shard. Altogether,
that’s 24 shard replicas for a single database. In a default 3-node cluster,
each node would receive 8 shards. In a 4-node cluster, each node would
receive 6 shards. We recommend in the general case that the number of
nodes in your cluster should be a multiple of n, so that shards are
distributed evenly.

CouchDB nodes have a etc/local.ini file with a section named
cluster which looks like this:

[cluster]q=8n=3

These settings can be modified to set sharding defaults for all
databases, or they can be set on a per-database basis by specifying the
q and n query parameters when the database is created. For
example:

$ curl -X PUT "$COUCH_URL:5984/database-name?q=4&n=2"

That creates a database that is split into 4 shards and 2 replicas,
yielding 8 shard replicas distributed throughout the cluster.

Depending on the size of the cluster, the number of shards per database,
and the number of shard replicas, not every node may have access to
every shard, but every node knows where all the replicas of each shard
can be found through CouchDB’s internal shard map.

Each request that comes in to a CouchDB cluster is handled by any one
random coordinating node. This coordinating node proxies the request to
the other nodes that have the relevant data, which may or may not
include itself. The coordinating node sends a response to the client
once a quorum of
database nodes have responded; 2, by default. The default required size
of a quorum is equal to r=w=((n+1)/2) where r refers to the size
of a read quorum, w refers to the size of a write quorum, and n
refers to the number of replicas of each shard. In a default cluster where
n is 3, ((n+1)/2) would be 2.

Note

Each node in a cluster can be a coordinating node for any one
request. There are no special roles for nodes inside the cluster.

The size of the required quorum can be configured at request time by
setting the r parameter for document and view reads, and the w
parameter for document writes. For example, here is a request that
directs the coordinating node to send a response once at least two nodes
have responded:

$ curl "$COUCH_URL:5984/<db>/<doc>?r=2"

Here is a similar example for writing a document:

$ curl -X PUT "$COUCH_URL:5984/<db>/<doc>?w=2" -d '{...}'

Setting r or w to be equal to n (the number of replicas)
means you will only receive a response once all nodes with relevant
shards have responded or timed out, and as such this approach does not
guarantee ACIDic consistency. Setting r or
w to 1 means you will receive a response after only one relevant
node has responded.

This section describes how to manually place and replace shards. These
activities are critical steps when you determine your cluster is too big
or too small, and want to resize it successfully, or you have noticed
from server metrics that database/shard layout is non-optimal and you
have some “hot spots” that need resolving.

Consider a three-node cluster with q=8 and n=3. Each database has 24
shards, distributed across the three nodes. If you add a fourth
node to the cluster, CouchDB will not redistribute
existing database shards to it. This leads to unbalanced load, as the
new node will only host shards for databases created after it joined the
cluster. To balance the distribution of shards from existing databases,
they must be moved manually.

Moving shards between nodes in a cluster involves the following steps:

Technically, copying database and secondary index
shards is optional. If you proceed to the next step without
performing this data copy, CouchDB will use internal replication
to populate the newly added shard replicas. However, copying files
is faster than internal replication, especially on a busy cluster,
which is why we recommend performing this manual data copy first.

Shard files live in the data/shards directory of your CouchDB
install. Within those subdirectories are the shard files themselves. For
instance, for a q=8 database called abc, here is its database shard
files:

Secondary indexes (including JavaScript views, Erlang views and Mango
indexes) are also sharded, and their shards should be moved to save the
new node the effort of rebuilding the view. View shards live in
data/.shards. For example:

Before telling CouchDB about these new shards on the node, the node
must be put into maintenance mode. Maintenance mode instructs CouchDB to
return a 404NotFound response on the /_up endpoint, and
ensures it does not participate in normal interactive clustered requests
for its shards. A properly configured load balancer that uses GET/_up to check the health of nodes will detect this 404 and remove the
node from circulation, preventing requests from being sent to that node.
For example, to configure HAProxy to use the /_up endpoint, use:

http-checkdisable-on-404optionhttpchkGET/_up

If you do not set maintenance mode, or the load balancer ignores this
maintenance mode status, after the next step is performed the cluster
may return incorrect responses when consulting the node in question. You
don’t want this! In the next steps, we will ensure that this shard is
up-to-date before allowing it to participate in end-user requests.

Finally, check that your load balancer has removed the node from the
pool of available backend nodes.

4.4.3.3. Updating cluster metadata to reflect the new target shard(s)¶

Now we need to tell CouchDB that the target node (which must already be
joined to the cluster) should be hosting
shard replicas for a given database.

To update the cluster metadata, use the special /_dbs database,
which is an internal CouchDB database that maps databases to shards and
nodes. This database is replicated between nodes. It is accessible only
via a node-local port, usually at port 5986. By default, this port is
only available on the localhost interface for security purposes.

shard_suffix: A timestamp of the database’s creation, marked as
seconds after the Unix epoch mapped to the codepoints for ASCII
numerals.

changelog: History of the database’s shards.

by_node: List of shards on each node.

by_range: On which nodes each shard is.

To reflect the shard move in the metadata, there are three steps:

Add appropriate changelog entries.

Update the by_node entries.

Update the by_range entries.

Warning

Be very careful! Mistakes during this process can
irreparably corrupt the cluster!

As of this writing, this process must be done manually.

To add a shard to a node, add entries like this to the database
metadata’s changelog attribute:

["add","<range>","<node-name>"]

The <range> is the specific shard range for the shard. The <node-name> should match the name and address of the node as displayed in
GET/_membership on the cluster.

Note

When removing a shard from a node, specify remove instead of add.

Once you have figured out the new changelog entries, you will need to
update the by_node and by_range to reflect who is storing what
shards. The data in the changelog entries and these attributes must
match. If they do not, the database may become corrupted.

Continuing our example, here is an updated version of the metadata above
that adds shards to an additional node called node4:

After you complete the previous step, CouchDB will have started
synchronizing the shards. You can observe this happening by monitoring
the /_node/<nodename>/_system endpoint, which includes the
internal_replication_jobs metric.

Once this metric has returned to the baseline from before you started
the shard sync, or is 0, the shard replica is ready to serve data
and we can bring the node out of maintenance mode.

Now, remove the source shard from the shard map the same way that you
added the new target shard to the shard map in step 2. Be sure to add
the ["remove",<range>,<source-shard>] entry to the end of the
changelog as well as modifying both the by_node and by_range sections of
the database metadata document.

4.4.3.8. Remove the shard and secondary index files from the source node¶

Finally, you can remove the source shard replica by deleting its file from the
command line on the source host, along with any view shard replicas:

You can configure CouchDB to put shard replicas on certain nodes at
database creation time using placement rules.

Warning

Use of the placement option will override the n option,
both in the .ini file as well as when specified in a URL.

First, each node must be labeled with a zone attribute. This defines
which zone each node is in. You do this by editing the node’s document
in the /_nodes database, which is accessed through the node-local
port. Add a key value pair of the form:

In the local config file (local.ini) of each node, define a
consistent cluster-wide setting like:

[cluster]placement=<zone-name-1>:2,<zone-name-2>:1

In this example, CouchDB will ensure that two replicas for a shard will
be hosted on nodes with the zone attribute set to <zone-name-1> and
one replica will be hosted on a new with the zone attribute set to
<zone-name-2>.

This approach is flexible, since you can also specify zones on a per-
database basis by specifying the placement setting as a query parameter
when the database is created, using the same syntax as the ini file:

curl -X PUT $COUCH_URL:5984/<dbname>?zone=<zone>

The placement argument may also be specified. Note that this will
override the logic that determines the number of created replicas!

Note that you can also use this system to ensure certain nodes in the
cluster do not host any replicas for newly created databases, by giving
them a zone attribute that does not appear in the [cluster]
placement string.