Description

The cbtransfer tool is the underlying, generic data transfer tool upon which cbbackup and cbrestore are built.

It is a lightweight extract-transform-load (ETL) tool that transfers data between clusters and to and from files.
The source and destination parameters are similar to URLs or file paths.

The cbtransfer tool makes snap-shots of memory at the time it is invoked.
All keys that are extracted pertain to the snapshot and not to the data that is added, edited or deleted while cbtransfer is running.

The most important way to use this tool is for transferring data from a Couchbase Server node that is no longer running to a cluster that is running.
The cbbackup, cbrestore, and cbtransfer tools do not communicate with external IP addresses for server nodes outside of a cluster.
Backup, restore, or transfer operations are performed on data from a node within a Couchbase Server cluster.
They only communicate with nodes from a node list obtained within a cluster.
If you install Couchbase Server with a default IP address, you cannot use an external hostname to access it.

Couchbase Server does not transfer design documents.
To back up a design document, use cbbackup to store the information and cbrestore to read it back into memory.

Single named bucket on the destination cluster that receives the transfer.
You can transfer to a bucket with a different name than your source bucket; if you do not provide a name, it defaults to the same name as the bucket-source.

-i ID, --id=ID

Transfer only items that match the vbucket ID.

-k KEY, --key=KEY

Transfer only items with keys that match the regexp.
1

-n, --dry-run

No actual transfer occurs, just the validation of parameters, files, connectivity, and configurations.

-u USERNAME, --username=USERNAME

REST username for the source cluster or the server node.

-p PASSWORD, --password=PASSWORD

REST password for the cluster or the server node.

-t THREADS, --threads=THREADS

Number of concurrent worker threads performing the transfer.
Memcached uses 75% of the number of cores reported of the system (with a minimum of 4 cores) 2.

-v, --verbose

Verbose logging; provide more verbosity.

-x EXTRA, --extra=EXTRA

Provide extra, uncommon configuration parameters.

--single-node

Transfer from a single server node in a source cluster.
This single server node is a source node URL.

--source-vbucket-state=SOURCE_VBUCKET_STATE

Only transfer from the source vbuckets occurs in this state, such as active (default) or replica.
Must be used with the Couchbase cluster as a source.

--destination-vbucket-state=DESTINATION_VBUCKET_STATE

Only transfer to destination vbuckets in this state, such as active (default) or replica.
Must be used with Couchbase Server Server cluster as the destination.

--destination-operation=DESTINATION_OPERATION

Perform this operation on transfer.
set will override an existing document;add will not override; get will load all keys transferred from a source cluster into the caching layer at the destination.

By default, the cbtransfer tool will use set and override any existing documents.

/path/to/filename

Export a .csv file from the server or import a .csv file to the server.

1: Regex (Regular Expression Syntax) specifies a set of strings that matches it; the functions in this module let you check if a particular string matches a given regular expression (or if a given regular expression matches a particular string, which comes down to the same thing).
For a complete reference, see https://docs.python.org/2/library/re.html.

2: To tune memcached to adjust dynamically the number of worker threads, do one of the following:

Export MEMCACHED_NUM_CPUS=number of threads you want before starting Couchbase Server.

Use the -t <number > command line argument.

Specify it in the configuration file that is read during startup.
Be aware that when started from the full server this file is regenerated every time, and you will loose the modifications.

The following are extra, specialized command options with the cbtransfer -x parameter.

Table 2. cbtransfer -x options

-x options

Description

backoff_cap=10

Maximum backoff time during the rebalance period.

batch_max_bytes=400000

Transfer this # of bytes per batch.

batch_max_size=1000

Transfer this # of documents per batch.

cbb_max_mb=100000

Split backup file on destination cluster if it exceeds the MB.

conflict_resolve=1

By default, disable conflict resolution.

data_only=0

For value 1, transfer only data from a backup file or cluster.

design_doc_only=0

For value 1, transfer only design documents from a backup file or cluster.
Default: 0.

max_retry=10

Max number of sequential retries if the transfer fails.

mcd_compatible=1

For value 0, display extended fields for stdout output.

nmv_retry=1

0 or 1, where 1 retries transfer after a NOT_MY_VBUCKET message.
Default: 1.

recv_min_bytes=4096

Amount of bytes for every TCP/IP batch transferred.

rehash=0

For value 1, rehash the partition IDs of each item.
Rehashing is required when transferring data between clusters with a different number of partitions, such as when transferring data from a Mac OSX server to a non-Mac OSX cluster.

report=5

Number of batches transferred before updating the progress bar in the console.

report_full=2000

Number of batches transferred before emitting progress information in the console.

seqno=0

By default, start seqno from beginning.

try_xwm=1

Transfer documents with metadata.
Default: 1.
The value of 0 is used only when transferring from 1.8.x to 1.8.x.

The cbtransfer tool is also used to import and export csv files.
Data is imported into Couchbase Server as documents and documents are exported from the server into comma-separated values.
Design documents associated with vBuckets are not included.

In these examples, the following records are in the default bucket where re-fdeea652a89ec3e9 is the document ID, 0 are flags, 0 is the expiration, and the CAS value is 4271152681275955.
The actual value is the hash starting with "{""key"".......

This example exports these items to a .csv file.
All items are transferred from the default bucket, -b default available at the node http://host:8091 and put into the /data.csv file.
If a different bucket is provided for the -b option, all items are exported from that bucket.
Credentials are required for the cluster when exporting items from a bucket in the cluster.

The following example syntax shows 1053 batches of data transferred at 550.8 batches per second.
The tool outputs "cannot save bucket design…." to indicate that no design documents were exported.
To import information from a.csv file to a named bucket in a cluster: