CDH

This is the documentation for CDH 4.7.1.
Documentation for other versions is available at Cloudera Documentation.

Configuring HBase Snapshots

About HBase Snapshots

In previous HBase releases, the only way to a backup or to clone a
table was to use CopyTable or ExportTable, or to copy all the hfiles in HDFS after disabling the table. The
disadvantages of these methods are:

CopyTable and
ExportTable can degrade region server
performance.

Disabling the table means no reads or writes; this is usually unacceptable.

HBase Snapshots allow you to clone a table without making data copies, and with minimal impact on Region
Servers. Exporting the table to another cluster should not have any impact on the the region servers.

Use Cases

Recovery from user or application errors

Useful because it may be some time before the database
administrator notices the error

Note:

The database administrator needs to schedule the intervals
at which to take and delete snapshots. Use a script or your preferred management
tool for this; it is not built into HBase.

The database administrator may want to save a snapshot right
before a major application upgrade or change.

Note:

Snapshots are not primarily used for system upgrade
protection because they would not roll back binaries, and would not necessarily
be proof against bugs or errors in the system or the upgrade.

Sub-cases for recovery:

Rollback to previous snapshot and merge in reverted
data

View previous snapshots and selectively merge them into
production

Backup

Capture a copy of the database and store it outside HBase for disaster
recovery

Export from snapshot on live system provides a more consistent
view of HBase than CopyTable and
ExportTable

Audit and/or report view of data at a specific time

Capture monthly data for compliance

Use for end-of-day/month/quarter reports

Use for Application testing

Test schema or application changes on like production data from snapshot and then
throw away

For example: take a snapshot; create a new table from the snapshot content (schema
plus data); manipulate the new table by changing the schema, adding and removing
rows, and so on (the original table, the snapshot, and the new table remain
independent of each other)

Offload work

Capture, copy, and restore data to another site

Export data to another cluster

Where Snapshots Are Stored

The snapshot metadata is stored in the .snapshot directory under the hbase
root directory (/hbase/.snapshot). Each snapshot has its own directory that includes all the
references to the hfiles, logs, and metadata needed to restore the table.

hfiles needed by the snapshot are in the traditional
/hbase/<tableName>/<regionName>/<familyName>/ location if the table is
still using them; otherwise they will be placed in
/hbase/.archive/<tableName>/<regionName>/<familyName>/

Zero-copy Restore and Clone Table

From a snapshot you can create a new table (clone operation) or restore the original table.
These two operations do not involve data copies; instead a link is created to point to the original
hfiles.

Changes to a cloned or restored table do not affect the snapshot or (in case of a clone) the original table.

Reverting to a Previous HBase Version

Snapshots don’t affect HBase backward compatibility if they are not used.

If you do use the snapshot capability, backward compatibility is affected as follows:

If you only take snapshots, you can still go back to a previous HBase version

If you have used restore or clone, you cannot go back to a previous version unless the cloned or restored tables have no links (there is no automated way to check; you would need to inspect the file system manually).

Storage Considerations

Since the hfiles are immutable, a snapshot consists of reference to the files that are in the
table at the moment the snapshot is taken. No copies of the data are made during the snapshot operation, but
copies may be made when a compaction or deletion is triggered. In this case, if a snapshot has a reference to
the files to be removed, the files are moved to an archive folder, instead of being deleted. This allows the
snapshot to be restored in full.

Because no copies are performed, multiple snapshots share the same hfiles, but in the worst
case scenario, each snapshot could have different set of hfiles (tables with lots of updates, and
compactions).

Configuring and Enabling Snapshots

Snapshots are off by default; to enable them, set the hbase.snapshot.enabled property in
hbase-site.xml to true:

To disable snapshots after you have enabled them, set hbase.snapshot.enabled to
false.

Note:

If you have taken snapshots and then decide to disable snapshots, you must delete the snapshots before
restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.

Snapshots don’t affect HBase performance if they are not used.

Shell Commands

You can manage snapshots by using the HBase shell or the HBaseAdmin Java API.

The following table shows actions you can take from the shell:

Action

Shell command

Comments

Take a snapshot of tableX called snapshotX

snapshot 'tableX', 'snapshotX'

Snapshots can be taken while a table is disabled, or while a table is online and serving traffic.

If a table is disabled (via disable <table>) an offline snapshot is taken. This snapshot is driven by the master and fully consistent with the state when the table was disabled. This is the simplest and safest method, but it involves a service interruption since the table must be disabled to take the snapshot.

In an online snapshot, the table remains available while
the snapshot is taken, and should incur minimal noticeable performance
degradation of normal read/write loads. This snapshot is coordinated by the
master and run on the region servers. The current implementation - simple-flush
snapshots - provides no causal consistency guarantees. Despite this shortcoming,
it offers the same degree of consistency as CopyTable and overall is a huge
improvement over CopyTable.

Restore snapshot snapshotX (it will replace the source table content)

restore_snapshot ‘snapshotX’

Restoring a snapshot attempts to replace the current version of a table with another version of the
table. To run this command, you must disable the target table. The restore command
takes a snapshot of the table (appending a timestamp code), and then essentially clones data into the
original data and removes data not in the snapshot. If the operation succeeds, the target table will
be enabled. Use this capability only in an emergency; see
Restrictions.

List all available snapshots

list_snapshots

List all available snapshots starting with ‘mysnapshot_’ (regular expression)

list_snapshots ‘my_snapshot_*’

Remove a snapshot called snapshotX

delete_snapshot ‘snapshotX’

Create a new table tableY from a snapshot snapshotX

clone_snapshot ‘snapshotX’, ‘tableY’

Cloning a snapshot creates a new read/write table that can serve the data kept at the time of the
snapshot. The original table and the cloned table can be modified independently without interfering
– new data written to one table will not show up on the other.

Exporting a Snapshot to Another Cluster

You can export any snapshot to another cluster. This involves the copy
of the table hfiles and logs retained
during the snapshot, and the snapshot metadata.

You can also use the Java -D option in many tools to specify MapReduce or other configuration. properties.
For example, the following command copies MY_SNAPSHOT to
hdfs://cluster2/hbase using groups of 10 hfiles per
mapper:

Restrictions

Warning:

Do not use merge in combination with snapshots. Merging two
regions can cause data loss if snapshots or cloned tables exist for this table.

The merge is likely to corrupt the snapshot and any tables cloned from the snapshot. In
addition, if the table has been restored from a snapshot, the merge may also corrupt the
table. The snapshot may survive intact if the regions being merged are not in the
snapshot, and clones may survive if they do not share files with the original table or
snapshot. You can use the Snapinfo tool (see Information and Debugging) to check the status of the snapshot. If the
status is BROKEN, the snapshot is unusable.

All the Masters and Region Servers must be running at least CDH4.2.

If you have enabled
the AccessController Coprocessor for HBase,
only a global administrator can take, clone, or restore a snapshot, and these
actions do not capture the ACL rights. This means that restoring a table
preserves the ACL rights of the existing table, while cloning a table creates a
new table that has no ACL rights until the administrator adds them.

Do not take, clone, or restore a snapshot during a rolling restart. Snapshots rely on
the Region Servers being up; otherwise the snapshot will fail.

Note: This restriction also
applies to rolling upgrade, which can currently be done only via Cloudera
Manager.

If you are using HBase Replication and you need to restore a snapshot: If you are
using HBase Replication the replicas will be out of synch
when you restore a snapshot. Do this only in an emergency.

Important:

Snapshot restore is an emergency tool; you need to disable the table and
table replication to get to an earlier state, and you
may lose data in the process.

If you need to restore a snapshot, proceed as follows:

Disable the table that is the restore target, and stop the replication

Remove the table from both the master and slave clusters

Restore the snapshot on the master cluster

Create the table on the slave cluster and use CopyTable to initialize it.

Note:

If this is not an emergency (for example, if you know that you have lost just a set of rows such as the rows
starting with "xyz"), you can create a clone from the snapshot and create a MapReduce job to copy the data
that you've lost.

In this case you don't need to stop replication or disable your main table.

Snapshot Failures

Region moves, splits, and other metadata actions that happen while a snapshot is in progress will probably cause
the snapshot to fail; the software detects and rejects corrupted snapshot attempts.

Information and Debugging

You can use the SnapshotInfo tool to get information about a snapshot, including status, files,
disk usage, and debugging information.