BookKeeper administration

A guide to deploying and administering BookKeeper

Requirements

A typical BookKeeper installation consists of an ensemble of bookies and a ZooKeeper quorum. The exact number of bookies depends on the quorum mode that you choose, desired throughput, and the number of clients using the installation simultaneously.

The minimum number of bookies depends on the type of installation:

For self-verifying entries you should run at least three bookies. In this mode, clients store a message authentication code along with each entry.

For generic entries you should run at least four

There is no upper limit on the number of bookies that you can run in a single ensemble.

Performance

To achieve optimal performance, BookKeeper requires each server to have at least two disks. It’s possible to run a bookie with a single disk but performance will be significantly degraded.

ZooKeeper

There is no constraint on the number of ZooKeeper nodes you can run with BookKeeper. A single machine running ZooKeeper in standalone mode is sufficient for BookKeeper, although for the sake of higher resilience we recommend running ZooKeeper in quorum mode with multiple servers.

Starting and stopping bookies

You can run bookies either in the foreground or in the background, using nohup. You can also run local bookies for development purposes.

To start a bookie in the foreground, use the bookie command of the bookkeeper CLI tool:

Local bookies

The instructions above showed you how to run bookies intended for production use. If you’d like to experiment with ensembles of bookies locally, you can use the localbookie command of the bookkeeper CLI tool and specify the number of bookies you’d like to run.

This would spin up a local ensemble of 6 bookies:

$ bookkeeper-server/bin/bookkeeper localbookie 6

When you run a local bookie ensemble, all bookies run in a single JVM process.

Upgrading

From time to time you may need to make changes to the filesystem layout of bookies—changes that are incompatible with previous versions of BookKeeper and require that directories used with previous versions are upgraded. If a filesystem upgrade is required when updating BookKeeper, the bookie will fail to start and return an error like this:

BookKeeper provides a utility for upgrading the filesystem. You can perform an upgrade using the upgrade command of the bookkeeper CLI tool. When running bookkeeper upgrade you need to specify one of three flags:

Flag

Action

--upgrade

Performs an upgrade

--rollback

Performs a rollback to the initial filesystem version

--finalize

Marks the upgrade as complete

Upgrade pattern

A standard upgrade pattern is to run an upgrade…

$ bookkeeper-server/bin/bookkeeper upgrade --upgrade

…then check that everything is working normally, then kill the bookie. If everything is okay, finalize the upgrade…

$ bookkeeper-server/bin/bookkeeper upgrade --finalize

…and then restart the server:

$ bookkeeper-server/bin/bookkeeper bookie

If something has gone wrong, you can always perform a rollback:

$ bookkeeper-server/bin/bookkeeper upgrade --rollback

Formatting

By default, formatting is done in interactive mode, which prompts you to confirm the format operation if old data exists. You can disable confirmation using the -nonInteractive flag. If old data does exist, the format operation will abort unless you set the -force flag. Here’s an example:

$ bookkeeper-server/bin/bookkeeper shell metaformat

You can format the local filesystem data on a bookie using the bookieformat command on each bookie. Here’s an example:

$ bookkeeper-server/bin/bookkeeper shell bookieformat

The -force and -nonInteractive flags are also available for the bookieformat command.

AutoRecovery

Missing disks or directories

Accidentally replacing disks or removing directories can cause a bookie to fail while trying to read a ledger fragment that, according to the ledger metadata, exists on the bookie. For this reason, when a bookie is started for the first time, its disk configuration is fixed for the lifetime of that bookie. Any change to its disk configuration, such as a crashed disk or an accidental configuration change, will result in the bookie being unable to start. That will throw an error like this:

If the change was the result of an accidental configuration change, the change can be reverted and the bookie can be restarted. However, if the change cannot be reverted, such as is the case when you want to add a new disk or replace a disk, the bookie must be wiped and then all its data re-replicated onto it.