Setting Up ZooKeeper State Manager

Heron relies on ZooKeeper for a wide variety of cluster coordination tasks. You
can use either a shared or dedicated ZooKeeper cluster.

There are a few things you should be aware of regarding Heron and ZooKeeper:

Heron uses ZooKeeper only for coordination, not for message passing, which
means that ZooKeeper load should generally be fairly low. A single-node
and/or shared ZooKeeper may suffice for your Heron cluster, depending on
usage.

Heron uses ZooKeeper more efficiently than Storm. This makes Heron less likely
than Storm to require a bulky or dedicated ZooKeeper cluster, but your use
case may require one.

ZooKeeper State Manager Configuration

You can make Heron aware of the ZooKeeper cluster by modifying the
statemgr.yaml config file specific for the Heron cluster. You’ll
need to specify the following for each cluster:

heron.class.state.manager — Indicates the class to be loaded for managing
the state in ZooKeeper and this class is loaded using reflection. You should set this
to com.twitter.heron.statemgr.zookeeper.curator.CuratorStateManager

heron.statemgr.root.path — The root ZooKeeper node to be used by Heron. We recommend
providing Heron with an exclusive root node; if you do not, make sure that the following child
nodes are unused: /tmasters, /topologies, /pplans, /executionstate, /schedulers.

heron.statemgr.zookeeper.is.initialize.tree — Indicates whether the nodes under ZooKeeper
root /tmasters, /topologies, /pplans, /executionstate, and /schedulers need to created,
if they are not found. Set it to True if you could like Heron to create those nodes. If those
nodes are already there, set it to False. The absence of this configuration implies True.

heron.statemgr.zookeeper.session.timeout.ms — Specifies how much time in milliseconds
to wait before declaring the ZooKeeper session is dead.

heron.statemgr.zookeeper.connection.timeout.ms — Specifies how much time in milliseconds
to wait before the connection to ZooKeeper is dead.

heron.statemgr.zookeeper.retry.count — Count of the number of retry attempts to connect
to ZooKeeper

heron.statemgr.zookeeper.retry.interval.ms: Time in milliseconds to wait between each retry

Example ZooKeeper State Manager Configuration

Below is an example configuration (in statemgr.yaml) for a ZooKeeper running in localhost:

# local state manager class for managing state in a persistent fashionheron.class.state.manager:com.twitter.heron.statemgr.zookeeper.curator.CuratorStateManager# local state manager connection stringheron.statemgr.connection.string:"127.0.0.1:2181"# path of the root address to store the state in a local file systemheron.statemgr.root.path:"/heron"# create the zookeeper nodes, if they do not existheron.statemgr.zookeeper.is.initialize.tree:True# timeout in ms to wait before considering zookeeper session is deadheron.statemgr.zookeeper.session.timeout.ms:30000# timeout in ms to wait before considering zookeeper connection is deadheron.statemgr.zookeeper.connection.timeout.ms:30000# timeout in ms to wait before considering zookeeper connection is deadheron.statemgr.zookeeper.retry.count:10# duration of time to wait until the next retryheron.statemgr.zookeeper.retry.interval.ms:10000