Meta

Maintainers

Project description

The zc.zk package provides some high-level interfaces to the low-level
zookeeper extension. It’s not complete, in that it doesn’t try, at
this time, to be a complete high-level interface. Rather, it provides
facilities we need to use ZooKeeper to connect services:

ZODB database clients and servers

HTTP-based clients and services

Load balancers and HTTP application servers

The current (initial) use cases are:

Register a server providing a service.

Get the addresses of servers providing a service.

Get and set service configuration data.

Model system architecture as a tree.

This package makes no effort to support Windows. (Patches to support
Windows might be accepted if they don’t add much complexity.)

You can install this as you would any other distribution. Note,
however, that you must also install the Python ZooKeeper binding
provided with ZooKeeper. Because this binding is packaged a number of
different ways, it isn’t listed as a distribution requirement.

An easy way to get the Python zookeeper binding is by installing
zc-zookeeper-static, which is a self-contained statically built
distribution.

The ZooKeeper constructor takes a ZooKeeper connection string, which is a
comma-separated list of addresses of the form HOST:PORT. It defaults
to '127.0.0.1:2181', which is convenient during development.

register_server creates a read-only ephemeral ZooKeeper node as a
child of the given service path. The name of the new node is (a
string representation of) the given address. This allows clients to
get the list of addresses by just getting the list of the names of
children of the service path.

Ephemeral nodes have the useful property that they’re automatically
removed when a ZooKeeper session is closed or when the process
containing it dies. De-registration is automatic.

When registering a server, you can optionally provide server (node)
data as additional keyword arguments to register_server. By default,
the process id is set as the pid property. This is useful to
tracking down the server process. In addition, an event is generated,
providing subscribers to add properties as a server is being
registered. (See Server-registration events.)

This example defines a tree with 3 top nodes, lb and cms, and
retail. The retail node has two sub-nodes, providers and
databases and a property threads.

The /retail/databases node has symbolic link, main and a
ugc sub-node. The symbolic link is implemented as a property named
`` We’ll say more about symbolic links in a later section.

The lb, cms and retail nodes have types. A type is
indicated by following a node name with a colon and a string value.
The string value is used to populate a type property. Types are
useful to document the kinds of services provided at a node and can be
used by deployment tools to deploy service providers.

Note that nodes containing (directly or recursively) ephemeral nodes
will never be trimmed. Also node that top-level nodes are never
automatically trimmed. So we weren’t warned about the unreferenced
top-level nodes in the import.

>>> zk.delete_recursive('/fooservice')
Not deleting /fooservice/providers/192.168.0.42:8080 because it's ephemeral.
Not deleting /fooservice/providers/192.168.0.42:8081 because it's ephemeral.
Not deleting /fooservice/providers/192.168.0.42:8082 because it's ephemeral.
/fooservice/providers not deleted due to ephemeral descendent.
/fooservice not deleted due to ephemeral descendent.

ZooKeeper doesn’t have a concept of symbolic links, but zc.zk
provides a convention for dealing with symbolic links. When trying to
resolve a path, if a node lacks a child, but have a property with a
name ending in ' ->', the child will be found by following the
path in the property value.

The resolve method is used to resolve a path to a real path:

>>> zk.resolve('/lb/pools/cms/providers')
u'/cms/providers'

In this example, the link was at the endpoint of the virtual path, but
it could be anywhere:

Children and Properties objects have a path attribute that
has the value passed to the children or properties
methods. They have a real_path attribute that contains the path
after resolving symbolic links. Let’s suppose we want to move the
database node to ‘/databases/cms’. First we’ll export it:

Property links are indicated with =>. The value is a node path and
optional property name, separated by whitespace. If the name is
ommitted, then the refering name is used. For example, the name could
be left off of the property link above.

If a node is deleted and Children or Properties instances have
been created for it, and the paths they were created with can’t be
resolved using symbolic links, then the instances’ data will be
cleared. Attempts to update properties will fail. If callbacks have
been registered, they will be called without arguments, if possible.
It would be bad, in practice, to remove a node that processes are
watching.

It’s common to use an empty string for a host name when calling bind
to listen on all IPv4 interfaces. If you pass an address with an
empty host to register_server and netifaces is installed, then
all of the IPv4 addresses [1] (for the given port) will be
registered. If netifaces isn’t installed and you pass an empty host
name, then the fully-qualified domain name, as returned by
socket.getfqdn() will be used for the host.

When register_server is called, a zc.zk.RegisteringServer
event is emmitted with a properties attribute that can be updated by
subscribers prior to creating the ZooKeeper ephemeral node. This
allows third-party code to record extra server information.

Events are emitted by passing them to zc.zk.event.notify. If
zope.event is installed, then zc.zk.event.notify is an alias
for zope.event.notify, otherwise, zc.zk.event.notify is an
empty function that can be replaced by applications.

zc.zk takes care of ZooKeeper session management for you. It
establishes and, if necessary, reestablishes sessions for you. In
particular, it takes care of reestablishing ZooKeeper watches and
ephemeral nodes when a session is reestablished.

Note

To reestablish ephemeral nodes, it’s necessary for zc.zk to
track node-moderation operations, so you have to access the
ZooKeeper APIs through the zc.zk.ZooKeeper object, rather than
using the low-level extension directly.

The zc.zk package provides a utility script for importing a
ZooKeeper tree. So, for example, given the tree:

/provision
/node1
/node2

In the file mytree.txt, we can import the file like this:

$ zookeeper_import zookeeper.example.com:2181 mytree.txt /fooservice

The import script provides the same features as the import_tree
method, with the exception that it provides less flexibility for
specifing access control lists. Use the --help option to see how
to use it.

The textual tree representation can be used to model and analyze a
system architecturte. You can get a parsed representation of a tree
using zc.zk.parse_tree to parse a text tree representation
generated by hand for import, or using the export_tree method.

If a session timeout (session_timeout) isn’t specified, the
ZooKeeper server’s default session timeout is used. If the
connection to ZooKeeper flaps, setting this to a higher value can
avoid having clients think a server has gone away, when it hasn’t.
The downside of setting this to a higher value is that if a server
crashes, it will take longer for ZooKeeper to notice that it’s
gone.

The wait flag indicates whether the constructor should wait
for a connection to ZooKeeper. It defaults to False.

If a connection can’t be made, a zc.zk.FailedConnect exception
is raised.

Note that there is a fair bit of machinery in zc.zk.Children
objects to support keeping them up to date, callbacks, and cleaning
them up when they are no-longer used. If you only want to get the
list of children once, use get_children.

create_recursive(path, data, acl)

Create a non-ephemeral node at the given path, creating parent
nodes if necessary.

close()

Close the ZooKeeper session.

This should be called when cleanly shutting down servers to more
quickly remove ephemeral nodes.

delete_recursive(path[, dry_run[, force]])

Delete a node and all of it’s sub-nodes.

Ephemeral nodes or nodes containing them are not deleted by
default. To force deletion of ephemeral nodes, supply the force
option with a true value.

The dry_run option causes a summary of what would be deleted to be
printed without actually deleting anything.

export_tree(path[, ephemeral[, name]])

Export a tree to a text representation.

path

The path to export.

ephemeral

Boolean, defaulting to false, indicating whether to include
ephemeral nodes in the export. Including ephemeral nodes is
mainly useful for visualizing the tree state.

name

The name to use for the top-level node.

This is useful when using export and import to copy a tree to
a different location and name in the hierarchy.

Normally, when exporting the root node, /, the root isn’t
included, but it is included if a name is given.

get_children(path)

Get a list of the names of the children the node at the given path.

This is more efficient than children when all you need is to
read the list once, as it doesn’t create a zc.zk.Children
object.

get_properties(path)

Get the properties for the node at the given path as a dictionary.

This is more efficient than properties when all you need is to
read the properties once, as it doesn’t create a
zc.zk.Properties object.

import_tree(text[, path='/'[, trim[, acl[, dry_run]]]])

Create tree nodes by importing a textual tree representation.

text

A textual representation of the tree.

path

The path at which to create the top-level nodes.

trim

Boolean, defaulting to false, indicating whether nodes not in
the textual representation should be removed.

acl

An access control-list to use for imported nodes. If not
specified, then full access is allowed to everyone.

dry_run

Boolean, defaulting to false, indicating whether to do a dry
run of the import, without applying any changes.

is_ephemeral(path)

Return True if the node at path is ephemeral,``False`` otherwise.

ln(source, destination)

Create a symbolic link at the destination path pointing to the
source path.

If the destination path ends with '/', then the source name is
appended to the destination.

Note that there is a fair bit of machinery in zc.zk.Properties
objects to support keeping them up to date, callbacks, and cleaning
them up when they are no-longer used. If you only want to get the
properties once, use get_properties.

register_server(path, address, acl=zc.zk.READ_ACL_UNSAFE, **data)

Register a server at a path with the address.

An ephemeral child node of path will be created with name equal
to the string representation (HOST:PORT) of the given address.

zc.zk.RegisteringServer events are generated during server
registration to allow third-party libraries to record additional
properties on new server nodes.

Added a testing module that provides ZooKeeper emulation for
testing complex interactions with zc.zk without needing a running
ZooKeeper server.

zc.zk.Children objects now have a __len__, which is mainly useful
for testing whether they are empty.

Separated parsing of tree representations into a separate fuction,
parse_tree, to support analysis and added a graphvis
demonstration module to show how one might use tree representations
for system modeling.

Added a zc.zk.ZooKeeperwait constructor argument to wait
indefinately for ZooKeeper to be available.

(1, 2) It’s a little more complicated. If there are non-local
interfaces, then only non-local addresses are registered. In
normal production, there’s really no point in registering local
addresses, as clients on other machines can’t make any sense of
them. If only local interfaces are found, then local addresses
are registered, under the assumption that someone is developing on
a disconnected computer.