This is experimental software. Interfaces and implementation are subject to change. If you are interested in using this in production, please get in touch to gauge the current state of stability.

This module implements an abstract interface to a sharded key-value store. The storage backends as well as the "continuum" are pluggable. "Continuum" is to mean "the logic that decides in which shard a particular key lives". Typically, people use consistent hashing for this purpose and very commonly the choice is to use ketama specifically. See below for references.

Beside the abstract querying interface, this module also implements logic to add one or more servers to the continuum and use passive key migration to extend capacity without downtime. Do make it a point to understand the logic before using it. More on that below.

ShardedKV allows instrumentation for logging and debugging by setting the logger attribute of the main ShardedKV object, and/or its continuum and/or any or all storage sub-objects. If set, the logger attribute must be an object implementing the following methods:

trace

debug

info

warn

error

fatal

which take a string parameter that is to be logged. These logging levels might be familiar since they are taken from Log::Log4perl, which means that you can use a Log::Log4perl::Logger object here.

Additionally, the following methods must return whether or not the given log level is enabled, to potentially avoid costly construction of log messages:

A hashref of storage objects, each of which represents one shard. Keys in the hash must be the same labels/shard names that are used in the continuum. Each storage object must implement the ShardedKV::Storage role.

Given a key, fetches the value for that key from the correct shard and returns that value or undef on failure.

Different storage backends may return a reference to the value instead. For example, the Redis and Memory backends return scalar references, whereas the mysql backend returns an array reference. This might still change, likely, all backends may be required to return scalar references in the future.

Given a ShardedKV::Continuum object, this sets the migration_continuum property of the ShardedKV, thus beginning a passive key migration. Right now, the only kind of migration that is supported is adding shards! Only one migration may be in effect at a time. The passive qualification there is very significant. If you are, for example, using the Redis storage backend with a key expiration of one hour, then you know, that after letting the passive migration run for one hour, all keys that are still relevant will have been migrated (or expired if they were not relevant).

If there is a migration continuum, then for get requests, that continuum is used to find the right shard for the given key. If that shard does not have the key, we check the original continuum and if that points the key at a different shard, we query that.

For delete requests, we also attempt to delete from the shard pointed to by the migration continuum AND the shard pointed to by the main continuum.

For set requests, we always only use the shard deduced from the migration continuum

end_migration() will promote the migration continuum to the regular continuum and set the migration_continuum property to undef.