To manage data storage efficiently, OpenIO has implemented a flexible rule-based system named “Storage policies”
which allows the platform administrator to easily organize the way data is stored, to fit his current and future needs.

In particular, “Storage policies” are the way to manage tiering and data protection.

Slots are names associated to services, which are useful for the administrator for categorizing services on his cluster.
Using slots is a way to tag services. For example, it can be useful for the administrator to declare a slot grouping
services sharing common characteristics like storage media type or physical location.
These slots are used in storage pools declaration.

The administrator can declare the location of each service as a dot-separated string like room1.rack1.server2.volume4
(1 to 4 words).
Locations can be used in pools configuration in order to allow finding services that are far from each other when
doing erasure coding or data replication.

The distance between two services is 4 less the number of words in common, starting from the left.

For example :

room1.rack1.srv12.vol4 and room1.rack1.srv12.vol5 have a distance of 1

room1.rack1.srv12.vol4 and room1.rack2.srv11.vol4 have a distance of 3.

Pools define the rules that the load-balancer should follow when picking services.
They are configured as a set of targets. Each target declares the number of services to pick,
followed by one or more slots to use. The slots will be consumed in the same order as they are declared,
thus providing a fallback mechanism when more that one slot is assigned to a target.

[pool:fastrawx3]
# Pick 3 SSD rawx, or any rawx if SSD is not available
targets=3,rawx-ssd,rawx
[pool:rawxevenodd]
# Pick one "even" and one "odd" rawx
targets=1,rawx-even;1,rawx-odd
[pool:rawx2]
# As with rawxevenodd, but with permissive fallback on any rawx
targets=1,rawx-even,rawx;1,rawx-odd,rawx
[pool:zonedrawx3]
# Pick one rawx in Europe, one in USA, one in Asia, or anywhere if none available
targets=1,rawx-europe,rawx;1,rawx-usa,rawx;1,rawx-asia,rawx
[pool:zonedrawx2]
# Pick 2 rawx services in Europe (or in Asia if there are no more in Europe) and 2 in the USA (or in Asia if there are no more in the USA), and ensure a minimum distance of 2 between each service.
targets=2,rawx-europe,rawx-asia;2,rawx-usa,rawx-asia
min_dist=2

OpenIO designed a dynamic mechanism that automatically selects the best data protection mechanism according to the characteristics of the stored object, thus combining optimal efficiency and data protection. It is thus possible to have several storage policies for each domain / user / bucket, then to assign rules to apply these policies, e.g. depending on the size of the object to be stored.

In a context where we do not know the type of file to store, the use of the dynamic data protection mechanism implemented by OpenIO is therefore recommended.

As an example, both x3 replication and 14+4 erasure coding can be assigned for a new bucket, with the rule that files smaller than 128KiB are replicated, while larger files use erasure coding.

The dynamic storage policies are configured in the oio-swift configuration file /etc/oio/sds/[NS]/oioswift-X/proxy-server.conf

Two parameters must be configured:

oio_storage_policies:

The list of storage policies that are actually manageable by the Swift gateway. All of them must exist in the target OpenIO SDS platform, and if they require erasure coding support, they must rely on an existing/deployed liberasurecode backend.

auto_storage_policies:

The dynamic storage policies configuration which which will be applied.

The format is: “DEFAULT,POLICY:THRESHOLD[,POLICY:THRESHOLD]*”

“DEFAULT”: the storage policy to be used when the size of the content is unknown, with a notable case for ‘Transfer-Encoding: chunked’ uploads.
“POLICY:THRESHOLD”: the policy POLICY must be used when the size of the content is over THRESHOLD.

At least one POLICY:THRESHOLD rule is required. When several tuples are present, upon an upload the whole list is iterated and the best match is kept as the actual value.

The fields storage_policy. corresponds to all the storage policies declared in the namespace.
The fields data_security. corresponds to all the data security policies declared in the namespace.
The field storage_policy corresponds to the default storage policy of the namespace.