A blog to share security, networking and cloud related technology information as @vCloudernBeer picked up on his search for his destiny in the cloud. (LinkedIn: https://www.linkedin.com/in/chowanthony)

Wednesday, November 19, 2014

OpenStack Series: Part 19 - Storage Policies for Object Storage

Storage Policy for object storage is an announced feature for the OpenStack Juno release.

Storage Policy is said to be the most significant feature for OpenStack Swift since its inception in 2010.

Why is Storage Policy so significant?Storage policy allows user to decide where and how to store the data. Storage policy is introduced in Swift 2.0. Before release 2.0 all data are stored the same way but in an OpenStack Infrastructure there can be multiple tenants and all will have different storage requirement. We do not want to create a Swift cluster for each tenants.

In Swift entities are stored in a container and is determined by the ring structure. I have an article on OpenStack Swift that talk about how each object is represented by:

/v1/account/container/object format.

With a modified MD5 hashing, the URI of the object is translated to where the data is stored in the physical device.

With Storage Policy, there are multiple rings and each rings is assigned a policy. The policy can control:

Level of replication

Type Storage device used to store the object that has different I/O performance

Geographic location/availability Zone that the data is to be store

Backend Storage system to be used such as Kinetic or GlusterFS

With the ability for the user to tune how and where the data is store allows for better support for multiple tenants in an OpenStack Infrastructure.

Take for example the level of replication, there are some data that can be reproduced from another source and do not need to be replicated 3 times in the storage system. User can now specify that the data to be replicated 2 times. With this change the user is able to save 33% of the physical storage required.

Storage Policy
For detailed description of OpenStack Swift Storage Policy, we can go to this OpenStack Documentation.

I will highlight a few important points from the article:

Policies are implemented at the container level

This allows for minimum changes to the existing API

X-Storage-Policy header is used to set the policy for the container

Each container has a new special immutable metadata element - Storage Policy Index

Policies are assigned when a container is created

Once a policy is assigned to a container, it can not be changed, unless it is deleted and re-created

Containers have a many-to-one relationship with policies - many containers can have the same policy

For working with pre 2.0 version of Swift, there is the Policy-0 in which the index 0 is used

User can specify a default policy for the Swift cluster

Default policy is used when no policy is configured by the user.

Default policy is to be use for Swift version 2.0 and beyond

Policy-0 is to be used by pre 2.0 version of Swift where Storage Policy is not supported.

Erasure Code

Storage
Policy paves the way for Erasure Code in OpenStack Swift which is another
significant enhancement for the project.

Erasure Code is one kind
of forward
error correction code which was invented by an American mathematician
Richard Hamming in 1950.It can be used
as a replacement of RAID for data protection.The advantage of Erasure Code over RAID is its ability to reduce the
resource for data recovery in terms of speed and require less storage
space.The only drawback for Erasure
code over RAID is that it is very computational intensive.With Storage Policy, user can chose between
erasure code or replication for data protection just as they can choose the
level of replication be it 3 time or 2 times.