Tag Archives: emc

This years EMC Forum 2014 New York is approaching quickly, October 8th is just around the corner and I am really excited!!!

Over the years we (FusionStorm NYC) have typically prepped a demo for EMC Forum and rolled a 20U travel rack complete with networking, servers, storage arrays, and whatever else we needed for the demo to EMC Forum. In the past we’ve done topics like WAN Optimization, VMware SRM, VMware vCOps, and last year XtremSW. As a techie it’s always been cool to show the flashing lights, how things are cabled, etc… but this year it’s all about the cloud, commodity compute and and SDW (Software Defined Whatever) and elasticity which is why there will be no 20U travel rack, nothing more than a laptop and an ethernet cable that will connect to a ScaleIO 1.3 AWS (Amazon Web Services) implementation. The base configuration that I have built in AWS specifically for EMC Forum looks like this:

10 x SDS (ScaleIO Data Server) Nodes in AWS (SLES 11.3)

Each node has 1 x 100 GB EBS SSD attached

1 x SDC (ScaleIO Data Client) Node in AWS (Windows 2008 R2)

Using IOmeter and vdbench on SDC to generate workload

Single Protection Domain: awspdomain01

Single Pool: awspool01

40GB awsvol01 volume mapped to Windows SDC

Terminology:

Meta Data Manager (MDM) – Configures and monitors the ScaleIO system. The MDM can be configured in a redundant Cluster Mode with three members on three servers, or in a Single Mode on a single server.

ScaleIO Data Server (SDS) – Manages the capacity of a single server and acts as a backend for data access. The SDS is installed on all servers that contribute storage devices to the ScaleIO system. These devices are accessed through the SDS.

ScaleIO Data Client (SDC) – A lightweight device driver that exposes ScaleIO volumes as block devices to the application residing on the same server on which the SDC is installed.

New Features in ScaleIO v1.30:ScaleIO v1.30 introduces several new features, listed below. In addition, it includes internal enhancements that increase the performance, capacity usage, stability, and other storage aspects.
Thin provisioning: In v1.30, you can create volumes with thin provisioning. In addition to the on-demand nature of thin provisioning, this also yields much quicker setup and startup times.Fault Sets: You can define a Fault Set, a group of ScaleIO Data Servers (SDSs) that are likely to go down together (For example if they are powered in the same rack), thus ensuring that ScaleIO mirroring will take place outside of this fault set.Enhanced RAM read cache: This feature enables read caching using the SDS server RAM.Installation, deployment, and configuration automation: Installation, deployment, and configuration has been automated and streamlined for both physical and virtual environments. The install.py installation from previous versions is no longer supported.This is a significant improvement that dramatically improves the installation and operational management.

VMware management enhancement: A VMware, web-based plug-in communicates with the Metadata Manager (MDM) and the vSphere server to enable deployment and configuration directly from within the VMware environment.

GUI enhancement: The GUI has been enhanced dramatically. In addition to monitoring, you can use the GUI to configure the backend storage elements of ScaleIO.

REST API: A Representational State Transfer (REST) API can be used to expose monitoring and provisioning via the REST interface.OpenStack support: ScaleIO includes a Cinder driver that interfaces with OpenStack, and presents volumes to OpenStack as block devices which are available for block storage. It also includes an OpenStack Nova driver, for handling compute and instance volume-related operations.Planned shutdown of a Protection Domain: You can simply and effectively shut down an entire Protection Domain, thus preventing an unnecessary rebuild/rebalance operation.
Role-based access control: A role-based access control mechanism has been introduced.Operationally Planned shutdown of a Protection Domain is a big enhancement!!

IP roles: For each IP address associated with an SDS, you can define the communication role that the IP address will have: Internal—between SDSs and MDMs; External—between ScaleIO Data Clients (SDCs) and SDSs; or both. This allows you to define virtual subnets.MDM— IP address configuring: You can assign up to eight IP addresses to primary, secondary, and tie-breaker MDM servers, thus enhancing MDM communication redundancy. In addition, you can configure a specific IP address that the MDM will use to communicate with the management clients. This enables you to configure a separate management network so you can run the GUI on an external system.

While big iron (traditional storage arrays) probably are not going away very soon the cool factor just doesn’t come close to SDW (Software Defined Whatever) so stop by the FusionStorm booth at EMC Forum and let’s really dig into some very cool stuff.

If you are really interesting in digging into to ScaleIO one-on-one please email me (rbocchinfuso@fusionstorm.com) or tweet me @rbocchinfuso and we can setup a time where we can focus, maybe a little more than will be possible at EMC Forum.

Looking forward to seeing at the FusionStorm booth at EMC Forum 2014 New York on October 8th. If you haven’t registered for EMC Forum you should register now, Forum is only 12 days away.

ScaleIO – Chapter III: Scale out for what? Answer: Capacity, performance, because it’s cool and because of AWS I can.

So at this point I decided I wanted to deploy 100+ SDS nodes in AWS, just because I can.

Note: I attempted to be as detailed as possible with this post but of course there are some details that I intentionally excluded because I deemed them too detailed and there may be some things I just missed.

The first thing I did was create an AMI image using one of the fully configured SDS nodes, figured this would be the easiest way to deploy 100+ nodes. Being new to AWS I didn’t realize the node I was imaging was going to take the node offline (actually reboot the node, I noticed later that there is a check box that allows you you chose if you want to reboot the instance or not). There is always a silver lining especially when the environment is disposable and easily reconstructed.

When i saw my PuTTY session disconnect I flipped over to the window and sure enough there it was:

Flipped to the ScaleIO console, pretty cool (yes I am easily amused):

aws1sds node down and system running in degraded mode. Flipped to the Linux host I have been using for testing just to see if the volume was accessible and data was intact (appears to be so although it’s not like I did some exhaustive testing here):

Flipped back tot he ScaleIO console just to see the state and aws1sds was back online and protection domain awspdomain01 was re-balancing:

Protection Domain awspdomain01 done rebalancing and system returned to 100% healthy state:

So now that my AMI image is created I am going to deploy a new instance and see how it looks, make sure everything is as it should be before deploying 100+ nodes.

Selected instance, everything looked good just had to add to the appropriate Security Group.

Named this instance ScaleIO_AWS5 and I am going to add to existing awspdomain01 as a test. When I do the 100 node deployment i am going to create a new Protection Domain, just to keep things orderly.

I think this is because the aws1sds was already added to the Protection Domain when I created the AMI image. Now that I removed it from the Protection Domain I am going to terminate the aws5sds instance and create a new AMI image from the aws1sds.

Note: I retagged my existing for nodes “ScaleIO_SDS_awspdomain01_pool03” I will use this tag on the 16 new nodes I am deploying, will make it easy to filter in the AWS console. Will be important when I grab the details to add the SDS nodes to the awspdomain01 and pool03.

Highlight the nodes in the AWS console and cut-and-paste to Excel (Note: I filter my list by the tag we applied in the previous step):

Tip: I like to highlight from the bottom right of the list to the top left (little easier to control). Cut-and-Past to Excel (or any spreadsheet).

Past (ctrl-v) to Excel without formatting and then do a little cleanup:

You should end up with a sheet that looks like this:

Step 2: Create the commands in Excel to add the new SDS AWS instances to ScaleIO awspdomain01 and pool03

Note: I am going to hide columns we don’t need to make the sheet easier to work with.

The only columns we really need are column I (Public IP) and Column L (Launch Time) but I am going to keep column A (Name/Tag ) as well because in a larger deployment scenario you may want to filter on the Name Tag.

I am also going add some new columns:

Column N (Device Name): This is the device inside the SDS instance that will be used by ScaleIO

Column O,P & Q (node uid, node type and sds_name): Probably don’t need all of these but I like the ability to filter by node type, sds_name is a concat of nod uid and node type.

Column R (Protection Domain): This is the Protection Domain that we plan to place the SDS node in

Column S (Pool): This is the Pool we want the SDS storage to be placed in

You will also notice that “mdm_ip” is in A1 and the mdm ip address is in A2 (A2 is also labeled mdm_ip)

Next I am going to create the commands to add the SDS nodes to our existing awspdomain01 Protection Domain and pool03.

That was pretty easy and pretty cool. So I am going to take a quick look what I have spent in AWS so far to do everything I posted in my ScaleIO – Chapter II and and ScaleIO – Chapter III posts. Going to kickoff some benchmarks and will revisit the cost increase.

$1.82 ,the moral of the story it it’s to cheap to stay stupid 🙂 The world is changing, get on board!

Any yes the title of this post may be an homage to Lil Jon and the complex simplicity of “Turn Down for What”

Below you can see the R/W concurrency across the nodes as the benchmark runs.

6/22/2014 Update: After running some IOzone benchmarks last night it looks like I used about $7 in bandwidth running the tests.

6/25/2014 Update: Burn it down before I build it up.

AWS cost over the 4-5 days I had the 20 nodes deployed, so I decided to tear it down before I do the 200 node ScaleIO AWS deployment.

From a cleanup perspective I removed 17 of the 20 SDS nodes, trying to figure out how to remove the last 3 SDS nodes, the Pool and the Protection Domain. Haven’t worked on this much but once I get it done I plan to start work on the 200 node ScaleIO deployment and testing.

Khan: “Dreadnought class. Two times the size, three times the speed. Advanced weaponry. Modified for a minimal crew. Unlike most Federation vessels, it’s built solely for combat.”

Extending ScaleIO to the public cloud using AWS RHEL 6.5 t1.micro instances and EBS and federating with my private cloud ScaleIO implementation.

This post is about federating ScaleIO across the public and private cloud not the “Federation” of EMC, VMware, Pivotal and RSA” Sorry but who doesn’t love the “Federation”, if for nothing else it takes me back to my childhood.

My Childhood:

If you don’t know what the above means and the guy on the right looks a little familiar, maybe from a Priceline commercial don’t worry about it, I just means your part of a different generation (The Next Generation ). If you are totally clueless about the above you should probably stop reading now, if you can identify with anything above it is probably safe to continue.

My Adulthood:

Wow, the above pictorial actually a scares me a little, I really haven’t come very far

Anyway let’s get started exploring the next frontier, certainly not the final frontier.

Note: I already deployed the four (4) RHEL 6.5 t1.micro AWS instances that I will be using in this post. This post focuses on the configuration of the instances not the deployment of the AWS instances. In Chapter III of this series I deploy at a larger scale using a AMI image that I generated from ScaleIO_AWS1 which you will see hos to configure in this posts.

Login to AWS RHEL instance via SSH (Note: You will have to setup the required AWS keypairs, etc…)

Note: I am adding AWS SDS nodes (ScaleIO Data Server NOT Software Defined Storage) to an existing private cloud ScaleIO implementation so this will only cover installing the SDS component and the required steps to add the SDS nodes to the exiting ScaleIO deployment.

ScaleIO Data Server (SDS) – Manages the capacity of a single server and acts as a backend for data access. The SDS is installed on all servers that contribute storage devices to the ScaleIO system. These devices are accessed through the SDS.

Below is a what the current private cloud ScaleIO deployment looks like:

The goal here is to create pool03 which will be a tier of storage that will reside in AWS.

Once logged into your AWS RHEL instance validate that the following packages are installed: numactl and libaio

#sudo –s

#yum install libaio

#yum install numactl

For SDS nodes port 7072 needs to opened. Because I have a the ScaleIO security group I can make the change in the Security Group.

Note: This is an environment that is only for testing, there is nothing here that I care about, the data, VMs, etc… are all disposable this opening port 7072 to the public IP is of no concern to me. In an actual implementation there would likely be a VPN between the public and private infrastructure components and there would not be a need to open port 7072 on the public IP address.

Storage Pool – A Storage Pool is a subset of physical storage devices in a Protection Domain. Each storage device belongs to one (and only one) Storage Pool. A volume is distributed over all devices residing in the same Storage Pool. This allows more than one failure in the system without losing data. Since a Storage Pool can withstand the loss of one of its members, having two failures in two different Storage Pools will not cause data loss.

Screenshot below is activity while running I/O load to the AWS volume:

Important to note that the public / private ScaleIO federation was PoC just to see how it could / would be done. It was not intended to be a performance exercise but rather a functional exercise. Functionally things worked well, the plan is now to scale up the number and type of nodes to see what type of performance I can get from this configuration. Hopefully no one will push back on my AWS expenses 🙂

I did some quick testing with FFSB and FIO, after seeing the results returned by both FFSB and FIO i wanted to grab some additional data so I could do a brief analysis so ran IOzone (http://www.iozone.org/) against the AWS ScaleIO volume (awspdomain01, pool03) and the local ScaleIO HDD volume (pdomain01, pool02) for comparison.

So this post is a slightly modified version of some internal documentation that I shared with my management, the folks at Dell who graciously donated the compute, PCIe SSDs and 10 Gig network for this project and the folks at EMC who of course donated the the ScaleIO licensing (and hopefully soon the ViPR 2.0 licensing). Due to the genesis of this post and my all around lack of time for editing some of the writing and tense in this post may not always be logical.

Just about everyone knows that Dell and EMC aren’t exactly best friends these days but could there be a better match for this architecture? Cough, cough, Supermicro, cough, cough Quanta…. but seriously the roll your own Supermicro, Linux, Ceph, Swift, etc… type architecture isn’t for everyone, some people still want reasonably supported hardware and software at pricing that rivals the likes of Supermicro and OSS (Open-source software). BTW, there is a cost to OSS, it’s called your time. Think I need to build a private scale-out architecture, I want it to be lower cost, high performance, support both physical and virtual environments and I want the elasticity and the the ability to scale to the public cloud and oh yeah, I want a support mechanism that is enterprise class for both the hardware and software that I deploy as part of this solution.

Most have heard the proverb “the enemy of my enemy is my friend”, the reality is that Dell and EMC are fenemies whether they know it or not, are willing to admit it or not because I am currently implementing Chapter III in this series and trust me the enemy (competition) is a formidable one, known as the elastic public cloud! Take your pick, AWS, Google, Azure, ElasticHosts, Bitnami, GoGrid, Rackspcae, etc… Will they replace the private cloud, probably not (at least not in the foreseeable future) as there are a number of reasons the private cloud needs to exist and will continue to exist, reasons like regulations, economics, control, etc…

In a rapidly changing landscape where the hardware market is infected with the equivalent of the ebola virus, hemorrhaging interest, value and margin. The sooner we accept this fact and begin to adapt (really adapt) the better our chances of avoiding extinction. Let’s face it there are many OEMs, VARs, individuals, etc… who are more focused on containment rather than a cure. All of us who have who have sold, architected, installed, maintained, etc… traditional IT infrastructure face a very real challenge from a very real threat. The opposing force possess the will and tactics of the Spartans and the might of the Persians, if we (you and I) don’t adapt and think we can continue with business as usual, more focused on containment than curing our own outdated business models we will face a very real problem in the not so distant future. Having said the aforementioned there is no doubt that EMC is hyperfocused on software, much of it new (e.g. – ViPR, ScaleIO, Pivotal, etc…) and many tried and true platforms already instantiated in software or planned to be (e.g. – RecoverPoint, Isilon, etc…). As compute costs continue to plummet more functionality can be supported at the application and OS layers which changes the intelligence needed from vendors. In the IT plumbing space (specifically storage) the dawn of technologies like MS Exchange DAGs and SQL AlwaysOn Availability Groups have been a significant catalyst for the start of a significant shift, the focus has begun to move to features like automation rather than array based replication.

The market is changing fast, we are all scrambling to adapt, figure out how we will add value in the future of tomorrow. I am no different than anyone else, spending my time and money on AWS.

Anyway there is too much to learn and not enough time, I read more than ever on my handheld device (maybe the 5” screen handheld device is a good idea, always thought it was too large). As I work on Chapter II of this series I found myself at dinner the other night with my kids reading documentation on the Fabric documentation. Trying to decide is I should use Fabic to do automate my deployment or just good old shell scripts and the AWSCLI, then my mind started wondering to what do I do after Chapter III, maybe there is a Chapter IV and V with different instance types or maybe I should try Google Compute or Azure, so many choices so little time

Update: Chapter II and Chapter III of this series already completed and I have actually begun working on Chapter IV.

For sure there will be a ScaleIO and ViPR chapter but I need to wait for ViPR 2.0

This exercise is just my humble effort to become competent in the technologies that will drive the future of enterprise architecture and hopefully say somewhat relevant.

ScaleIO environment is up and running and able to be demoed (by someone who knows the config and ScaleIO because most of the configuration is done via CLI and require some familiarity given the level of documentation at this point)

ScaleIO Console

Below you can see that there is 136 GB of available aggregate capacity available across all the ScaleIO nodes (servers).

This is not intended to be a ScaleIO internals deep dive but here is some detail on how the ScaleIO usable capacity is calculated:

Total aggregate capacity across SDS nodes:

100/# of SDS servers = % for spare capacity

1/2 half of the remaining capacity for mirroring

For example in a ScaleIO cluster with 4 nodes and 10 GB per node the math would be as follows:

40 GB of aggregate capacity

100/4 = 25% (or 10 GB) for spare capacity

.5 * 30 GB (remaining capacity) = 15 GB of available/usable capacity

Configured VMware datastores:

svrsan201#_SSD – This is the local PCIe SSD on each ESX server (svrsan201#)

svrsan201#_local – This is the local HDDs on each ESX server (svrsan201#)

The ScaleIO GUI console seen above is a jar file that needs to be SCPed from the MDM host to your local machine to be run (it lives in /opt/scaleio/ecs/mdm/bin/dashboard.jar). I found this to be a bit arcane so installed thhtp (http://www.acme.com/software/thttpd/) on the MDM server to make it easy to get the dashboard.jar file.

Due to limited time and the desire to capture some quick statistics a single run was made against each device using IOzone using the local HDD and SSD devices for the baseline sample data and the ScaleIO volumes as the comparative data set.

Note: Local (HDD | SSD) = a single device in in a single ESX server, ScaleIO (HDD | SSD) makes used the same HDD and SSD device in the server used in the local test but also all other HDD | SSD devices in other nodes, to provide aggregate capacity, performance and protection.

ViPR Installed and Configured

ViPR is deployed but version 1.1.0.2.16 does not support ScaleIO.

Note: ScaleIO support will be added in ViPR version 2.0 which is scheduled for release in Q2.

EMC ViPR SRM deployed but haven’t really done anything with it to date.