Those of you who have followed me over the years know that I’m not shy when it comes to a good competitive dust-up. I’m OK with the usual puffery and slightly exaggerated claims. All part of the fun.

I’m not OK when I believe the claims are misleading.

One startup is working very hard to convince everyone that they (and they alone) are leading the current trend in HCI — hyperconverged infrastructure. One of their spokespeople even published a thoughtful piece listing the ten reasons why they thought they deserved the “leader” mantle.

While I admire their bravado, I felt the piece did a disservice to both the industry and to customers. I thought it grossly misrepresented both the current and future state of the market.

Perhaps most importantly, there was little talk about what mattered most to customers.

So — while staying positive — I’d like to share my "ten reasons" why I think VMware is leading — and will continue to lead — the hyperconverged marketplace.

Why Hyperconverged?

If we’re going to have a polite argument, we ought to at least define what we’re discussing.

The first wave was “converged” infrastructure: traditional compute, storage and network products engineered to be consumed and supported as a single block of infrastructure.

The fundamental appeal was a drastically simplified customer experience, which gave IT time and resources to go do other things. VCE Vblocks established the market and validated the model, with several others following suit. As we stand today, converged infrastructure is a successful and proven model that continues to grow.

Meanwhile, a few enterprising startups created a software-based storage layer that eliminated the need for an external storage array, and dubbed themselves “hyperconverged”.

Hence our discussion today ...

#1 — Hyperconverged Is About Software, Not Hardware

Hyperconverged solutions derive their value from the hypervisor being able to support all infrastructure functions in software, and without the need for separate dedicated hardware, such as a storage array or fibre channel switch. All players in this segment would mostly agree with this statement.

If hyperconverged is really about software (and not hardware), what’s the core software technology in just about every hyperconverged product available today?

VMware vSphere

It's ubiquitous in the data center, which explains why it's ubiquitous in the hyperconverged market. A key part of the story: vSphere implements the most popular infrastructure mgmt APIs in use today.

The harsh market reality is that there’s just not a lot of demand for non-vSphere-based hyperconverged solutions.

IT professionals know vSphere -- it's tried, tested and proven -- and that's what they want.

If we could convince a few industry analysts to focus on hyperconverged software vs. counting largely similar boxes with different vendor labels on them, their picture of the landscape would be quite different.

As far as claims to "market leadership" — without the power and presence of the VMware vSphere platform, there wouldn't be a converged or hyperconverged market to argue about.

#2 — Built-In Is Better Than Bolted-On

If the value proposition of hyperconverged derives from integrating infrastructure in software, it’s reasonable to argue that deeper, structural integration will be more valuable than various software assemblages that lack this key attribute.

There shouldn’t be a need for a separate management interface.

There shouldn’t be a need for a separate storage layer that runs as a guest VM with dedicated resources, consuming precious resources and demanding attention.

There shouldn’t be a need for multiple installation / patching / upgrade processes.

There shouldn’t be a need to get support from two or more vendors.

And so on.

Within VMware, we use the term “hypervisor converged” to differentiate this important architectural difference between built-in vs. bolted-on.

If our discussion of "market leadership" includes any notion of creating a more simple experience for users, I would argue that it’s hard to compete with features that are simple extensions of the hypervisor.

#3 — Having Lots Of Hardware Choices Is A Really Good Thing

If hyperconverged is really about software, why are many so paradoxically focused on “the box”? It’s nothing more than a convenient consumption option for someone who wants a fast time-to-value over other considerations.

Ideally, hyperconverged -- as a concept -- shouldn’t be welded to specific hardware.

For those that want a convenient consumption option such as prefab appliance with a locked-down config, great! That's certainly useful to a certain segment of the market.

But others might want a bit more flexibility, with a well-defined starting point. Yet another useful option.

And for those that really want to roll their own, a list of what is supported, augmented by tools to help you design and size a config that's right for you.

There's a vast list of reasons why more hardware choice is a good thing ...

Maybe there’s an existing investment that’s already been made.

Maybe there are requirements that aren’t satisfied well by the static configs available.

Maybe you've got a great sourcing arrangement for servers.

Maybe there’s a desire to use the latest-greatest technology, without waiting for an appliance vendor to offer it. Etc. etc.

Whatever the reason, increased hardware choice makes hyperconverged more compelling and more attractive for more people.

EVO:RAIL currently has 9 qualified EVO:RAIL partners. Virtual SAN (as part of vSphere) has dozens and dozens of ReadyNodes from server partners that can be ordered as a single SKU. And for everyone else, there's an extended HCL that allows for literally millions of potential configurations - plus the tools to figure out what's right for you.

If market leadership includes any notion of hardware choice, VMware stands apart from the rest of the hyperconverged crowd. Because, after all, it's software ...

#4 — There’s More To Enterprise IT Than Just Hyperconverged

Yes, there’s that old joke that when all you have is a hammer, everything looks like a nail :)

But there’s a more serious consideration here: when it comes to even modestly-sized IT functions, hyperconverged is only one part of a broader landscape that needs to be managed.

There’s inevitably a strong desire for common tools, processes and workflows that support the entire environment, and not just an isolated portion of it.

From an enterprise IT perspective, it's highly desirable to use the same operational framework for both virtualized and hyperconverged environments.

Going back to that controversial “market leadership” thing, how about the need for enterprise-scale management tools that aren’t limited to a single hyperconverged offering?

#5 — Customer Support Matters

If extreme simplicity is an integral part of the hyperconverged value proposition, customer support has to figure in prominently.

But there’s a structural problem here.

Not all of the hyperconverged appliance vendors have elected to be vSphere OEMs. That means that they don’t have the right to distribute the VMware software used in their product. It also means that they are not entitled to provide support for VMware software.

This arrangement has the potential to put their customers in an awkward position.

While I’m very sure all us vendors use our collective best efforts to support mutual customers, this state of affairs certainly isn’t ideal. Since all of these vendors provide a critical storage software layer, it may not be obvious where a problem actually lies.

Let’s say you have a performance problem with your hyperconverged appliance — who do you call?

The appliance vendor? VMware? Ghostbusters?

When it comes to providing customer support, VMware is typically ranked at the top (or near the top) in customer satisfaction — even though there are always potential areas for improvement. One call.

No argument: the customer support model and execution should factor into our notion of “market leadership”.

#6 — Useful Things Should Just Work

Most shops have gotten accustomed to using all the cool functionality in vSphere. And, presumably, they’d like to continue doing the same in their hyperconverged environment.

But that’s not always the case. Here’s one example ...

You’re probably familiar with vSphere HA — a great feature that automatically restarts workloads on a surviving host if there’s a failure.

In a shared storage environment, vSphere HA uses the management network to ascertain the state of the cluster, and coordinate restarts if necessary. HA assumes that external storage is always available, and all hosts can see essentially the same thing.

But what if there’s no external storage, and we’re using a distributed cluster storage layer?

While it’s true that many of the newer hyperconverged appliances set up their own logical network (primarily for storage traffic), you can see the potential problem: vSphere HA doesn’t know about the vendor's storage network, and vice versa.

Imagine if, for example, the storage network partitions and the management network doesn’t. Or if they partition differently. Sure, that’s not going to happen every day, but when it does — what exactly happens?

In the case of vSphere and VSAN, vSphere HA has been redesigned to use VSAN’s network, so there is zero chance of an inconsistent state between the two.

Let’s go for two examples, shall we?

Using vMotion to balance clusters is just about ubiquitous. You’d like to be able to move things around without screwing up application performance due to slow storage.

Well, one vendor’s attempt at “data locality” didn’t help so much. Move a VM, and performance degrades due to a design decision they made. Try and move it back, more degradation.

So another cool and useful vSphere feature now has sharp edges on it.

Not to pile on, but let's consider maintenance mode.

VMware admins routinely want to put a host in maintenance mode to work on it. All the workloads are conveniently moved to other servers, and nothing gets disrupted. But in our hyperconverged world, there's now storage to be considered.

VSAN has an elegant solution as part of the standard vSphere maintenance mode workflow -- the administrator gets a choice as to what they'd like to do with the affected data, and proceeds.

All other approaches require a separate workflow to detect and evacuate potentially affected data -- which creates not only a bit more complexity, but also that special opportunity to have a really bad day.

I’m sharing these annoying nits just to illustrate a point: a good hyperconverged environment should reasonably support the same everyday virtualization functionality and workflows you already use.

And, hopefully, with a minimum of “gotcha!”

Let’s factor that into our notion of “market leadership” as well …

#7 — Don’t Forget About Networking …

If we’re *seriously* discussing hyperconverged software, we have to ultimately consider software-defined networking in addition to compute and storage.

Otherwise, our stool only has two legs :)

The ultimate goal should be to give customers the option of running all three infrastructure functions (compute, storage, network) as an integrated stack running on their hardware of choice.

No, we’re not there yet today, but …

Converge server virtualization with both SDS and SDN, and the potential exists for even more efficiency, simplicity and effectiveness. Not to mention, a whole new set of important security-related use cases, like micro segmentation.

But to integrate SDN, you’ve got to have the technology asset.

Within the VMware world, that key asset is NSX. And while no vendor can offer a seamless integration between the three disciplines today, VMware has a clear leg up in this direction.

Dig deep into VSAN internals, and you can see progress to date. For example, VSAN works closely with NIOC to be a well-behaved citizen over shared 10Gb links. More to come.

Should hyperconverged vendors who claim market leadership have a plan for SDN and security use cases? I think so.

#8 — Is There A Compatible Hybrid Cloud Option?

Not all infrastructure wants to live in a traditional data center. There are many good reasons to want an operationally compatible hybrid cloud option like vCloud Air: cost arbitrage, disaster recover, short-term demands, etc.

Ideally, customers could have access to a hyperconverged experience that works the same -- using the same tools and workflows -- whether the hardware assets are in the data center, in a public cloud, or ideally both using the same management tools, sharing behaviors, etc.

It’d be great if the industry pundits factored this into their definition of “market leadership”. I’m not hopeful, though.

#9 — Is It Efficient?

One of the big arguments in favor of virtualization and hyperconverged approaches is efficiency: doing more with less.

Not to belabor an old argument, but there’s a certain economic appeal to hyperconverged software that uses compute and memory resources judiciously. The big motivator here for customers is better consolidation ratios for server and VDI farms. Better consolidation ratios = less money spent.

A hyperconverged storage solution that demands a monster 32 GB VM and potentially a few dedicated physical cores on each and every server gets in the way of that.

#10 — Where Do You Go From Here?

I remember clearly a meeting with a customer who introduced the purpose of the meeting: “we’re here to decide what to buy for tomorrow’s legacy”. I couldn't stop smiling :)

But it's an interesting perspective: one that reflects that IT strategy is often the result of many tactical decisions made along the way.

At one level, I’ve met IT pros who have an immediate need, and want an immediate solution. They want a handful of boxes racked up ASAP, and aren’t that concerned with what happens down the road.

Trust me, I can fully appreciate that mindset.

But there are many other IT pros who see each and every technology decision as a stepping stone to bigger and better things.

There are over a half-million IT shops who have built their data center strategy around VMware and vSphere. Every one of them already owns many of the key ingredients needed for a hyperconverged solution.

More importantly, they trust VMware to take them forward into the brave new world of IT. : virtualized, converged, hyperconverged, hybrid cloud and ultimately to a software-defined data center.

]]>
]]>
http://chucksblog.emc.com/chucks_blog/2015/03/why-i-think-vsan-is-so-disruptive.htmlWhy I Think VSAN Is So Disruptivetag:typepad.com,2003:post-6a00d83451be8f69e201b8d0e9b8db970c2015-03-11T23:23:26-04:002015-03-11T23:28:58-04:00

Related Stories

Looking for a great disruption story in enterprise IT tech? I think what VSAN is doing to the established storage industry deserves to be a strong candidate.

I've seen disruptions -- small and large -- come and go. If you're into IT infrastructure, this is one worth watching.

A few years ago, I moved from EMC to VMware on the power of that prediction. So far, it’s played out pretty much as I had hoped it would. There’s now clearly a new dynamic in the ~$35B storage industry, and VMware’s Virtual SAN is very emblematic of the changes that are now afoot.

There’s a lot going on here, so it’s worth sharing. In each case, you’ll see a long-held tenent around The Way Things Have Always Been Done clearly up for grabs.

See if you agree?

I began this post by making a list of changes — deep, fundamental changes — that VSAN is starting to bring about in the storage world.

To be clear, I’m not talking so much about specific technologies, or how this vendor stacks up against that other one.

I’m really far more interested in the big-picture changes around fundamental assumptions as to “how storage is done” in IT shops around the globe: how it's acquired, how it's consumed, how it's managed.

If you’re not familiar with Virtual SAN, here’s what you need to know: it’s storage software built into the vSphere hypervisor. It takes the flash and disk drives inside of servers, and turns them into a shared, resilient enterprise-grade storage service that’s fast as heck. Along the way, it takes just about every assumption we've made about enterprise storage in the last 20 years and basically turns it on its head.

Storage Shouldn’t Have To Be About Big Boxes

Most of today’s enterprise storage market is served by external storage arrays, essentially big, purpose-built hardware boxes running specialized software. Very sophisticated, but at a cost.

If your organization needs a non-trivial amount of storage, you usually start by determining your requirements, evaluating vendors, selecting one, designing a specific configuration, putting your order in, taking delivery some time later, installing it and preparing it for use.

Big fun, right?

The fundamental act of simply making capacity ready to consume — from “I need” to “I have” — is usually a long, complex and often difficult process: best measured in months. I think the most challenging part is that IT shops have to figure out what they need well before actual demand shows up. Of course, this approach causes all manner of friction and inefficiency.

We’ve all just gotten used to it — that’s just the way it is, isn’t it? Sort of like endlessly sitting in morning commute traffic. We forget that there might be a better way.

The VSAN model is completely different. Going from “I need” to “I have” can be measured in days — or sometimes less.

For starters, VSAN is software — you simply license the CPUs where you want to use it. Or use it in evaluation mode for a while. The licensing model is not capacity-based, which is quite refreshing. That makes it as easy to consume as vSphere itself.

The hardware beneath VSAN is entirely up to you, within reason. Build a VSAN environment from hand-selected components if that’s your desire. Grab a ReadyNode if you’re in a hurry. Or go for something that’s packaged the ultimate in a simplified experience: EVO:RAIL. Choice is good.

Depending on your hardware model, getting more storage capacity is about as simple as ordering some new parts for your servers. Faster, easier, smaller chunks, less drama, etc. No more big boxes.

Yes, there is a short learning curve the first time someone goes about putting together a VSAN hardware configuration (sorry!), but — after that — there’s not much to talk about.

There are some obvious and not-so-obvious consequences from this storage model.

Yes, people can save money (sometimes really big $$$) by going this way. Parts is parts. We’ve seen plenty of head-to-head quotes, and sometimes the differences are substantial.

But there’s more that should be considered …

Consider, for example, that storage technologies are getting faster/better/cheaper all the time.

Let’s say a cool new flash drive comes out — and it looks amazing. Now, compare the time elapsed between getting that drive supported with VSAN, and getting it supported in the storage arrays you currently own.

There's a big difference in time-to-usability for any newer storage tech. And that really matters to some people.

One customer told us he likes the “fungibility” of the VSAN approach, given that clusters seem to be coming and going a lot in his world. He has an inventory of parts, and can quickly build a new cluster w/storage from his stash, tear down a cluster that isn’t being used for more parts, mix and match, etc.

Sort of like LEGOs.

Just try that with a traditional storage array.

More Performance (Or Capacity) Shouldn’t Mean A Bigger Box

A large part of storage performance comes down to the storage controllers inside the array: how many, how fast.

Add more servers that drive more workload, and you’re often looking at the next-bigger box — and all the fun that entails: acquiring the new array, migrating all your applications, figuring out what to do with the old array, etc.

Yuck. But that’s the way it’s always been, right?

VSAN works differently.

As you add servers to support more virtualized applications, at the same time you’re also adding the potential for more storage performance and capacity. A maxed-out 64 node VSAN cluster can deliver ~7m cached 4K read IOPS.

Want more performance without adding more servers? Just add another disk controller and disk group to your existing servers, or perhaps just bigger flash devices, and you’ll get one heck of a performance bump.

Without having to call your storage vendor :)

Storage Shouldn’t Need To Be Done By Storage Professionals

I suppose an argument could be made about it being best to have your taxes done by tax professionals, but an awful lot of people seem to do just fine by using TurboTax software.

There certainly are parts of the storage landscape that are difficult and arcane — and that’s where you need storage professionals. There are also an awful lot of places where a simple, easy-to-use solution will suffice quite nicely, and that’s what VSAN brings to the table.

With VSAN, storage just becomes part of what a vSphere administrator does day-to-day. No special skills required. Need a VM? Here you go: compute, network and storage. Policies drive provisioning. Nothing could really be simpler.

No real need to interact with a storage team — unless there’s something special going on.

Can't We All Just Work Together?

Any time you get a team greater than a handful of people, people split up into different roles. The classic pattern in enterprise IT infrastructure has a dedicated server team, a dedicated network team, a storage team, etc.

The vSphere admins are usually dependent on the others to do basic things like provision, troubleshoot, etc. For some reason, I’ve observed particular friction between the virtualization team and the storage team. As in people on both sides pulling their hair out.

Many virtualization environments move quickly: spinning up new apps and workloads, reconfiguring things based on new requirements — every day (or every hour!) brings something new.

That’s what virtualization is supposed to do — makes things far more flexible and liquid.

When that world bumps up against a traditional storage shop that thinks in terms of long planning horizons and careful change management — well, worlds collide.

With VSAN, vSphere admins can be self-sufficient for most of their day-to-day requirements. No storage expertise required. Of course, there will always be applications that can justify an external array, and the team that manages it.

It’s just that there will be less of that.

Storage Software Is Now Not Just Another Application

The idea of doing storage in software is not new. The idea of building a rich storage subsystem into a hypervisor is new. And, when you go looking, there are plenty of software storage products that run as an application, also known as a VSA or virtual storage appliance.

In this VSA world, your precious storage subsystem is now just another application. It competes for memory and CPU like all other applications, but with one exception: when it gets slow, everything that uses it also gets slow.

Because it’s built into the hypervisor, its resource requirements are quite reasonable. It doesn’t have to compete with other applications, because it isn’t a standalone application like a VSA is. Your servers can be smaller, your virtualization consolidation ratios better — or both.

Why do I think this will change things going forward?

Because VSAN now establishes the baseline for what you should expect to get with your hypervisor. Any vendor selling a VSA storage product as an add-on has to make a clear case as to why their storage thingie is better than what already comes built into vSphere.

Not only in justifying the extra price, but also the extra resources as well as the extra management complexity. Clearly, there are cases where this can be done, but there aren’t as many as before.

And that’s going to put a lot of pressure on the vendors who use a VSA-based approach.

The Vendor Pecking Order Changes

The last wave of storage hardware vendors were all array manufacturers — they got all the attention. In this wave, the storage component vendors are finding some new love.

As a good example, the flash vendors such as SanDisk and Micron are starting to do a great job marketing directly to VSAN customers. Why? A decent proportion of a VSAN config goes into flash, and how these devices perform affects the entire proposition.

This new-found stardom is not lost on them — especially as we start with all-flash configurations.

At one time, there was a dogfight between FC HBA vendors who wanted to attach to all the SANs that were being built. In this world, it’s the storage IO controller vendor. Avago (formerly LSI) as well as some of their newer competitors are aware that there’s a new market forming here, and realizing they can reach end users directly vs. being buried in an OEM server configuration.

There’s A Lot Going On In Storage Right Now …

We’ve seen one shift already from disk to flash — that much is clear. Interesting, but — at the end of the day, all we were really doing was replacing one kind of storage media with another.

What I’m seeing now has the potential to be far more structural and significant. Now up for grabs is the fundamental model of "how storage is done" in IT shops large and small.

An attractive alternative to the familiar big box arrays of yesterday.

Storage being specified, acquired, consumed, delivered and managed by the virtualization team, with far less dependence on the traditional storage team.

Storage being consumed far more conveniently than before.

Storage software embedded in the hypervisor having strong architectural advantages over other approaches.

Storage being able to pick up all the advances in commodity-oriented server tech far faster than the array vendors.

Component vendors becoming far more important than before.

And probably a few things I forgot as well :)

Yes, I work for VMware. And VSAN is my baby.

But there’s a reason I chose this gig — I thought VMware and VSAN were going to be responsible for a lot of healthy disruptive changes in the storage business. Customers would win as a result.

Related Stories

]]>
http://chucksblog.emc.com/chucks_blog/2015/02/considering-the-next-wave-of-storage-automation.htmlConsidering The Next Wave Of Storage Automationtag:typepad.com,2003:post-6a00d83451be8f69e201b7c753a3e2970b2015-02-24T15:26:27-05:002015-02-24T13:59:42-05:00

Related Stories

From the time enterprise data centers sprang into existence, we’ve had this burning desire to automate the heck out of them.

From early mainframe roots to today’s hybrid cloud, the compulsion never wanes to progressively automate each every aspect of operations.

The motivations have been compelling: use fewer people, faster responses, be more efficient, make outcomes more predictable, and make services resilient.

But the obstacles have also been considerable: both technological and operational.

With the arrival of vSphere 6.0, a nice chunk of new technology has been introduced to help automate perhaps the most difficult part of the data center – storage.

It's worth digging into these new storage automation features: why they are needed, how they work, and why they should be seriously considered.

Background

Automating storage in enterprise data centers is most certainly not a new topic.

Heck, it's been around as least as long as I have, and that's a long time :)

Despite decades of effort by both vendors and enterprise IT users, effective storage automation still is an elusive goal for so many IT teams.

When I'm asked "why is this so darn hard?", here's what I point to:

Storage devices had very limited knowledge of applications: their requirements, and their data boundaries. Arrays had to be explicitly told what to do, when to do it and where it needed to be done.

Cross-vendor standards failed to emerge that facilitated basic communications between the application’s requirements and the storage array’s capabilities.

Storage arrays (and their vendors) present a storage-centric view of their operations, making it difficult for non-storage groups to easily request new services, and ascertain if end-to-end application requirements were being met.

Here's the message: the new storage capabilities available in vSphere 6.0 show strong progress towards addressing each of these long-standing challenges.

To the extent that each aspect infrastructure can be made programmatically aware of individual application requirements, far better automation can be achieved.

However, when it comes to storage, there have been significant architectural challenges in achieving this.

The first challenge is that applications themselves typically don’t provide specific instructions on their individual infrastructure requirements. And asking application developers to take on this responsibility can lead to all sorts of unwanted outcomes.

At a high level, what is needed is a convenient place to specify application policies that can be bound to individual applications, instruct the infrastructure as to what is required, and be conveniently changed when needed.

The argument is simple: the hypervisor is in a uniquely privileged position to play this role. It not only hosts all application logic, but abstracts that application from all of infrastructure: compute, network and storage.

While these policy concepts have been in vSphere for a while, in vSphere 6.0 a new layer of storage policy based management (SPBM) is introduced. This enables administrators to describe specific storage policies, associate them with groups of applications, and change them if needed.

But more is needed here.

Historically, storage containers have not aligned with application boundaries. External storage arrays have historically presented LUNs or file systems - large chunks of storage shared by many applications.

Storage services (capacity, performance, protection, etc.) were specified at the large container level, with no awareness of individual application boundaries.

This mismatch has resulted in both increased operational effort and reduced efficiency.

Application and infrastructure teams need to go continually back and forth with the storage team regarding application requirements. And storage teams are forced to compromise by creating storage service buckets specified in excess of what is actually required by applications. Better to err on the side of safety, right?

No longer. vSphere 6.0 introduces a new storage container – Virtual Volumes, or VVOLs – that precisely aligns application boundaries and the storage containers they use. Storage services can now be specified on a per-application, per-container basis.

We now have two key pieces of the puzzle: the ability to conveniently specify per-application storage policy (as part of overall application requirements), and the ability to create individualized storage containers that can precisely deliver the requested services without affecting other applications.

So far, so good.

Solving The Standards Problem

Periodically, the storage industry attempts to define meaningful, cross-vendor standards that facilitate external control of storage arrays. However, practical success has been difficult to come by.

Every storage product speaks a language of one: not only in the exact set of APIs it supports, but how it assigns meaning to specific requests, and communicates results. Standard definitions what exactly a snap means, for example, are hard to come by.

The net result is that achieving significant automation of multi-vendor storage environments has been extremely difficult for most IT organizations to achieve.

To be clear, the need for heterogeneous storage appears to be increasing, and not decreasing: enterprise data centers continue to be responsible for supporting an ever-widening range of application requirements: from transaction processing to big data to third platform applications. No one storage product can be expected meet every application requirement (despite vendor's best intents) multiple types are frequently needed.

De-facto standards can be driven by products that are themselves de-facto standards in the data center, and here vSphere stands alone with regards to hypervisor adoption. When VMware defines a new standard for interacting with the infrastructure (and customers adopt it), vendors typically respond well.

vSphere 6.0 introduces a new set of storage APIs (VASA 2.0) that facilitate a standard method of application-centric communication with external storage arrays. VMware’s storage partners have embraced this standard enthusiastically, with several implementations available today and more coming.

Considering VASA 2.0 together with SPBM and VVOLs, one can see that many of the technology enabling pieces are now in place for an entirely new storage automation approach. Administrators can now specify application-centric storage policies via SPBM, communicate them to arrays via VASA 2.0, and receive a perfectly aligned storage container – a VVOL. Nice and neat.

Who Should (Ideally) Manage Storage?

It’s one thing to conveniently specify application requirements, it’s another thing to ensure that the requested service levels are being met, and – more importantly – how to fix things quickly when that’s not happening.

Historically, the storage management model has evolved in many IT organizations to be essentially a largely self-contained organizational “black box”. Requests and trouble tickets are submitted with poor visibility to other teams who depend greatly on the storage team’s services.

Although this silo model routinely causes unneeded friction and inefficiency (not to mention frustration all around), it can be particularly painful is in resolving urgent performance problems: is the problem in the application logic, the server, the network – or storage?

The storage management model created by vSphere 6.0 is distinctly different than traditional models: storage teams are still important, but more information (and responsibility) is given to the application and infrastructure teams in controlling their destiny.

Virtual administrators now see “their” abstracted storage resources: what’s available, what it can do, how it’s being used, etc. There should be no need to directly interact with the storage team for most day-to-day provisioning requirements. Policies are defined, VVOLs are consumed, storage services are delivered.

Through vCenter and the vRealize suite, virtual administrators now have enough storage-related information to ascertain the health and efficiency of their entire environments, and have very focused conversations with their storage teams if there’s an observed issue.

Storage teams still have an important role, although somewhat different than in the past. They now must ensure sufficient storage services are available (capacity, performance, protection, etc.), and resolve problems if the services aren’t working as advertised.

However, operational and organizational models can be highly resistant to change. That's the way the world works -- unless there is a forcing function that makes the case compelling to all parties.

And VSAN shows every sign of being a potential change accelerator.

How Virtual SAN Accelerates Change

As part of vSphere 5.5U1, VMware introduced Virtual SAN, or VSAN. Storage services can now be delivered entirely using local server resources -- compute, flash and disk – using native hypervisor capabilities. There is no need for an external storage array when using VSAN – nor a need for a dedicated storage team, for that matter.

VSAN is designed to be installed and managed entirely by virtual administrators independently of interaction with the storage team. These virtualization teams can now quickly configure storage resources, create policies, tie them to applications, monitor the results and speedily resolve potential problems – all without leaving the vSphere world.

As an initial release, VSAN 5.5 had limited data services, and thus limited use cases. VSAN 6.0 is an entirely different proposition: more performance (both using a mix of flash and disk, or using all-flash), new enterprise-class features, and new data services that can significantly encroach on the turf held by traditional storage arrays.

Empowered virtualization teams now have an interesting choice with regards to storage: continue to use external arrays (and the storage team), use self-contained VSAN, or most likely an integrated combination depending on requirements.

Many are starting to introduce VSAN alongside traditional arrays, and have thus seen the power of a converged, application-centric operational model. And it’s very hard to go back to the old way of doing things when the new way is so much better -- and readily at hand.

The rapid initial growth of VSAN shows the potential of putting a bit of pressure on traditional storage organizations to work towards a new operational model, with improved division of responsibilities between application teams, infrastructure teams and storage teams. And they'll need the powerful combination of SPBM, VASA 2.0 and VVOLs to make that happen.

Change Is Good -- Unless It's Happening To You

I have spent many, many years working with enterprise storage teams. They have a difficult, thankless job in most situations. And there is no bad day in IT quite like a bad storage day.

Enterprise IT storage teams have very specific ways of doing things, arguably built on the scar tissue of past experiences and very bad days. You would too, if you were them.

That being said, there is no denying the power of newer, converged operational models and the powerful automation that makes them so compelling. The way work gets done can -- and will -- change.

Enterprise storage teams can view these new automation models as either a threat, or an opportunity.

How much easier is it to order a block, brick, node, etc. of IT infrastructure as a single supportable product, and move on to more important matters?

A lot easier, it seems ...

Reference architectures have been around for ages. I think of them as a blueprints for building a car, and not like buying one. Some assembly required. Useful, yes, but there’s room for more.

VCE got the party started years back with Vblocks: pre-integrated virtualized infrastructure, sold and supported as a single product — with their success to be quickly followed by other vendors who saw the same opportunity.

A group of smaller vendors took the same idea, but did storage in software vs. requiring an external array, dubbing themselves “hyper-converged”: Nutanix, Simplivity and others. They, too, have seen some success.

Last August, VMware got into this market in a big way by introducing EVO:RAIL — an integrated software product that — when combined with a specific hardware reference platform from an OEM partner — delivered an attractive new improvement over the first round of hyper-converged solutions.

While EVO:RAIL had several partners who offered immediate availability, EMC decided to take their time, and do something more than simply package EVO:RAIL with the reference hardware platform.

Today, we get to see what they’ve been working on — VSPEX BLUE. It’s not just another EVO:RAIL variant, it’s something more.

And, from where I sit, it’s certainly been worth the wait …

The Lure Of Convergence

Think of IT as a black box: money and people go in one side, IT services come out the other side. To the people who actually pay for IT organizations, this is not an entirely inaccurate representation.

If you’re responsible for that black box, you continually ask yourself the question: what should I have my people be spending time on? That’s the expensive bit, after all.

While there are many technical staffers out there who thoroughly enjoy hand-crafting IT infrastructure with the “best” components, it’s hard to point to situations where that activity actually creates meaningful value. Worse, those hand-crafted environments create a long integration and support shadow, lasting many years after the decision has been made.

More and more IT leaders are coming around to the perspective that they want their teams to be focused on using IT infrastructure to deliver IT services, and not excessively bound up in design, integration, support and maintenance.

If it doesn't add value, don't do it.

This is the the fundamental appeal of all forms of converged infrastructure: spend less time and effort on the things that don’t matter much, so your team can spend more time on the things that do matter. The approach is now broadly understood, and the market continues to grow and expand.

Of course, IT infrastructure isn’t one-size-fits-all. Any debate around what’s “best” really should be “what’s best for you?”.

EVO:RAIL

I think of EVO:RAIL as a fully-integrated vSphere stack with value-added management tools, packaged for use as an appliance. The idea is to make the user experience as simple and predictable as possible.

Software is only as good as its hardware, so the EVO:RAIL program provides a narrow specification for hardware partners: 4-node bricks, prescribed processors, memory and controllers, disk bays, etc. Partner value-add and differentiation comes in the form of additional software and services above and beyond a stock EVO:RAIL configuration.

Strategically, I think of EVO:RAIL as well within the “software defined” category: compute, network and storage. In particular, EVO:RAIL is built on VSAN — the product I’ve been deeply involved with recently.

VSPEX BLUE

The hardware is an EMC-specific design, and not a rebadge. If you've been following EVO:RAIL, the design and specs will look very familiar.

VSPEX BLUE will be sold through distributors, who set street pricing. For those of you who are interested in pricing, you'll have to contact a reselling partner.

When evaluating any new EVO:RAIL-based offering, the key question becomes — what’s unique? After all, the base hardware/software is specified to be near-identical across different partner offerings.

In EMC’s case, there is quite a list of differentiated features to consider, so let’s get started, shall we?

VSPEX BLUE Manager

While EVO:RAIL itself has a great user-centric manager to simplify things, EMC has taken it one step further by making a significant investment in their own VSPEX BLUE manager that complements and extends the native EVO:RAIL one in important ways.

First, the VSPEX BLUE Manager provides a portal to any and all support services: automatic and non-disruptive software upgrades, online chat with EMC support, knowledge base, documentation and community.

All in one place.

Second, the VSPEX BLUE Manager is “hardware aware”. It can display the physical view of your VSPEX BLUE appliances, and indicate — for example — which specific component might need to be replaced.

It also directly monitors low-level hardware status information, and feeds it upwards: to vCenter and LogInsight, as well as to EMC support if needed.

More importantly, this feature is tied into EMC’s legendary proactive “phone home” support (ESRS) which means you might be notified of a problem directly by EMC, even though you haven’t checked the console in a while :)

The VSPEX BLUE Manager also manages the software inventory. It discovers available updates, and non-disruptively installs them if you direct it to. More intriguing, there’s the beginnings of a “software marketplace” with additional software options from EMC, with presumably more coming from other vendors as well.

Bundled Software

Part of any potential EVO:RAIL value-add is additional software, and here EMC has been very aggressive. Three fully functional and supported software packages are included with every VSPEX BLUE appliance, but with a few restrictions.

First up is EMC’s RecoverPoint for VMs. Those of you who have followed me for a while know that I’m a huge fan of RecoverPoint: great technology and functionality, very robust and now runs nicely in vSphere (and on top of VSAN) protecting on a per-VM basis.

The limitation is 15 protected VMs per four-node appliance. More appliances, more protected VMs. Since full functionality is provided, your choice of protection targets is wide open: another VSPEX BLUE appliance, or something else entirely.

Next up is the CloudArray cloud storage extender, based on EMC’s recent TwinStrata acquisitions. CloudArray can present file shares or iSCSI to applications requiring extra capacity, or potentially a file share between multiple applications — something VSAN doesn’t do today.

The back-end store can be any compatible object storage: your choice of cloud, an on-prem object store, etc. The included license is for 10TB of external capacity per appliance, not including the actual physical capacity.

And finally, VMware’s VDP backup software (based on DataDomain/Avamar technology) is included. The upgrade is to the full-boat VDP-A. Stay tuned, though.

What I Like

For starters, EMC’s offering is entirely based on VSAN for storage. There is no packaging with an external array, as NetApp did. Since my world is very VSAN-centric these days, that’s a huge statement, coming from the industry’s largest and most successful storage vendor.

VSPEX BLUE Manager is a great piece of work, and adds significant value. The fact that EMC supports the entire environment online (just like their arrays and other products) is a big differentiator in the real world. The software bundle is attractive as well; demonstrating EMC’s commitment to the product and making it stand out in the market.

And then there’s the fact that EMC is obviously jumping into the hyper-converged marketplace with both feet. Thousands of trained field people and a hugely influential partner network. Global distribution and support. An army of skilled professional services experts. A very proficient marketing machine. A large and successful VSPEX channel. The proven ability to move markets.

Those that just focus on the product itself will miss the bigger context here.

What’s Next For The Hyper-Converged Market?

If we’re being honest, the smaller startups had the nascent hyper-converged market to themselves in the early days.

Good for them.

But now the big boys are starting to jump in with vigor: first VMware with EVO:RAIL, and now EMC itself with VSPEX BLUE.

I have been completely immersed in VSAN for the last six months. It's been great. And now I get to publicly share what’s new and — more importantly — what it means for IT organizations and the broader industry.

If there was a prize for the most-controversial storage product of 2014, VSAN would win. In addition to garnering multiple industry awards, it’s significantly changed the industry's storage discussion in so many ways.

Before VSAN, shared storage usually meant an external storage array. Now there’s an attractive alternative — using commodity components in your servers, with software built into the world’s most popular hypervisor.

While the inevitable “which is better?” debate will continue for many years, one thing is clear: VSAN is now mainstream.

This post is a summary of the bigger topics: key concepts, what’s new in 6.0, and a recap of how customers and industry perspective has changed. Over time, I’ll unpack each one in more depth — as there is a *lot* to cover here.

The Big Idea

Virtual SAN extends the vSphere hypervisor to create a cluster-wide shared storage service using flash and disk components resident in the server.

It does not run “on top of” vSphere, it’s an integral component. All sorts of design advantages result.

In addition to a new hardware and software model, VSAN introduced a new management model: policy-based storage management. Administrators specify the per-VM storage policy they’d like (availability, performance, thin provisioning, etc.), and VSAN figures out the rest. Change the policy, VSAN adapts if the resources are there.

The marketing literature describes it as “radically simple”, and I’d have to agree.

During 2014, VSAN 5.5 exceeded each and every internal goal we set out: performance, availability, reliability, customer adoption, roadmap, etc. A big congratulations to a stellar team!

Of course, now we need to set much more aggressive goals :)

What’s New

Quite a bit, really. The engineering team is accelerating their roadmap, and I couldn’t be more pleased. All of this should be available when vSphere 6.0 goes GA.

All-flash

VSAN 6.0 now supports all-flash configurations using a two-tiered approach. The cache tier must a be write-endurant (e.g. not cheap) flash device; capacity can be more cost-effective (and less write-endurant) flash.

With all-flash configurations, cache is not there to accelerate performance; it’s there to minimize write activity to the capacity layer, extending its life.

Performance is extreme, and utterly predictable as you’d expect. Sorry, no dedupe or compression quite yet. However -- and this is a big however -- the write-caching scheme permits the use of very cost-effective capacity flash without burning it out prematurely. Everyone's numbers are different, but it's a close call as to which approach is more cost-effective: (a) more expensive capacity flash with dedupe/compression, or (b) more inexpensive capacity flash without dedupe/compression.

Please note that all-flash support is a separate license for VSAN.

New file system, new snaps

Using the native snapshots in vSphere 5.5 was, well, challenging. A new on-disk filesystem format is introduced in VSAN 6.0 that’s faster and more efficient, derived from the Virsto acquisition. Snaps are now much faster, more space-efficient and perform better when used -- and you can take many more of them. Up to 32 snaps per VM are now supported, if you have the resources.

VSAN 5.5 users can upgrade to 6.0 bits without migrating the file format, but won’t get access to some of the new features until they do. The disk format upgrade is rolling and non-disruptive, one disk group at a time. Rollbacks and partial migrations are supported. Inconvenient, but unavoidable.

Bigger and better — faster, too!

Support is included for VMDKs up to 62TB, same as the vSphere 6.0 max. VSAN clusters can have as many as 64 nodes, same as vSphere 6.0.

The maximum number of VMs on a VSAN node is now 200 for both hybrid and all-flash configs (twice that of VSAN 5.5), with a new maximum of 6400 VMs per VSAN cluster.

More nodes means more potential capacity, up to 64 nodes, 5 disk groups per node and 7 devices per disk group, or 2240 capacity devices per cluster. Using 4TB drives, that’s a humble ~9 petabytes raw or so.

The marketing statement is that VSAN 6.0 in a hybrid configuration (using flash as cache and magnetic disk for capacity) offers twice the performance than VSAN 5.5. I want to come back later and unpack that assertion in more detail, but VSAN 6.0 is noticeably faster and more efficient for most workloads. Keep in mind that VSAN 5.5 was surprisingly fast as well.

And, of course, the all-flash configurations are just stupidly fast.

Go nuts, folks.

Fault domains now supported

VSAN 5.5 was not rack-aware, VSAN 6.0 is. When configuring, you define a minimum of 3 fault domains to represent which server is in which rack. After this step, VSAN will be smart enough to distribute redundancy components across racks (rather than within racks) if you tell it to.

Note: this is not stretched clusters — yet.

New VSAN Health Services

Not surprisingly, VSAN is dependent on having the correct components (hardware, driver, firmware) as well as a properly functioning network. The majority of our support requests to date have been related to these two external issues.

VSAN 6.0 now includes support for a brand new Health Services tool that can be used to diagnose most (but not all) of the external environmental issues we've encountered to date. Log collection is also simplified in the event you need VMware customer service support.

A must for any serious VSAN user.

Operational improvements

Lots and lots of little things now make day-to-day life with VSAN easier.

The UI now does a better job of showing you the complete capacity picture at one go. There’s a nifty screen that helps you map logical devices to physical servers. You can blink drive LEDs to find the one you’re interested in. Individual capacity drives or disk groups can be evacuated for maintenance or reconfiguration. There’s a new proactive rebalancing feature.

A default storage policy is now standard, which of course can be edited to your preferences. There’s a new screen that shows resynchronization progress of rebuilding objects. There’s a new “what if” feature that helps you decide the impact of a new storage policy ahead of time.

For larger environments, vRealize Automation and vRealize Operations integration has been substantially improved — VSAN is now pre-configured to raise selected status alarms (VOBs) to these tools if present.

And much more.

Behind the scenes

There’s been a whole raft of behind-the-scenes improvements that aren’t directly visible, but are still important.

VSAN is now even more resource-efficient (memory, CPU, disk overhead) than before, allowing higher consolidation ratios, among other things. VSAN’s resource efficiency is a big factor for anyone looking at software-delivered storage solutions as part of their virtualized environment, helping folks achieve even higher consolidation ratios.

The rebuild prioritization could be a bit aggressive in VSAN 5.5; it now plays much more nicely with other performance-intensive applications. Per-node component counts have been bumped up from 3000 to 9000, and there’s a new quorum voting algorithm that uses far fewer witnesses than before. As a result, there’s much less need to keep an eye on component usage.

VSAN 6.0 requires a minimum of vCenter 6.0 and ESXi 6.0 on all hosts. As mentioned before, you can defer the file system format conversion until later, but no mix and matching other than that.

New ReadyNodes

If you’d like to shortcut the build-your-own experience, that’s what VSAN ReadyNodes are for. Many new ones will be announced shortly, some with support for hardware checksums and/or encryption.

ReadyNodes can either be ordered directly from the vendor using a single SKU, or use them as a handy reference in creating your own configurations.

Or skip all the fun, and go directly to EVO:RAIL

See something missing?

Inevitably, people will find a favorite feature or two that's not in this release. I have my own list. But don't be discouraged ...

I can't disclose futures, but what I can point to is the pace of the roadmap. VSAN 5.5 has been out for less than a year, and now we have a significant functionality upgrade in 6.0. Just goes to show how quickly VMware can get new VSAN features into customers' hands.

The Customer Experience

By the end of 2014, there were well over 1000 paying VSAN 5.5 customers. Wow. Better yet, they were a broad cross-sample of the IT universe: different sizes, different industries, different use cases and different geographies.

For the most part, we succeeded in exceeding their expectations: radically simple, cost-effective, blazing performance, reliable and resilient, etc.

One area we took some heat on with VSAN 5.5 was being a bit too conservative on the proposed initial use cases: test and dev, VDI, etc.

Customers wondered why we were holding back, since they were having such great experiences in their environment.

OK, call us cautious :)

With VSAN 6.0, there are almost no caveats: bring on your favorite business-critical workloads with no reservations.

Another area where we’re working to improve? In the VSAN model, the customer is responsible for sizing their environment appropriately, and sourcing components/drivers/firmware that are listed on the VMware Compatibility Guide (VCG) for VSAN.

Yes, we had a few people who didn’t follow the guidelines and had a bad experience as a result. But a small number of folks did their best, and still had some unsupported component inadvertently slip into their configuration. Not good. Ideally, we’d be automatically checking for that sort of thing, but that’s not there — yet.

So the admonishments to religiously follow the VSAN VCG continue.

If I’m being critical, we probably didn’t do a great job explaining some of the characteristics of three-node configurations, which are surprisingly popular. In a nutshell, a three node config can protect against one node failure, but not two. If one node fails, there are insufficient resources to re-protect (two copies of data plus a witness on the third node) until the problem is resolved. This also means you are unprotected from a failure during maintenance mode when only two nodes are available.

Storage — and storage technology — has been around for a long time, so there’s an established orthodoxy that some people adhere to. VSAN doesn’t necessarily follow that orthodoxy, which is why it’s disruptive — and controversial.

There was a lot of head-scratching and skepticism when VSAN was introduced, but I think by now most of the industry types have gotten their head wrapped around the concept.

Yes, it’s highly available. Yes, it offers great performance. No, the world won’t end because people are now using server components and hypervisor software to replace the familiar external storage array. And there is plenty of real-world evidence that it works as advertised.

However, a few red herrings did pop up during 2014 that are worth mentioning.

One thread was around why people couldn’t use any hardware that might be handy to build a VSAN cluster. The rationale is the same as why array vendors won’t let you put anything unsupported in their storage arrays — the stuff might not work as expected, or — in some cases — is already known not to work properly.

If you don’t follow the VCG (vSphere Compatibility Guide), we’re awfully limited in the help we can provide. And there are some truly shoddy components out there that people have tried to use unsuccessfully.

Another thread was from a competitor around the attractiveness of data locality.

The assertion was that it made performance sense to keep data and application together on the same node, with absolutely no evidence to support the claim.

Keep in mind that, even with this scheme, writes still have to go to a separate node, and any DRS or vMotion will need its data to follow. And that’s before you consider the data management headaches that could result by trying to hand place the right data on the right node all the time.

Hogwash, in my book.

VSAN creates a cluster resource that is uniformly accessible by all VMs. DRS and/or vMotions don’t affect application performance one bit. Thankfully, the competitor dropped that particular red herring and went on to other things.

A related thread was the potential attractiveness of client-side caching vs. VSAN’s server-side caching. A 10Gb network is plenty fast, and by caching only one copy of read data cluster-wide, it’s far more space efficient and thus there’s a much greater likelihood that a read request will come from cache vs. disk. Our internal benchmarks continually validate this design decision.

A more recent thread from the networking crowd wasn’t happy with the fact that VSAN uses multicast for cluster coordination, e.g. for all nodes to stay informed on current state. It’s not used to move data.

Do we have cases where customers haven’t set up multicast correctly? Yes, for sure. Once it gets set up correctly, does it usually run without a problem? Yes, for sure.

There was also the predictable warning that VSAN 5.5. was a “1.0” product, which was essentially a true statement. That being said, I’ve been responsible for bringing dozens of storage products to market over the decades, and — from my perspective — there was very little that was “1.0” about it.

And I’ve got the evidence to back it up.

Perhaps the most perplexing thread was the specter of the dreaded “lock in” resulting from VSAN usage. To be fair, most external storage arrays support all sorts of hypervisors and physical hosts, and Virtual SAN software is for vSphere, pure and simple.

Enterprise IT buyers are quite familiar with products that work with some things, but not others. This is not a new discussion, folks. And it seems like the vSphere-only restriction is AOK for many, many people.

The Big Picture?

Using software and commodity server components to deliver shared storage services is nothing new in the industry. We’ve been talking about this in various forms for many, many years.

But if I look back, this was never an idea that caught on — it was something you saw people experimenting with here and there, with most of the storage business going to the established storage array vendors.

With VSAN’s success, I’d offer this has started to change.

During 2014, VSAN has proven itself to be an attractive, mainstream concept that more and more IT shops are embracing with great success.

Related Stories

Like so many others, I found the industry move to cloud fascinating on so many levels: new technology models, new operational models, new application models, new consumption models, etc.

I wrote endless, lengthy blog posts attempting to explore every nook and cranny. Even to this day, the topic continues to intrigue me.

One of the things I spent much time considering was what I dubbed the "cloud supply chain".

As an example, supply chains in the physical world are responsible for transforming raw materials into finished goods we all consume. Every company along the way specializes at what they do best at, and cooperates with others who are good at other things. It's rare when you see a single company responsible for everything from raw materials to customer service.

Cloud services should be no different, I thought.

Specialized players -- each with different strengths -- could and should combine into supply chains to create more value than any single player alone.

Today, VMware's vCloud Air service announced a strategic partnership with Google's Cloud Platform Services. Customers of vCloud Air can now use select Google Cloud Platform services relatively transparently: with a single contract, and a single point of support.

It's a great example of a cloud supply chain -- but will we see more?

And the bigger question, still not answered: what will be the dominant industry model that serves enterprise IT?

The News

You can go read all about it here, here and here -- but if you're pressed for time: vCloud Air customers can now use selected Google Cloud Platform services directly from vCloud Air, using their existing contract and support agreements.

As VMware and Google typically sell to different customers, both parties clearly benefit. VMware gets the benefit of Google's services for vCloud Air, and Google gets a convenient route to market to enterprise IT. Customers get the convenience of a single provider selling and supporting an expanded set of services.

Why This Is Interesting To Me

In one sense, I see this as a battle in strategic alternatives.

Amazon's basic message is "we can do it all for you". There is no notion of a cloud services supply chain here: Amazon produces, sells and delivers all of the cloud services in its portfolio. And, historically, this has done well for them.

On the other hand, cloud service providers such as VMware and Google whose message is "here is what we do well, here is what our partners do well, we'll make it easy for you use it all together".

Without getting into a debate as to which approach is "better", an argument can be made for the most likely long-term successful model: a cloud services supply chain. Why? As in the physical world, specialization and focus matters. It's very hard for any company -- regardless of size -- to service every potential customer with everything they might need -- and to be really good at every aspect.

Why should cloud services be any different?

Where You Are In The Supply Chain Matters

If you look at supply chains in the physical world, you'll quickly realize the most influential position is being closest to the ultimate consumer or customer.

When it comes to delivering enterprise IT services, being close to the customer really matters. So much of enterprise IT is "high touch" -- skilled people on both sides of the table working towards a common goal.

In the case of VMware, there's more than that. VMware products define a data center computing experience: the technology as well as the operational model. The design goal for vCloud Air is to replicate and improve that familiar experience using a cloud services model. Put differently, VMware is not only close to the customer, they're close to the operational model that much of IT is using today.

And, thus, I would argue occupying a fortunate spot in the cloud services supply chain market that is starting to emerge.

What We Might See In The Future

I recently succumbed to upgrading my iPhone from a battle-scarred 4S to a new iPhone 6. The migration? Completely painless. The user experience? Utterly familiar but with a few new cool features.

Just what I wanted -- exactly what I was familiar with, but better.

I never really considered an Android-based phone, or any other alternative. Too much hassle, and I already liked what I had. Based on Apple's recent quarterly results, it seems I was not alone in my thinking.

I can make a strong argument that as enterprise IT groups start to adopt more cloud services, many will be looking for the same thing: what they're familiar with, only better.

And an awful lot of IT professionals are familiar with VMware products and the experience they deliver.

I used to regularly do my list of New Year predictions. My success rate has been reasonable, but this year is different.

Why? Because this year, I believe there is one vastly important trend that will begin to drive more change across the IT landscape than all the other possible candidates combined.

And that driving force is the 3rd platform — and the new breed of applications it supports.

We’ve all been talking about it for a few years. It’s not a contentious discussion, although it's been rather abstract for many. But, in 2015, it shows every sign of getting very real for many more IT groups.

The required ingredients are now in place. The spark is beginning to ignite the mixture. And the changes should come very quickly as a result.

Like a meteor hitting the earth — the IT world is going to look very different before too long.

IT Reality 101

Those of us buried in the trenches can lose sight of a fundamental truth: IT exists solely for the purpose of delivering the applications people want to use.

When the desired application model changes, the IT world is forced to change around it.

Quite often, you read a rant from an infrastructure person (network, storage, security, etc.) about how darn inconvenient it can be when applications demand certain things outside the established historical norm.

My advice? Now would be a good time to fasten your seat belts …

The Last Wave

Once you’ve seen an industry tsunami, you have an idea of what to look for. I had the privilege of seeing one up close and personal.

I entered the workforce when mainframes and minicomputers were king. My newly-minted UNIX and C skills were not in demand yet -- but COBOL, JCL and MVS were certainly marketable.

Most applications were monolithic — green screens anyone? Networks existed primarily to connect terminals to servers. Storage was all direct-attached.

The first ingredient behind the demise of the first platform was a new form of low-cost computing — UNIX running on microprocessors. The combination delivered an entirely different cost structure than what came before. And with it, a new management and operations style that was more agile and responsive than the previous mainframe world.

All you needed was the root password :)

At roughly the same time, the preferred user experience was was starting to change as well — PCs were quickly finding their way onto most knowledge worker’s desktops.

These two ingredients set the stage for a dramatic shift in the desired application model — from traditional monolithic to a more progressive client-server approach.

Once client-server applications became the preferred approach, the resulting industry changes started to come fast and furiously.

A entirely new IT ecosystem emerged, with new players. People needed local area networks and routers, and Cisco ended up doing quite well. Mainframe databases gave way to UNIX-based (and later NT-based) products, with Oracle coming out on top. Microsoft owned the user experience, and leveraged that to establish a valuable position in server operating systems, quickly eliminating NetWare’s reign.

And much more.

Looking back, it can be hard to appreciate just how dramatically things changed during this period. Long gone are the most of the mainframe and minicomputer titans of yore. The application vendors who thrived in the monolithic world are largely no longer with us. While some of the legacy skills are still in demand, the market continues to dwindle.

When a new application paradigm hits, the structural changes come quickly.

And now a new one is upon us.

The New Wave

At a high level, the two required ingredients we saw previously are clearly evident, although slightly different.

A new form of efficient and cost-effective computing is becoming popular. Combine commodity components with a cloud operational model, and you’re playing at an entirely new level of efficiency and flexibility than before.

Whether you decide to label it public cloud, private cloud, hybrid cloud, converged, hyper-converged, etc. is up to you — the central thought is there is a new model associated with infrastructure and operations, and it’s very different than the old one.

Needless to say, the preferred user experience has now permanently changed to mobile for many use cases. “It doesn’t work well on my iPhone” can be a death knell for so many applications these days.

Yes, we’ll still have desktops and web browsers and all that, but that’s not the dominant investment model any more. Slick mobile experiences can trump all alternatives, so — in the long term — this is what people will generally prefer.

Digging deeper, there's a fascinating world of data fabrics, analytical tools, app frameworks, etc. that gather, analyze and help people act on data in entirely new ways.

It is important to note that the majority of 3rd platform applications are revenue-generating, strategic — or both. So they typically come with a lot of firepower, and can thus drive a lot of change very quickly.

The Spark That Ignites?

An air-gasoline mixture can be quite stable — until a spark is present. The ingredients for the 3rd platform have been around for a while: cloud, mobile, big data, tools, frameworks, etc. — so it is fair to ask: what will ignite it? And why now?

For me, the answer is simple yet subtle: a fundamental change in business strategy.

Most company leadership teams are now frantically re-imagining themselves as digital entities vs. based in the physical world. Every new business process, offering, service, etc. is now thought of digitally.

And this new digital business model demands a new application factory to create an endless stream of new, modern software widgets.

It’s important to note: this is not an IT thing, this is a business thing. Actually, a digital business thing.

If I look at 2014, the pace of customers I met who were clearly shifting to a digital business model picked up rapidly. As 2014 came to a close, it became commonplace. I would only expect this to accelerate through 2015. Once a critical mass is achieved, the tipping point is passed, and the ball starts to roll downhill very quickly.

The Prediction

So I’m willing to go out on a limb and flatly state that — in 2015 — the 3rd platform starts to clearly move from powerpoint to production — driven by a new generation of applications that are needed to power modern digital business models.

Few will be left untouched: IT vendors, IT professionals — just about everyone in our cozy world. And by the end of 2015, it should be clear to most everyone involved that “that was then, this is now”.

When critical applications are king, all others must obey.

We in the IT infrastructure world sometimes have difficulty internalizing this harsh reality, especially if it’s inconvenient to our traditional ways of doing things. For example, I remember clearly the mainframe guys thinking UNIX was absolute heresy, not robust enough, inefficient, etc.

To be clear, the legacy world doesn’t disappear overnight. But all the serious attention will turn to the new 3rd platform and its new requirements. The 2nd platform soldiers on, (as does the 1st platform) but it won’t be the investment center any more.

Why Software Defined Really Matters Now

Much as a digital new business model drives the need for the 3rd platform, those 3rd platform applications will drive the need for software-defined infrastructure and services in all of its flavors.

One of the central premises of software-defined anything (networks, storage, security, etc.) is that — by abstracting functionality from hardware — it can place all aspects of infrastructure services under consistent programmatic control if desired.

Services can be dynamically composed without manual human involvement.

Based on observations to date, 3rd platform applications are sprawling, dynamic beasts with multiple morphing components and associated hard-to-predict behavior. Things move fast, and change all the time.

Whether supporting infrastructure services are directly under application control, or controlled by a next-generation management framework (e.g. devops), there’s no room for the previous technology management model. Every aspect must be under software control, and automated as much as possible.

Just like mainframe management techniques didn’t work for the new world of UNIX-based client-server, I’m arguing that our familiar silos of manual-intensive service delivery and management just won’t work for the new world of 3rd platform applications. They are too slow, too inflexible and too expensive.

As a result, the growing wave of (revenue-generating and strategic) 3rd platform applications will be forcing functions to rapidly adopt software-defined management techniques, and quickly.

Consider modern aviation: highly automated, with humans providing exception management as needed. Every aspect of a modern airliner is entirely under software control. Yes, the model is not without its occasional shortcomings, but there's no going back.

How This Typically Plays Out

I’ve been in meetings that have gone something like this …

On one side of the table is the application lead with their business partner. They’re on a mission to stand up an Important New Thing. Maybe it’s a big data thing, or a new mobile application, or perhaps something similar.

They are using new tools and new agile techniques to iterate fast and learn as they go. They have adequate funding and executive support. You don’t want to say “no” to these people.

On the other side of the table is the infrastructure team(s): server, network, storage, security. They have their established ways of doing things, aimed at predictable delivery of services for all applications, and not just the new thing.

They are justifiably concerned that there are no well-defined requirements, things can change at a moment’s notice, most of their questions can’t be answered, what they want to do isn’t compatible with existing processes, etc.

Anxious looks all around.

A compromise is typically reached: a dedicated environment defined by the app team, supported by the infrastructure team. Call it a sandbox, a private cloud — whatever.

Here is where the software-defined technology is finding its beachhead today. Here is where application developers and devops people will collaborate to continually improve the effectiveness of operations.

The presumption — of course — is that every aspect of IT infrastructure delivery is under software control: dynamically composed and abstracted from specific hardware implementations.

History Does Repeat Itself

When UNIX and client-server first appeared in data centers, it was managed off to the side by a dedicated team. They had a handful of applications they were responsible for, and the worked closely together to deliver the results.

Application and business owners liked what they saw. Things got done faster, more flexibly — and more cost-effectively — than working with the mainframe team. Relatively quickly, the UNIX footprint expanded, and the mainframe footprint declined.

One of the historical turning points: when SAP changed the way that companies ran their business operations, it usually ended up on a UNIX platform, and not a mainframe.

As we enter 2015, it’s happening again. Many private clouds are already up and running in one form or another. New 3rd platform applications are starting to arrive fast and furiously, and they generally end up on a platform tailored for their business requirements: fast, flexible and very automatable.

Their footprint will grow, and rapidly. They will be thought of separately at first, but — before long — they will become the new core of the application landscape. And they will drive the software-defined model as an integral component.

I suppose we’ll have to wait until 2016 to learn whether this is the year that we all feel the shift in our industry.

Related Stories

Yesterday morning, there was an interesting announcement: NetApp’s intent to sell EVO: RAIL bundled/integrated with their FAS storage array products.

“Hmm, makes total sense” I thought over my morning coffee.

But the level of resulting confusion on the interwebs was exceptional.

So I thought I’d share my personal opinions as to what’s going on here, and why I think it makes sense.

A Little Background?

Our story starts with VSAN (specifically VMware Virtual SAN), a software-only storage product from that is deeply integrated with vSphere.

Although many industry-watchers think that VSAN “competes” with other storage product out there (external arrays like NetApp and EMC, software-only products like Scality and ScaleIO), I don’t look at it that way.

In my world, products are “competitors” only if they are reasonable alternatives for similar use cases.

If I’m looking at hand tools, two different hammers might compete.

A hammer and a screwdriver certainly don’t.

Anyone who has spent time looking at VSAN comes away with the same impression — it’s a distinctly different storage offering than other alternatives. The only real discussion that remains is around use cases and suitability for specific requirements.

The big news at VMworld in August was EVO: RAIL — a hyperconverged appliance with a unique business model: VMware supplies unique software and specifies the hardware envelope; partners add value through services and additional product integrations.

EVO: RAIL incorporates VSAN as its primary storage service. The list of partners who’ve signed up so far is impressive, with more coming I’m sure.

So far, so good.

The NetApp Announcement

As a value-added EVO: RAIL partner, NetApp certainly brings a lot to the table. Their storage products offer strong data management capabilities and great control-plane integrations, including upcoming VVOL support.

They’ve also been successful with FlexPod, so they understand the converged/hyperconverged segment and how it works.

That’s why it made so much sense to me. But it seemed to confuse the heck out of other folks, so …

First, EVO: RAIL always includes VSAN -- it's integral to the product. So the NetApp FAS products don’t replace VSAN, they complement it, as would most external arrays. This is not a radical idea -- I even wrote about this a while back.

Going a bit further, vSphere policy mechanisms make it easy to automate placement across one, two or several different types of compliant datastores. Add in management integration via plug-ins, etc. and it's an entirely reasonable proposition.

Third, while it makes sense that server vendors would want to be EVO: RAIL partners, the appeal isn’t limited to that crowd and that crowd only.

While EVO: RAIL is very attractive in its stand-alone form, it also can be seen as a building-block for value-added integration: data protection, vertical applications — the list is endless.

Essentially, any value-add that works in the virtualization world also works in the EVO: RAIL world — it’s just another consumption option.

And NetApp has certainly built up considerable value-added integration and expertise in the vSphere world.

All of that comes forward with EVO: RAIL.

The Industry Perspective

The fact that NetApp and other non-server vendors would be interested in EVO: RAIL shouldn’t be surprising.

Virtualization is now ubiquitous, with VMware leading the pack. The market has responded very positively to all forms of pre-integrated infrastructure: converged, and now hyperconverged. Initial response to EVO: RAIL has been exceptionally impressive, and I can only trust that will continue.

Related Stories

Way back when VSAN was introduced, I wrote how its deep integration with the vSphere kernel and software stack gave it some pretty interesting advantages — especially as compared to any storage software that ran in a guest OS.

I sparked a bit of a debate, but that's normal :)

Well, VMware is not the kind of company that wants to preclude deep, value-added integration with vSphere, so I knew it was only a matter of time until one or more storage software vendors could claim that, yes, they too were “kernel integrated” with vSphere — at least, to some degree.

So now I’m in the slightly awkward position of having to dig even deeper into the topic for those that care.

The technical distinctions do matter to a certain crowd; everyone else might want to skip this post :)

If You’re Doing Storage Software …

External storage arrays are well-established entities when it comes to performance, availability, functionality, support model, etc. So what inherent challenges arise when one attempts to translate that known model to software running on generic servers?

Quite a few, it turns out …

For starters, you’ve got to get the basics right: performance, availability, functionality and manageability. Just the same as with external arrays.

Once you get past that, there are more subtle challenges to deal with. And the path to solving many of them lead directly to interacting with the kernel, especially in virtualized environments.

Hence the growing interest in storage software products that are kernel integrated to some degree.

How “Kernel Integrated” Are You? Let Me Count The Ways …

As we start to dig in, you’ll hopefully notice that being “kernel integrated” isn’t exactly a black and white proposition — there are at least 50 shades of grey.

And -- at least in my opinion -- there is a relevant difference between lightly touching the kernel in one area, and being deeply integrated in many areas.

As more storage software vendors start to claim kernel integration with vSphere (or other hypervisors), IT pros will want a deeper understanding of what questions to ask, and how the answers might matter.

One relevant area where the topic pops up is resource utilization.

External storage arrays have ample, dedicated CPU power, purpose-built networks as well as plenty of memory that doesn’t need to be shared with other applications.

When bringing storage software onto servers, one approach is to simply demand similar, vast amounts of dedicated resources — up to and including insisting on dedicated servers to run the storage software.

Want your storage software to use the same server resource pool as everything else? Dynamically share compute, memory and network? If so, your storage software is going to have to be awfully smart about how it uses resources: taking only what it needs, and only when it needs it.

In a virtualized world, those resources are controlled by the hypervisor, so we have our first desirable aspect of “kernel integration” — the opportunity for an efficiently shared pool of dynamic resources vs. carving out dedicated resources just for the storage software.

Another area where kernel integration becomes relevant is performance.

Storage arrays have dedicated control of the underlying hardware they use, and are frequently front-ended by low-latency, dedicated networks, e.g. FC.

How does this translate to storage software running on servers?

If we’re talking about storage software running on dedicated servers, that’s one approach which gives the storage software complete control of the underlying hardware.

But that’s not without a few drawbacks: a distinct management environment, no ability to pool resources, etc.

Ideally, all hardware assets would be virtualized — using the same tools, etc. — and the storage stack could know enough about the underlying hardware to get decent performance without bypassing the entire virtualization model.

And then there’s the matter of optimized IO paths that use a minimum of CPU cycles to get their work done -- more goodness from deep kernel integration.

So we have a second desirable aspect of “kernel integrated” — the potential to extract the most performance from the available (virtualized) hardware.

Turning our attention to the network, one attractive approach to boost performance is to use a non-standard client protocol (vs. iSCSI or NFS) to communicate between client and storage server, especially in all-virtualized environments. VSAN, for example, uses this approach to great effect.

Once again, doing this sort of custom storage client (at least in vSphere) requires access to the kernel, so we have yet another desirable aspect of kernel integration — improved protocol performance.

On To Management … And Problem Resolution

In any converged environment, the goal is to manage a dynamic pool of shared resources vs. individual technology stacks. You have a pool of CPU, memory, network, etc. — what is it doing as an integrated whole, from the perspective of the application?

Management plug-ins are fine for what they are, but the best they can offer is a bridge between one domain (e.g. storage) and the big picture. Creating the logical connections between what storage says it is doing — and the rest of the picture — is often left to the lucky administrator.

So we have one more desirable aspect of “kernel integration” — a potential building block for storage services to better appear as a seamless part of the whole vs. presenting a distinct management domain that must be bridged.

Problem resolution is a mixed bag, sorry to say. One of the strengths of a dedicated storage model — whether hardware-based or software-based — is that everything is reasonably well bounded, and communicates over standard protocols — making it somewhat easier to isolate problems when they inevitably arise.

Much less so the case when we’re talking about storage software and kernel integration. There are new ways that one actor’s code can impact another, and it takes a different set of skills and tools to figure out what’s really going on.

Better resource utilization, greater performance, a foundation for better management — is it no wonder storage software vendors want to get deeply embedded in the hypervisor kernel?

Given the discussion here, best to think of kernel integration as being on a continuum vs. an absolute.

However, lots of detailed questions inevitably resulted. Better to write a few things down, no?

While I prefer not to discuss the precise details of exactly how deeply a given vendor’s product is or is not kernel integrated, I did want to help create a framework for understanding what questions one might ask if curious, and how the answers might matter.

To be clear, I am not picking on ScaleIO, or anyone else. For VMware, they are the very first non-VMware storage product that has invested the required effort and thus make this claim. Again, I believe this is a good thing.

Before long, there will certainly be others.

Why? When it comes to storage software, customers are beginning to understand the benefits of different aspects of deep kernel integration, and it’s incumbent on VMware to work with selected partners to achieve this in a reasonable fashion.

Just be aware that — as always — there’s more to this than meets the eye :)

I was born in 1959. I guess I have the dubious honor of watching the world change for over half a century.

Yes, I could fill several uninteresting pages with the rapid pace of innovation in technology, human health, physics, economics, chemistry, etc.

It seems the boundaries of human knowledge continue to expand like a supernova.

More interesting to me is how this new world is changing us — as individuals, and as members of society.

It’s easy to get caught up in the wave of “now”, and lose sight of how we used to think about the world.

But, make no mistake, as we change the world, the world changes us.

A Starting Point

If I could point to one world-morphing change above all else, it would be the internet — and everything that goes with it: the web, mobile devices, search engines, big data, the proverbial IoT, social media, constant connectivity — the whole online world

We all realize the internet is a big deal, but just what is it doing to us personally?

Staying Constantly Connected Is A Given

I do love staying constantly connected to my family and friends. Huge win. But not all is rosy.

Sometimes I miss the days when I’d go home from work, and work would stay at work until my return. Now many of us are “reachable” around the clock — you never really get away from work unless you make a focused effort.

I still can use airplanes as an excuse, though. For the time being, that is.

It appears we all have quickly adapted to having our work life and personal life blend. But I’m not sure we’re all truly happy with this new state of affairs.

We prefer to have some modicum of control over our lives, and it's clear we've given a up a bit.

Feeling Suddenly Disconnected

Ever gone somewhere and realized you’ve left your mobile device behind? I get struck by a brief moment of anxiety where I realize I’m no longer connected to world: naked, isolated and alone.

The horror!

I then remind myself that I’ve spent most of my life without a mobile phone welded to my hand, and haven’t suffered for it. I’m being irrational.

The formal term for an irrational fear is a phobia.

So, yes, our connected world has given rise to a new phobia: the fear of not being connected.

The New Complexity

Doing everything online is certainly simpler, but can create a new complexity.

Everything is ducky until it’s not working. Example: it’s now easier for me to reset a password than to remember the damn thing. Example: auto-paying your bills is wonderful until you get lazy and realize someone has been ripping you off for the last eight months. And so on.

Don’t even try to change an email address (or a mobile phone number, credit card number, mailing address, etc.). You’ll be updating online forms forever. Just making a list of what needs to be changed is a major endeavor.

And I haven’t adapted well.

Expertise (And Ignorance) Is Now A Choice

The collective universe of human knowledge and opinion is now just a few clicks away. All you need is a bit of motivation and simple technology. If you choose, you can be incredibly well-informed on all sorts of topics.

How has this changed us?

We now make clear decisions on what subjects we wish to be well-informed, and those that don’t interest us. Our choices have little to do with what we studied in school, what we do for a living, etc.

Technology, economics, business, politics and cosmology interest me greatly. Fashion and pop culture, not so much. As a result, I generally suck at trivia games.

Our personal expertise is now driven by our interests and passions, and not our formal education.

We also make clear choices as to what we want to ignore. Big change.

Privacy? What’s That?

The notion of personal privacy was an important social value in my past. I was taught to respect other people’s privacy, and to expect that they respect mine. No longer, it seems.

I am now fully aware that any interaction I have in the online world should not be considered as private nor confidential.

As an example, my bank keeps my financial records reasonably private (unless some government entity asks for them), but my demographic information appears to be for sale!

The same goes with the retailers I deal with, the phone and cable companies, my health care providers, and so on. Everything I do or say is now part of the global digital record, and theoretically up for grabs to almost anyone who’s interested.

While I certainly continue to appreciate privacy and discretion, I can no longer can assume this is the default.

A few years ago, this was a very discomforting thought for me, as I’m sure it is for others. But it appears we are largely powerless to stop this new societal norm.

Being philosophical, I remember we were taught in school how to prepare for a surprise nuclear attack by ‘duck and cover’ under our desks. Lovely.

I suppose if I was able to get my head wrapped around the idea of imminent nuclear destruction, I can learn to adapt to zero expectations of privacy.

I had no choice in either matter.

Learning To Filter

Most anyone who spends time connected learns to filter: click bait, scams, trolls, email, etc. Yes, you can automate some aspects, but we’ve all learned to look at something, size it up quickly, and decide “pass”.

If I think back just a decade or so ago, there wasn’t such a big need to do this. But the small dribble of unwanted communication has become a torrent, so we’ve learned to defend ourselves — if nothing else, to reclaim our lives and our sanity.

Side note: while learning this new skill took conscious effort on my part, this particular skill has come quite naturally to all of my children.

But there’s a dark side. If you have a very narrow point of view on a particular issue, it is now very easy to filter so that all you see is content that you agree with. I occasionally meet people who live in these self-reinforcing ideological bubbles, and it is most definitely not healthy.

I’m guessing that before long, we’ll have a formal name for this syndrome.

The Rise Of Individual Voices

Authority used to be automatically conferred on large groups: big companies, the government, political parties, associations and the like. We tended to believe what they said because they were, well, big!

It looks like size has become a disadvantage. I see that many of us now routinely discount what we hear from larger organizations, perhaps believing that there’s another agenda at play.

We now appear to give more credence to individuals than groups. Maybe it's because we can relate to individuals; faceless organizations less so.

For those of us who have learned to harness our voices, this has largely been a boon. But now, if we want to be heard, we are required to learn how write, email, blog, tweet, present, etc. effectively.

We also have to become comfortable with being occasionally demonized for what we’re saying. But it’s a small price to pay.

Back when, you mostly interacted with people who were geographically close to you: your family, friends, neighbors, co-workers, etc.

As I think about it, I now routinely interact with people on every continent, usually with very different perspectives and experiences than my own.

Often, I’ve never met them in person. As a result, much of the latent stereotypical processing on my part is thus defeated.

I consider this an enormous gift.

It’s changed my perspective on the world for the better. As I trust it is changing others as well.

Forming Independent Opinions

I’ve always tried to form independent opinions, but in an un-connected world, it can be damn hard. Your sources of information are limited. Your personal network is constrained.

Now it’s much easier. I have the benefit of a wide world of diverse information sources (none of which is entirely accurate) and an enormous universe of knowledgeable people (also rarely entirely accurate).

As a result, I’d like to think I’ve gotten a lot better at constructing my personal opinions: better informed, more data, etc. My children appear to be getting pretty good at this as well.

But it does take some work.

And The Changes Have Just Begun

This pervasive connectivity is a comparatively recent phenomenon — only the last decade or so. We now constantly consume, interact — and act as sensors.

Like so many people in this industry, I can get easily enamored by Big Ideas. Powerfully intoxicating, they take your mind off the day-to-day, and transport you to a different place that might exist in the future.

Like a moth to a flame, I’m drawn in — and it takes major willpower to put them down, and move on to something else. Fortunately, I don’t appear to be alone in this regard.

Over the course of eight years of blogging and 1200+ blog posts, there are clearly times when I have fallen prey to the seductive power of Big Ideas.

I thought it might be fun to go back and ask the question — where are they now?

It was an interesting time. Virtualization (primary VMware) was clearly on its journey of encapsulating the majority of enterprise workloads.

Nick Carr had written “The Big Switch”, which foretold that a world of cheap, limitless computing from public clouds would consume all forms of enterprise computing — a premise I didn’t entirely agree with at the time, and still don’t to this day. Amazon and Microsoft were trying to convince the world that public was morally superior. Again, I would disagree.

The clouderati argument was that enterprises couldn’t possibly compete with vast, public clouds. I saw that enterprise needs were different, and enterprise IT groups were fully capable of using the same technologies and operational models found in public clouds.

So I postulated the emergence of private clouds: cloud operational models, entirely under enterprise IT control. Not for everyone, certainly, but for a lot of people. I think I was one of the first people to talk about this idea -- not that it matters.

I took a lot of flak. Simple enough idea, but would require a complete transformation of the IT model from silos to services — that was going to be the hard part.

So, where are we in 2014?

The majority of larger IT organizations I work with have something that they’d describe as a private cloud. Maybe it’s only being used for dev and test; maybe it’s running most production workloads — but it’s there. And, alongside, new processes that don’t look like the old ones.

The private cloud discussion is now evolving to hybrid clouds (federating private clouds with external service providers), but — make no mistake — the notion of a private cloud is now integral part of IT thinking these days.

IT Transformation

If one accepts that the IT model has to move from project management to service delivery, IT has to transform its fundamental operational model. More challenging, the broader organization has to transform how it views and interacts with IT.

This deep realization that the next transformation would be more about people and process (and less about cool technologies) hit me like a ton of bricks in 2010. We, as technology vendors, could make all sorts of cool cloud-enabled stuff for the enterprise, but could anyone consume it?

At that time, I was beginning to be exposed to IT leaders who had realized the world was changing, and had to lead their organizations through a fundamental change. They had powerful stories to tell.

IT becomes the internal service provider of choice. They build and broker services that the business wants to consume. They see themselves competing with external service providers, and end up emulating their models and practices.

I presumed that the forces that were causing these few IT leaders to undertake perhaps the most massive undertaking of their careers weren’t unique. More would inevitably follow.

So, where are we in 2014?

To be sure, many IT organizations have gone through a fundamental model shift, and many more are somewhere along the spectrum towards a newer model. The really big IT shops generally move more slowly, the mid-sized organization can move more quickly.

Yes, this one came to pass, but if I’m being honest, it’s been a much more gradual process than I originally envisioned. As a result, I have a new appreciation on just how hard it is to transform a large organization — even when the evidence for change is overwhelming.

Corporate Social Media

One fun gig I did back in 2006 was to hammer together a corporate social media strategy for my then-employer: EMC. While the topic might seem almost humdrum today, back then it was a powerful mix of excitement and potential danger.

I quickly gained a deep appreciation on how powerful these new engagement techniques could be, and — at the same time — how these new ideas could absolutely terrify risk-adverse people. I wasn’t the first one to set out in this direction, but I certainly was one of the first.

Clearly, the idea of having a corporate social media strategy is very mainstream. The obligatory Corporate Twitter Account, for example. Internal social collaboration, not so much.

The disappointing part is that I see that many, older and established companies never really “got” what this was all about. I see them using social media as frosting, vs. baking a new cake using its principles.

Big Data

EMC acquired Greenplum in 2010. As part of that process, I started to get exposed to what people were doing with massive decision-support databases built on data from diverse sources. I even met a few data analysts who had clearly taken their game to a new level.

The industry had started to use the term “big data” for this new way of thinking about harvesting and capitalizing on vast and various data sources. Sure, there was no clear definition — is there ever? — but it was plain that something was happening.

The first thing that was clear that this wasn’t your father’s data warehouse — it was something new. The second is that a new magician was required — the proverbial “data scientist” who could test the data against different hypotheses.

And, finally, organizations that wanted to benefit had some serious culture change to embrace — including learning to listen to the data vs. the HIPPO (Highest Paid Person’s Opinion)

Very quickly, the technology faded into the background for me, and the real story was how different organization were reacting to the opportunity/threat.

So, where are we in 2014?

Safe to say, most business and IT leaders “get it”. Maybe they’re doing a lot with big data (many are), maybe they have a few projects and learning where it fits best (the majority), and a few have decided it’s not a relevant concept in their world.

I said at the time that big data would change the world, as did many others. It has, and it will continue to do so.

Digital Business Models

In 2012, I began to realize that the big ideas in my world (social, cloud, mobile, big data, etc.) were actually manifestations of something even bigger going on — a fundamental transformation in business models. The Great Attractor, if you will -- moving everything in the universe with a soft, persuasive power.

Most familiar business models have their roots in the physical world. But in a world where everything is digital, shouldn’t business models evolve as well?

Showing my continued lack of creativity in naming things, I dubbed these new models “digital business models”. I saw that they had a repeatable structure and a natural evolution, and could often be traced back to a small group of visionary executives within the organization who saw what was happening — and decided to get busy!

So, where are we in 2014?

I’m not quite sure. There are clear examples of many successful companies that are going all-in the required components: mobile, cloud, big data, social, content, etc. I’m not privileged to see whether there’s a master strategy behind all of that, or whether it’s just the natural ad-hoc evolution of things.

One thing is clear. Just about everyone who thinks about these things is clearly aware that we now live in a digital world — and, as a result, we’re going to need a business model built for that world vs. the one that came before.

Converged Systems

I was one of the earliest and largest proponents of converged infrastructure, going back to 2009 when VCE started to be talked about. Yes, you could say it was part of my job, but there was more at play.

I had been exposed to literally thousands of IT shops that were getting buried in IT infrastructure complexity: disparate technologies, firmware revisions, interoperability, design and architecture, etc. The world needed something different to consider.

Enter VCE and Vblocks — data center infrastructure, delivered and supported as a product, and not a collection of piece parts. I felt that this model could be a huge boon to IT groups who wanted to use infrastructure, and not hand-craft it.

So I started to passionately write about it. There was an enormous amount of blowback from many corners — how could you ever suggest limiting people’s choices? — but I saw it as a tradeoff that many were more than willing to make.

Fast forward to 2014

VCE and Vblocks are quite successful. Every major hardware infrastructure vendor has “their version” of converged infrastructure. We’ve also seen the growing popularity of “hyper-converged” infrastructure — but not quite accurately named, in my book.

Convergence has been a force in almost every area of technology — why not IT infrastructure?

Storage Stuff

I spend a lot of time with storage, so — naturally — many of my predictions came true. Not because I had a magic crystal ball, more because it just seemed so obvious.

Going back many years, I was an early fan of object-based storage with rich, user-definable metadata. It could solve problems that ordinary file systems and databases couldn’t.

As we sit here in 2014, object storage is most definitely a thing — if not in the enterprise, then certainly with service providers and cloud operators.

EMC shipped their first enterprise flash drive at the beginning of 2008. At the time, I realized that the inevitable had begun: a transition from disks being the primary storage medium, to flash. Now in 2014, I think it’s obvious to just about everyone.

There was a time when certain people thought that iSCSI would take over the world, and bury FC interfaces. Not so fast, I said. I think I even remember taking a few bets on how things would end up. Well, it’s 2014 and both iSCSI and FC are alive and well, serving their respective use cases.

Nobody "won".

There was a similar wave where the NAS aficionados thought that their simpler presentation would obsolete block-based SANs. Not so fast, I said — there’s transactional applications, and today’s NAS implementations can’t offer the same predictable low-latency as block-based approaches. Still largely true in 2014, as near as I can tell.

But those were all easy calls, I thought.

Software Defined Storage

It’s early days with this new Big Idea. People are arguing definitions, the value proposition, use cases, etc. Same thing happens when any new Big Idea manifests itself.

But, make no mistake, software-defined storage will change how we think about storage.

It just might take a year or two to become self-evident to others :)

What I've Learned

If you're close to certain topics, it's not all that hard to predict what will happen next. There's very little mystery involved, nor are massive amounts of intellect required.

If you decide to share your opinions about what might come next, be prepared for stiff opposition. Don't look for validation or rewards, it won't be coming. And no one wants to hear someone crowing "See?? I told you so!".

Just think of it as your token contribution to the efforts of others :)

After finishing university work, I assiduously kept many of my textbooks, thinking they would be incredibly useful at some point in the future.

After all, they were so expensive!

I was a fool. After the second house move, I tossed them. They were interesting, but certainly not worth lugging around. And nobody wanted them, either.

As I reflect back on various chapters in my personal Storage Encyclopedia, I’m realizing that there are vast tranches that are candidates for leaving behind.

Yes, many topics were once interesting, but not exactly useful going forward.

I suppose that’s progress?

On Knowing A Lot About A Topic Most People Find Incredibly Boring

When working with colleagues or customers, I’ll often get the comment “gee, you know a lot about this stuff”.

I guess I should by now. I’ve spent over twenty years working with just about every aspect of storage — from the underlying technologies and their supply chains, to macroeconomic consumption patterns and operational models and the structure of the overall storage industry.

With a whole lot in-between.

I never set out to be an expert in anything, but I guess that sort of happened along the way. It's not very useful at parties, though …

As I watch the industry changes start to unfold, I’m now fully cognizant of the fact that much of what I’ve picked up over the last twenty years will largely be obsolete in the next two. It’s humbling, to say the least.

I thought I’d share with you my personal list of “once cool, now not so much” storage topics that I’m faced with.

Understanding Disks

For the last twenty years, storage has mostly meant disks: rotating rust. Yes, there are forays into cache and tape, but for so many years it’s been all about the disks, baby.

The more you knew about them, the better: how big, how fast, how reliable, who made them, etc.

But in a world of flash, who cares so much about all that? Optimal data layouts: sequential data, transfer rates, striping, short-stroking, sweet-spotting, head settle times — all of that? Failure modes and reliability curves?

Yes, it’s still somewhat relevant, but now it’s at the margin vs. mainstream.

Sort of like writing optimized COBOL code.

Tiering disks for different performance/capacity tradeoffs used to be a big deal. Then it was tiering between flash and disks. Before long, we’ll be just throwing lots of flash at anything that requires a modicum of performance, and call it a day. Easier and more predictable.

Bye-bye, my arcane disk knowledge ...

Understanding Storage Arrays

Working as I did for an array company for so many years, I spent a lot of time appreciating the finer details of storage array physical design: internal architectures, redundancy provisions, caching algorithms, power management, processor choice, density, cooling, serviceability and so on.

Fascinating stuff, to be sure.

But in the emerging world of virtualized storage software that runs neatly alongside other tasks in a standardized server farm, how much of that arcane knowledge is useful going forward?

Not as much, I’d argue.

Modern storage software is being designed to run on generic server clusters: exploit their strengths, and compensate for their weaknesses. Yes, there are a few storage-specific considerations when designing server farms to support internal storage: blades aren't as attractive for example, and IO controller queue depth really, really matters :)

But it's comparatively not a lot to be concerned with for most folks -- certainly many orders of magnitude easier than the vast body of arcane knowledge required to understand enterprise storage array design.

Not too long ago, an enormous amount of human effort went into designing dedicated storage networks with defined performance, availability and manageability attributes.

In the next wave of software storage, you build a nice server fabric, and everything else runs over it, including storage traffic. No, that’s not heresy, just being pragmatic.

Heck, newer products like VSAN don’t even need to support familiar storage protocols — all they do is serve storage to VMs, so none is needed.

Before long, those server fabrics will reach out to embrace familiar storage arrays that will now look like just part of the overall data center network topology using RDMA protocols. Our familiar storage presentations will just be optional access models that are largely there for legacy purposes.

And we’ll become quite comfortable in accessing storage in higher-level terms, such as key-value.

Bye-bye, SCSI protocol?

Understanding Storage Management

For so many years, storage management was a largely self-contained topic. Here is how you set up your environment, here is how you run operations, here is how you troubleshoot, etc.

Going forward, though, storage can no longer afford to be an island. At its essence, that’s what software-defined storage is all about: the dynamic composition of storage services.

Storage now becomes a service that’s requested programmatically, without waiting for the storage admin team to get around to your low-priority work order.

The high-level storage disciplines are still important, but now they’re done in an entirely different way.

And troubleshooting now means considering the whole stack, and not just the familiar storage pieces.

Unfortunately, there isn’t a book anywhere yet that shows you how to do this :)

What Won’t Change

Thankfully, I think I’ll be able to hang on to at least a few useful chapters in my personal Storage Encyclopedia in a few years time.

Data volumes continue to grow, and budgets will always be tight. There will never be any shortage of people looking for the best bang-for-the-buck. Thankfully, that will never change — although the best techniques to achieve this certainly will!

There will always be periodic waves of venture money pouring into storage startups, so we’ll always see the latest incarnation of the Storage “Circle of Life” — continuous creation and destruction.

Keeps things interesting, doesn’t it?

Storage management models show no signs of finding a theoretical optimum. There will always be new models, new approaches to automation and new approaches to defining and implementing policies. And they will always be difficult to adopt for any organization that has an entrenched way of doing things.

Solving problems at scale will always remain fascinating. No matter how far the industry pushes technology, there are always interesting use cases just beyond what can be practically done today. It’s just that the numbers get stupidly bigger and bigger.

Making things ever-more simple and predictable is one of those never-ending goals, either. No matter how simple / easy / integrated / converged we make storage, there’s always a way to do better.

Hopefully, I’ll have enough chapters in my personal Storage Encyclopedia to remain relevant in a few years time.

For the most part, people were disagreeing without being disagreeable. How civil.

That being said, I tend to reflect on things after the fact. I came to the conclusion that it was an incomplete discussion on several levels.

If you’re not into storage stuff, perhaps this would be a good post to skip. A friendly reminder: everything in this blog represents my personal opinion, and is not reviewed nor approved by employer. Or anyone else for that matter.

Why Are We Talking About This?

If you’re in the world of enterprise IT, storage is a big deal — it’s where all the information lives. Storage in an enterprise IT setting can be costly and difficult to manage.

Storage professionals can make a good living by helping IT teams to tame the storage beast so that’s it a bit less unruly.

The winds of change are upon us, though. Server designs (and storage software) has matured to the point where it can be seriously considered as an alternative to the familiar external storage array for a growing number of use cases. Management models are converging, where it's becoming more common to see servers and storage managed via a single role.

New choices abound. And when there are new choices to consider, the debates start in earnest. All part of the fun!

The Contestants

Most everyone should be familiar with external storage arrays. These are purpose-built boxes, often comprised of commodity components, running specialized software with a wealth of functionality: snaps, replication, tiering, dedupe, encryption.

It’s a comfortable, familiar model. The servers go over here, and are run by the server guys. The storage arrays go over there, and are run by the storage guys.

Next up, VSAs, or virtual storage appliances, if you prefer.

These are pure software stacks that run on familiar servers. Their presentation model is also familiar: they expose iSCSI, NFS, etc. and are basically run like dedicated storage, except using off-the-shelf hardware. The software may either run physically on bare metal, or virtualized in a VM alongside other applications. Many examples of these, with more coming.

And, finally, the new kid in town: VSAN, or VMware Virtual SAN more formally.

Like a VSA, VSAN runs on most anything that supports VMware, alongside other applications, with a few caveats. Unlike a VSA, VSAN is an extension to the ESXi hypervisor: it’s deeply integrated into everything the hypervisor does. This may sound like a minor implementation detail, but it’s not — it fundamentally changes the value proposition of the product.

So, What’s Important Here?

Being a huge fan of oversimplification, I’d like to use three primary criteria for comparing the “goodness” of our contestants.

First, does it do the intended job? There’s a vast universe of workloads out there, each different, and a given technology is either a good fit or not. Because we tend to use imprecise labels to describe workloads, there’s a bit of confusion, e.g. all Oracle workloads are not created alike.

Second, does it give good value? Here we have a broad discussion around price/performance, price/capacity, price/functionality, etc. We also want to consider the operational aspects over the life cycle: implementation, management, upgrading, etc.

Finally, does it prepare me for the future? Storage evolution continues to pick up speed. We’ve got new devices (NVMe, motherboard-based flash), new converged management models, and entirely new application workloads to consider. Nobody wants to end up in a technological cul-de-sac, especially when you consider the 3-5 year average life cycle of a storage investment.

What will the world look like in 2020?

Cross-Compare: VSA and External Arrays

Before we go too far, it’s important to point out that VSAs seem to target two distinct markets. The first are smaller environments where budget and resources are inherently constrained. The second are larger, scale-out environments that look more like a service provider or cloud model.

Both VSAs and external arrays preserve the familiar storage-centric management model. You provision and manage storage separately, and make it available through a network protocol: iSCSI, NFS, object, etc.

If we go to our first criteria (does it do the job?) the first thing you’ll notice is that very few vendors are positioning VSAs for bread-and-butter IT workloads: Microsoft apps, databases and the like. Here, external arrays are the more likely choice. Write-heavy transactional workloads and most VSAs don’t seem to get along well.

VSA vendors, please correct me if I'm wrong here :)

On the second criteria (good value for money?), there are pros and cons on both sides. VSAs have the obvious advantage of running on most anything, which is especially convenient if you have unused (or underused) hardware on hand.

Apples-to-apples on a new acquisition is a much closer compare, financially speaking. Parts is parts, although pricing on server hardware is generally much more transparent than larger array hardware.

Operationally, larger-scale VSAs seem to require more “fiddling” than purpose-built arrays, but that's just my observation. The arrays also have the advantage that their software knows *exactly* what it’s running on, which confers noticeable benefits in supportability and performance optimization.

On the third criteria (does this prepare me for the future?), some may see the VSA to have a small advantage.

Upgrading to new hardware with a VSA is usually the incremental insertion of new technology into your server farm and updated software. While arrays can be upgraded within a range, they periodically go through a major product revision, which generally results in acquiring a completely new array.

Most often, you can take your VSA licenses to new hardware, new hypervisor, etc. Similarly, you can take your array to support new servers, operating systems, hypervisors, etc. In my eyes, that’s sort of a wash.

When cool new storage technology comes out (new flash devices, etc.) the VSA approach makes it easier to adopt vs. waiting for your array vendor to support it, either in the array you own today, or more likely the next array you buy. Depending on how hard you want to push the technology curve, that could be a factor.

Choices, choices …

Cross-Compare: VSAs vs. VSAN

We actually need to make two compares here: one compare for smaller environments, and another for larger, scale-out environments.

At a high level, there are two key differences to be aware of.

The first is the technology model; VSA storage software runs in user space as does any other application; VSAN is an integral part of vSphere and the ESXi kernel.

The second is the management model: VSAs present as familiar, separately managed storage, while VSAN is an integral part of VM management — which can be frustrating for traditional storage types :)

Let’s take on our first criteria — does it do the job?

When we look at the smaller end of the market, VSAN has a minimum of 3 nodes (two for protection plus a witness), while some VSAs can support 2 or even 1 node — of course, with associated compromises in certain kinds of availability handling.

But if you’re driven on price, price, price — that could be a factor.

At the other end of the market (large scale out), VSAN does scale considerably today (32 nodes, 4.4 petabytes using 4TB drives), with rumored expansion to 64 nodes and 6TB drives (that’s 13.4 PB for those of you keeping score). I'm not going to confirm or deny rumours here, but my point is -- not small, by any means.

But, by comparison, some VSAs can get truly big: many hundreds or even thousands of nodes. That could be appealing to cloud-like service provider environments where you’d like to manage the storage farm separately, or huge content farms.

Arguably, many of today’s VSAs offer richer data services than today’s VSAN. But not everyone needs a great deal of those features, and -- besides -- I don’t think that situation will persist for long :)

Let’s move to the second, more controversial question — good value for money?

Both VSAs and VSAN run on server hardware, but — in every test I’ve seen run — VSAN consistently extracts more performance, and uses far less server resources, than any VSA I’ve seen.

No, I can’t point you at published documentation on this -- I really wish I could -- but it’s real and easily observable over a wide range of configurations and workloads. Do your own testing (or talk to someone who has), and you’ll see what I mean.

That’s just one of the advantages of being integrated into the kernel vs. being forced to run in user application space.

As a standalone, list price proposition, some have thought the license cost could have been lower for VSAN, but there’s more to it than that. First, VMware customers tend to buy a lot of different VMware products, so list pricing is certainly not the norm. Second, it’s per-socket pricing (vs. capacity pricing) which can be very cost-effective if you want, errr, capacity.

But all of this masks the real win: operational efficiency. Talk to any VSAN user, and the first thing they’ll usually say is “nothing could be easier” in day-to-day operations. It’s really quite remarkable.

If you’ve got (or want!) a single role managing both servers and storage for your VMware farm using a converged, application-centric policy-driven model, you’ll understand the unique appeal.

Now onto an even more controversial subject — prepared for the future?

Both VSAs and VSAN can accommodate new storage technology quickly, simply through software updates. Let’s call that a tie.

One of Nigel’s points in his blog post is that VSAN is tied to vSphere, and many VSAs are hypervisor agnostic. Without sounding too partisan, this is a red herring for most people. Many shops are fully committed to vSphere, and plan to be that way for a while.

It is a rare and painful occurrence for someone to actually attempt to replatform a perfectly good vSphere cluster onto another hypervisor, but — theoretically — that’s a valid point. And if you do decide to migrate, there are nice tools (e.g. storage vMotion) to help you out. Just to be complete, there are a handful of shops that fully intend to run multi-hypervisor, and want a single storage solution that’s uniform across all environments. VSAs would win here, as would external arrays.

More strategically, if you believe the future involves software-defined data centers (and software-defined storage), I would offer the mildly partisan argument that VSAN has a long term advantage here. VMware is pouring substantial resources to make VSAN the poster child for software-defined storage as part of the bigger, fully-automated SDDC picture. You see a lot of that already in today's product.

Cross-compare: VSAN vs. External Arrays?

Although this was the topic that Nigel set out to cover, I don't see that as particularly useful.

There's just too much discontinuity that results from trying to directly compare the two. Many people get lost in the discussion, and miss the key points. Instead, I've found the topics are easier to understand when you put them on a continuum.

One set of things happen when you go from external arrays to considering VSAs, and break the vapor lock between external storage hardware and storage software. Pros and cons, to be sure.

And another set of benefits materialize when you consider moving from VSAs to VSAN, and no longer have storage software being forced to run as any other user application.

Stepping Back A Bit

A few years back, VCE was created on the premise that many people would see value in having storage, network and compute integrated and supported as a single product. The category of converged infrastructure was born. Very popular, as it turns out.

A bit later, a few enterprising startups took VSA technology, and wrapped into a single, integrated appliance. They started to use the term hyperconverged infrastructure to describe this approach. Also becoming popular.

VMware came along this year, and went one better -- integrating storage and network as part of the hypervisor.

The most logical term I can think of is hypervisor-converged infrastructure. So far, doing quite well thank you. And now there's EVO:RAIL for those that prefer the simplicity of pre-integrated hardware.

The point here is that more and more of the market is starting to prefer to consume storage functionality as an integrated part of the whole vs. a standalone proposition. Whether that's converged, hyperconverged or hypervisor-converged -- it's an undeniable factor in the storage market and broader IT thinking today, and will continue to be so in the future.

Despite some people banging on about lock-in being the work of Satan, or similar :)

The nice thing about the storage market these days is that there are so many choices out there -- technology, management model, consumption model, etc. -- keeping every storage vendor on their toes.

Especially when they neatly summarize a very complex and nuanced discussion. I heard this one being used at one of our internal VMware meetings, and it rang like a bell in my brain. I’ll reserve the right to give credit where credit is due when I track down the original source.

Hybrid — as in hybrid clouds. “-icity” — a measure of degree, as in “elasticity”.

Hybridicity: a measure of just how hybrid is a particular cloud computing environment. In this context, more hybridicity is better than less hybridicity.

If we’re going to see more mainstream adoption of public cloud services by enterprise IT groups, I believe the core argument is going to be around hybridicity — how much, what kind, how good?

Standard Experiences Matter

I travel a lot, and I frequently rent a car. Fortunately for me, driving a car rented from vendor A is remarkably similar to driving a car rented from vendor B. The control surfaces are the same. The behaviors are the same. And, if something should go wrong, my problem resolution process is the same.

Not to belabor the point, but imagine if every car rental agency rented fundamentally different cars?

This one uses a familiar steering wheel, but this other one uses handlebars. The throttle on this one is a pedal, but over here it’s a squeeze grip. The fuel gauge is easy to read here, but this other car rental company put it somewhere I can’t see, and I’m not sure what it’s telling me.

Silly, right? Of course all cars use a consistent interface with consistent behaviors, otherwise how could you expect people to rent one?

But that’s exactly the situation we’re in when it comes to today's public clouds.

Every vendor has “their way” of doing things when you go to rent infrastructure. Maybe it’s better, maybe it’s not — but it’s guaranteed to be not only inconsistent with other public clouds, but also inconsistent with what most enterprise IT groups are doing internally.

And people wonder why public cloud adoption is so low in mainstream enterprise IT — there is very little in the way of a consistent experience.

Or, put differently, very little hybridicity.

The Reality Of Enterprise IT

I’ll let you in on a big secret: enterprise IT really isn’t about technology so much, it’s about people and process. Look on any IT "balance sheet", and you'll quickly see the massive investment is in people: their skills, their knowledge and how they go about doing their work.

The best technology in the world won’t fix people and process issues.

Present these people with a technology, or a public cloud service, that is incompatible with existing skills, tools and processes, and it immediately becomes an unattractive proposition.

The barrier to adoption is quite high, and IT management is faced with the additional requirement to peanut-butter the available skills and resources even thinner to cover the new, inconsistent thing.

Keep in mind, the argument here is not that the public cloud service is theoretically “better” or not, just that it’s quite often inconsistent and incompatible with what an average enterprise IT shop is doing today.

The Goal State Of Hybridicity

Let’s take an average enterprise shop running, say, a moderate number of VMs, maybe a thousand or more?

Ideally, any public cloud resource would appear as a simple extension of their existing clusters.

The behaviors would be the same. The control surfaces would be the same. Troubleshooting problems would be the same. Things like load balancing, and setting up for high availability and DR would be the same. Monitoring and reporting would be the same.

You’d have to look hard for differences between the on-premise resources and the rented resources.

Just like jumping into a rental car at the airport, you'd be able to figure it out without too much trouble — and start driving immediately. And switch to another provider with a minimum of hassle.

But that’s not the world we’re in — why?

Needed: A Consistent Starting Point

The norm in many enterprise IT environments is home-grown: a kaleidoscopic array of tools and scripts, a bewildering set of processes — some ad-hoc, some more formal. While that’s not ideal on several levels, it is what it is.

If we put up the concept of hybridicity and public clouds against this unpleasant reality, the challenge is obvious: by definition, there can’t be a compatible public cloud service that runs the way the average enterprise IT shop does -- they're mostly one-offs.

How can a public cloud service expect be compatible with each and every enterprise IT snowflake out there?

I’m not going to pin the blame for low public cloud adoption by mainstream enterprise IT on this and this alone. There are the unarguable economic advantages of owning vs. renting an asset if it’s highly utilized, for example.

Here is where I am optimistic.

I have now met enough enterprise IT shops heavily invested in VMware’s vCloud Suite and vRealize Suite to observe a pattern: they all have a reasonably consistent set of skills and operational processes — not to mention identical technology choices in key areas.

Much like SAP implies a good way to run your business, vCloud Suite and vRealize Suite imply a good way to run enterprise IT: continuous service delivery, consumption portals, cost transparency, integrated management and monitoring, etc.

These tools encourage processes and methodologies that make sense, and are inherently consistent with other similar users.

Here’s the win: these outfits as a group can now consider a compatible public cloud service (vCloud Air, or more broadly the vCloud Air Network) without facing the cliff of learning how to drive a car all over again. Other considerations (economic, etc.) are still in play, but a very substantial one is now off the table.

As such, this group enjoys an advantage that others don’t — easier and more productive consumption of compatible public cloud resources, should they be interested.

By standardizing on these particular tools and workflows, they’ve achieved a two-fer: a better operational environment, and new convenient consumption options via hybrid cloud.

I also get exposed to organizations trying to figure out their management stack going forward. All sorts of mix-and-match options out there. Only a few of them consider whether or not there will be compatible public cloud consumption options as a result of their choices.

Maybe they should.

Back To Hybridicity

When it comes to complicated things in our lives, we demand standardization. Cars work a certain way, home entertainment systems work a certain way, computers, smart phones, etc. We value consistency in the user experience, the control surfaces, the underlying behaviors — and what to do if there’s a problem.

Enterprise IT is no different. A stupendous amount of effort is invested in people and process which results in a de-facto standard for each and every IT organization.

Moving that investment forward seems to be job #1 for most of the IT leaders I meet these days. As it should be. “Getting to cloud” really means two things: transforming internal IT operations to an as-a-service model, and establishing convenient consumption options that are entirely compatible with their skills and operational processes.

Hybrid cloud, in its essential form.

And it seems that there are more than a few VMware customers who’ve cracked this code.

The enterprise IT model is changing fast: IT now strives to become the internal service provider of choice, offering an attractive portfolio of services that are convenient for the business to consume.

All of the sudden, IT starts to look more like modern manufacturing, and less of a project-oriented crafts guild.

As anyone who has run a good-sized business will know, it’s all about “the numbers”: how much, how good, how fast, etc.

You wouldn’t try to run a business without good financial instrumentation — and the same is quickly becoming true for many progressive IT shops.

But the question comes up — what tools exist that do all of this?

I’d like take you on a quick tour through VMware’s vRealize Business offering.

I think is one of the many “hidden gems” in the VMware portfolio. It offers a surprisingly rich suite of ITBM (IT Business Management) functionality that’s becoming essential in modern IT environments — and stands head-and-shoulders above many alternatives.

Say what you want about bean-counters: money matters.

Why This Is Becoming More Important

When IT didn’t have to move fast, its business management tools didn’t have to either. It was perfectly acceptable to run around and manually gather key metrics at the end of a quarter or financial year.

Most of the environments I’ve seen were basically home-grown: Excel spreadsheets, small databases, and scripting with a reliance on manual data gathering. Big fun, I'm sure.

That sepia-toned image doesn’t play out as well today.

Financial stakeholders (business consumers, finance, operations) are demanding near-real-time metrics around their IT consumption: not just how much it costs, but service level reporting, comparisons with other alternatives as well as modeling what-if scenarios.

They want tools. That work with a platform.

At a fundamental level — providing better information to more people results in better choices being made -- or at least better-informed choices ...

That’s the big idea here.

The Basics

Formerly marketed as “VMware IT Business Management Suite”, these capabilities are now being integrated into the broader vRealize cloud management platform and referred to as “VMware vRealize Business”.

Logically, it fits in with the other core disciplines needed for cloud management: automated delivery, intelligent operations and unified management.

I see this is a result of VMware’s ongoing commitment to the category, as well as the predominance of VMware products in enterprise private clouds.

Either way, VMware vRealize should make the shortlist when people are considering their options.

The Big Picture

Think of vRealize Business as a platform for IT financial management -- in a world where everyone's involved.

If you think the topic is strategically important (and I would argue you should), you look for a platform with great out-of-the-box capabilities as well as near-infinite extensibility.

You'll be wanted to integrate all sorts of data source (now and in the future), as well as export data to other, more specialized tools.

Hybrid cloud is a given: it's presumed you'll be using a mix of internally-produced services as well as a selection of external services. And you'd probably prefer a battle-hardened product that comes with significant IP around how things really work in an enterprise IT setting.

That's the pitch.

Key Concepts

The first big concept here is the notion of pricing IT services: a methodology for arriving at unit costs for everything from base services like storage or compute, to more complex services like application consumption or help desk.

These service costs can be marked up (or down) to expose prices to business users. Everything from license costs to manpower can be modeled using either pre-defined templates or customized approaches.

And more than just cloud services can be priced using these tools: IT projects, standard IT services like help desk, email, etc. are amenable to this approach.

Given how I’ve seen so many IT groups struggle to price consumed services (vs. simply listing the cost of the ingredients), this is a big step forward.

Anything that’s priced can be compared. Here, the comparisons can be against alternative services (or alternative sources of services, e.g. public clouds) in terms of price, functionality and service level availability.

I found it great that uptime stats were an integral part of the model here.

The second big concept here is the notion of a stakeholder: someone who needs a view into IT financial and service delivery information. While this might ostensibly be an IT role, more commonly it is not: either a consumer of IT services, or the finance function.

Needless to say, when it comes to showing dashboards, reports, etc. — everyone wants a customized view that aligns with their needs and wants. And here the product doesn’t disappoint.

In addition to a vast number of out-of-the-box capabilities, the dashboard views and reports are easily customizable. Everyone gets the view they want with a minimum of effort. Better yet, there are modeling capabilities that allow users to build what-if alternative scenarios.

There’s a boatload of additional capabilities that are worth investing the time in digging into, if you’re so inclined. While no product does everything, it’s scarily comprehensive and reflects the maturity that comes with years of real-world customer feedback.

Additionally, there’s been significant work on creating forecasting models when considering new IT services, or modifying existing ones.

And On To IT Benchmarking

Related to — but sold separately from vRealize — is the VMware IT Benchmarking Service.

Using a combination of tools and methodologies, it gives IT organizations the opportunity to rigorously compare their efficiency as compared to peer groups, using a database of thousands of data sets and over 3,500 comparable metrics.

IT benchmarking — while important — has typically been a lengthy, consultancy-led exercise that spans many months.

The VMware pitch here is simple — by using a standard methodology and automated analysis tools, accurate and actionable results can be had in a matter of weeks.

Implementing a financial management platform implies that the IT organization is well on their way to an IT-as-a-service model.

They’ve embraced the philosophy, made the necessary organizational changes, and have already stood up initial takes show back, consumption portals and the like.

I also think people shouldn’t underestimate the considerable effort required to fully integrate its capabilities into the broader organization.

To me, this starts to look like “ERP for IT Service Delivery” which means getting a lot of people to wrap their heads around the new processes. No small undertaking, but inevitably necessary.

Comes with the territory of running IT as a business ...

The Path Is Clear

That being said, I see vRealize Business is the right product at the right time for a non-trivial number of progressive IT organizations.

They’re down the path of ITaaS, they strive to run IT as an efficient and proactive business -- and they understand the need for financial transparency

Maybe they’ve tried to stand up their own home-grown solutions, or perhaps welded together a handful of point products. They now have an appreciation for how important these functions can be — and will go looking for a platform to solve today's challenges as well as tomorrow's.

]]>
]]>
http://chucksblog.emc.com/chucks_blog/2014/10/emc-federation-solutions-and-the-new-it-agenda.htmlFederation Solutions And The New IT Agendatag:typepad.com,2003:post-6a00d83451be8f69e201b7c6f0434b970b2014-10-09T11:26:55-04:002014-10-09T11:27:52-04:00 ]]>
Chuck Hollis

One of the more fascinating efforts across the EMC Federation (EMC II, VMware and Pivotal) is the substantial investment behind what is now known as Federation Solutions

Like many of you, I have developed a natural skepticism of just about anything branded a “solution”, as so many are just marketing wrappers for familiar technologies.

This time, though, it appears to be quite different — there’s real engineering at work here, pointed at real heavy-duty challenges.

Once you wade into the gory details, there’s a lot to like.

The New IT Agenda

The rationale for Federation Solutions is based on the presumption that most enterprise IT organizations will have to field 5 new capabilities in the near future. None of these should be a surprise to anyone.

1) Mobility — provide access to all applications and data through mobile devices

2) Apps — use agile development to build new customer-centric applications

3) Big data — build a business data lake to deliver insights using all available data

Any one of these is a major undertaking by itself. As I see it, many IT organizations will find themselves needing to deliver most (if not all) of these outcomes before too long.

And that’s what these Federation Solutions are all about.

The Rationale Behind The Federation

Certainly one of the more unusual business structures in the IT industry, the EMC Federation is comprised of EMC’s storage businesses, VMware, Pivotal as well as RSA. EMC, VMware and Pivotal are all independent entities: free to act independently — or collaboratively — as needed.

Examples: EMC products work well with any hypervisor, VMware partners with other storage vendors, and Pivotal is famously “cloud agnostic”.

It sounds counter-intuitive to some, but — from a customer perspective — this creates the ability to mix and match technologies and vendors more easily. Yet, as we'll see in a moment, when these technologies are integrated and easy-to-consume, very powerful capabilities result.

From my perspective as a long-time employee, it all feels rather logical and natural -- have it both ways, if you like.

Historically, there’s been a great deal of integration work done between the sister companies: EMC storage products works well with VMware’s vSphere, Pivotal’s Cloud Foundry works well on vSphere-based clouds, and so on. Choose to do business with all three sister companies, and there’s a lot of baked-in goodness ready to go.

All good, but in the context of these “Big 5” IT agenda items, much more is needed.

What's In A Federation Solution?

The idea behind Federation Solutions is simple: shorten time-to-value and reduce implementation risk for these big enterprise IT agenda items.

Hand customers a proven template demonstrating the current "art-of-the-achievable" using GA products, a methodology for simplified implementation and support, as well as professional services a-la-carte as needed.

The Federation Lab that produces this work is a real thing: real people, real equipment and a really big ongoing investment. To date, over 40K hours have been invested in design, integration, testing, validation and documentation.

The first Federation Solution out of the gate is foundational: software-defined data center, or SDDC. The remaining Federation solutions (apps, big data, mobility, security) will be built on SDDC as use cases.

An interesting wrinkle is the use of key partners to validate that Federation Solutions can be implemented from the provided documentation with expected results.

As an example, WWT was able to take the SDDC documentation, follow the provided recipe and get the exact results expected. Reproducibility and predictability is an important attribute of these solutions.

Unpacking The SDDC Federation Solution

The SDDC Federation Solution is remarkably complete in the functionality it undertakes to provide out-of-the-box. Just describing it all takes some time :)

From the outside, the user view is based on a familiar application store model; and presumes that IT will act as an approver, depending on request. Naturally, the provided templates can easily be extended for additional services.

The model is inherently hybrid: IT services can either be provided locally, or using vCloud Air — completely transparent to the user.

VMware’s vRealize suite is used to drive the required infrastructure workflows across compute, network and storage. ViPR is used to provision the desired storage service level (independently of underlying hardware) and a separate workflow is used to provision the desired data protection (e.g. backup, remote replication, etc.), including integration with VMware’s SRM (Site Recovery Manager).

The Power Of Software-Defined Networking

The integration with VMware’s NSX is very slick, and is worthy of closer examination.

VMware’s IT Business Operations is used to provide each stakeholder a transparent, customized perspective of resource costs and alternatives — giving people the data they need to make intelligent choices.

Self-Service Data Protection

One particular integration I found compelling was the combination of vCAC, EMC’s Data Protection Advisor and Avamar/Recoverpoint.

The net result is that end users can (if desired) specify their own data protection policies, monitor that backups are being done successfully, and even initiate recovery events if they choose.

As it should be :)

On To The (Public) Cloud

Finally, the SDDC Federation Solution includes the ability to transparently migrate workloads back and forth to a vCloud Air instance.

The operational framework remains the same, the tools and processes remain the same — all that has changed is where the workload is located.

Although this is an explicit process today (sorry, no automated load balancing!) it's one of the slickest production-ready examples of a hybrid cloud that I’ve seen.

What Does All Of This Mean?

While there will undoubtedly be a few IT organizations that will go for the turnkey nature of the SDDC Federation Solution (maybe on a VCE Vblock or VSPEX), I think its broader appeal will be as a reference guide for the many organizations that are consistently migrating their current environments towards SDDC.

In particular, the Federation Lab team has written a *lot* of glue code showing how specific integrations work.

As new product technologies become available — and existing ones gain new features — newer versions of the SDDC Federation Solution will continue document the state-of-achievable in a pragmatic and reproducible way.

But what was once new and exotic is now commonplace wisdom. It is the rare IT group I meet that hasn’t embarked — or plans to embark soon — on such a transformational journey.

In technology, form follows function. These new environments have to be more nimble, more agile, more robust and more efficient than the ones of just a few short years ago.

Software-defined, if you will.

Much like planets and stars warp the space around them, these new IT agenda items are bending the fabric of IT thinking.

On a more practical note, it’s nice to see the leap from airy architectural concepts to pragmatic, detailed reference architectures -- hardware, software and methodologies -- that can be put into production today.

Let’s hope we see more along these lines from IT vendors ... a lot more!

At one level, one could simply say that this is the familiar changing-of-the-guard in the IT business: larger companies with familiar business models being overtaken by smaller and more nimble upstarts.

But I think there’s more going on here that meets the eye.

When I talk about the future of the IT industry with my colleagues and customers, there's all sorts of theories about what's happening, and why.

And, yes, I do have my opinions.

Theory #1 — Startups Are Eating The Established Player’s Lunches

There’s something many people find exciting about following the startup scene. Me, not so much. It’s like the lottery: many play, few win.

In my little corner of the IT world (e.g. storage), we’ve seen all manner of new startups in the last few years.

The insider consensus is that supply exceeds demand: the market doesn’t need that many new storage players, so there will be winners and losers.

Some seem to be competing on how fast they can burn cash. Too much money chasing too little opportunity.

It’s interesting to watch the pretty flames from a safe distance, but sooner or later the fuel will run low, and that will be that.

Besides, most of these newer outfits would jump at the chance to sell themselves to a larger, established IT vendor.

Add up all their (estimated) revenues, and it can’t explain things.

Theory #2 — It’s All Going To The Cloud, Or Is It?

There were some wild-eyed zealots just a few years back, prophesying the complete collapse of enterprise IT groups everywhere in the face of the almighty public cloud Armageddon.

There’s no arguing about the remarkable rise of public cloud services like Amazon, Google and others. But my working theory here is that these outfits have grown large by serving needs not easily met by traditional enterprise IT groups.

But I don’t think it’s a zero-sum game.

While the amount of money being spend on public cloud services is growing, so is the amount being spent on in-house IT, hosting, SaaS applications et. al. Smart enterprise IT types know that owning an asset (and utilizing it fully) is far less expensive than on-demand rentals.

Plus, there’s that whole “control” thing :)

So, no, I can’t point to public cloud causing substantial turmoil with the enterprise IT vendors. I just haven’t seen the evidence: factual, anecdotal or otherwise.

Although this might change in the future.

Theory #3 — Enterprise IT Is Getting Commoditized Faster

Now we’re getting closer to the truth. Commoditization is nothing new, but I would argue it’s now occurring at an accelerated rate, and attacking all segments of the market with a vengeance.

Yesterday’s revolutionary feature becomes today’s table stakes before too long. While everyone has their favorites, a server is a server is a server at a certain level. UNIX has been almost completely displaced by enterprise-grade Linux, and open-source tools are slowly seeping in to the heart of IT.

In the storage array world, it’s gotten progressively harder for vendors to claim technical superiority. Most of the products out there today are pretty good, so differentiation becomes more difficult.

Software-defined networking and storage clearly moves the value “up the stack” and away from the underlying devices that support it.

But that doesn’t really explain it all sufficiently.

Theory #4 — Lack Of Agility From The Larger Players

Again, we’re honing in on the truth here. If you’re a larger IT player, you’ve learned to get good at certain things. At the same time, you present a clear “disruption” target for everyone else in the business.

How much do you invest in protecting your turf vs. investing in new business opportunities? Remember: not all your bets will pay off. It’s a delicate balancing act.

Invest too much in defense, and you’re justly perceived as not innovating any more. Invest too much in offense, and shareholders get disappointed that you’re risking the business. Plenty of examples of both in the industry today.

One of the more painful quotes today was “HP has the agility of a bag of cement”. That’s an unfair shot from my perspective. HP has invested in all sorts of interesting things: cloud, Moonshoot, etc. It’s just that none of these has paid off well enough to support its enormous revenue targets.

Clayton Christensen’s innovator’s dilemma is well-understood by most; but that doesn’t make things any easier.

Theory #5 — Enterprise IT Groups Have Become Very Smart Shoppers

I like this theory the best.

I believe the prolonged economic downturn between 2008-2011 caused a structural change in how most enterprises buy IT stuff. Based on my own experience, I can see a marked difference in buying psychology between then and now.

But I can’t prove it.

Evaluations are longer. There are more vendors being looked at. Every transaction is scrutinized and justified. All alternatives are seriously considered. People buy what they need, when they need it — and don’t “stock up” on either hardware or software. Second (and third!) sources are commonplace. Operational expense is often considered more than capital outlays. Negotiations are more frequently brutal.

Vendor “bling” (shiny new features coupled with slick marketing) grows progressively less effective in the face of hard-nosed, pragmatic IT consumers. Good enough is frequently just that: good enough.

It’s a very tough environment if you’re a large IT vendor.

Enterprise IT groups have gotten very smart indeed in extracting the most value from their IT dollar, and they’re getting better at it all the time. Enterprise IT vendors have to become as lean and as mean as possible in this new environment -- and that forces hard choices on vendor strategy.

The Outlook

Size alone is no guarantee of success anymore. At one time, larger vendors could offer one-stop-shopping, and customers found value in that. Now, it’s more about who offers the most bang-for-the-buck.

That being said, I think there’s still a clear opportunity for the larger vendors, but it won’t be easy for them to achieve.

One of the biggest costs in IT today is the operational costs associated with complex, multi-vendor, roll-your-own environments. Witness the success of converged infrastructure (VCE et. al.) and the current interest in hyperconverged infrastructure (EVO: RAIL)

To the extent that larger vendors can “re-productize” their huge array of offerings into larger functional chunks that are easier to understand, deploy, consume, operate, integrate, etc. — they can turn their sprawling portfolios into competitive weapons.

But that entails serious integration investment across multiple product lines, as well as sacrificing existing cash cows as fodder for entirely new “products”.

Give in to external pressure, and you can end up miserable before long. Take a strong, independent stand, and everyone will likely be unhappy with you before long.

So how do you navigate that quintessential win-win?

That’s how I see mentoring: helping good people craft their approach to getting what they want out of their career.

The advantages for those being mentored are substantial: an impartial perspective, good advice, new connections, and honest feedback. But being a mentor for others also has its rewards, often unappreciated by many.

I’ve had the good fortune of being mentored, and being a mentor. I’d recommend both for just about anyone so inclined.

On Being Mentored

Much has been written on this topic. Some of it is good advice, and some of it is completely unrealistic based on my experience.

Not every senior person can be a decent mentor, it demands a specific profile which isn’t exactly common. And not every person who fits the profile has the time or interest in being a mentor.

One frequent error I see is people aiming way up the org chart for a potential mentor, the bigger the title the better. Not always the best approach.

First, this person may not have the ability or the interest to mentor effectively. Second, their world might be so removed from your own that there’s little in the way of shared perspectives. And, finally, very senior people tend to be exceptionally busy, and find it hard to make the time for effective mentoring.

A better approach?

Find someone you respect on multiple levels: what they’ve done, and — more importantly — how they’ve done it. Do they have a well-balanced perspective? Are they empathetic to the situations of others? Do they have clear values which you agree with?

You’re looking for a “personality and values” fit first and foremost. They may not have a title of Executive Chief Lord Of The Universe, but that’s OK.

Better yet if you can find this person outside of your immediate organization. A small bit of separation is useful: close enough that they have some context, but far enough away that they’re not directly involved in the same things you are.

You’re looking for an outside perspective anyway, right?

Some people wait to go look for a mentor when they're facing a tough decision, or in a crisis situation. Not good. Far better if you've established a good mentor relationship ahead of time. That's hard on everyone, including your just-informed mentor.

Plan ahead against the inevitable -- because at some point, we all hit a rough patch.

Preparing For An Effective Mentoring Session

When I’m mentoring someone, we might talk about a lot of things — maybe adding some context and clarity to what’s going on, or commenting on this event or that. I sort of let the discussion wander a bit at the outset, with no real agenda.

But — sooner or later — the discussion boils down helping this person figure out how to get what they want from their career.

This, of course, presumes that the person being mentored has a clear picture of what they want, why they want it — and what they’d be prepared to do to get what they want.

And the vast majority of people haven't clearly thought this through, so that's where we have to start.

I end up spending a lot of time helping people on those basics. Without that clear picture, it’s hard to offer up specific advice on how to help them.

Other than “go figure out what you want”, which isn't very helpful.

Beware Surface Goals

OK, maybe you think you know what you want, but do you really? I say this because I usually have to spend considerable time digging deeper than surface goals as first presented.

Example: “I want to lead a larger organization”. When I ask why, the answer comes out: leaders of larger organizations tend to get more money and get more respect.

Needless to say, these people also work their butts off. It’s not an easy gig. It takes a very specific set of skills: organizational, political, etc. And you won't be hands-on anymore.

So I turn it around on them — so it’s really about more money and more respect — and not about the intrinsic rewards of leading a larger organization? Because if you don’t love that gig, you’re going end up being miserable, and ultimately not being successful.

Maybe we should discuss other potential paths to more money and more respect?

Another example is money. Everyone wants more, right? So many times, a "career" discussion is actually a "I want more money" discussion. Hint: there's more to a career than just money.

So when this comes up (and it always does), I ask — what else is important to you? I often get an “everything else” response: time with family, hobbies, recreation, exercise, mental health, etc. — e.g. having a real life outside of work.

I offer up that — at the end of the day — isn't your goal is to be happy with your life? Money certainly helps, but it’s not the only thing, is it? For example, if you were offered a gig that involved constant travel — and paid well — you wouldn’t be happy, would you?

So -- for these people -- their answer is more along the lines of "making a bit more money without completely disrupting my happy life". Which is a very reasonable perspective.

"I'm Underpaid"

Very often, I encounter people who feel their skills and contributions are undervalued by their employer. They get frustrated and resentful, which doesn’t exactly help them to be effective in their role.

My answer might seem harsh: the best way to determine your true market value is by putting yourself on the market, and seeing what people will pay for your skills. Things are only worth what people are willing to pay for them.

Of course, that decision comes at a non-trivial cost: time and effort, starting over again somewhere else, financial uncertainty, etc. But if you think you might be worth more, that’s how you find out.

There are other examples, but here’s the point: if you’re going to be mentored, the better grip you have on what you want, why you want it, and what you’re prepared to do to get it — the better.

Context Matters

There's a lot more to people than what you see at work. They may be going through a financial crisis, hitting a rough spot in a relationship, having serious health issues, etc. All of that will manifest itself in the persona you see in the office.

While I don't really want to probe into the gory details, I do try to do a quick assessment of what else is going on in their life. Example: if you've just had your first kid and locked yourself into a massive mortgage, maybe it's better to put your head down and grind through things for a while?

On Being A Mentor

I have mentored several dozen people on-and-off during my career. Simply put, I find it very rewarding. Many of my peers, apparently not so much.

I enjoy helping other people when I can. I often learn a lot from the conversations. And it’s a nice break from the hustle-and-bustle of the average work day — taking the time to look at the world through someone else’s eyes.

Keeps it real.

Not everyone’s cup of tea, but if you’re considering it — a few guidelines that might help?

Getting Your Mentor Focus Straight

If you're considering being a mentor, the first task is to get clear on what's going on here.

This isn’t about you, or the company — this is 100% about the person being mentored. If all you do is parrot the company line during a mentoring session, or recycle familiar cliches, it won't be productive. For example, are you willing to honestly admit the the strengths and weaknesses of the organization? Or critique mistakes you've made?

Transparency and honesty is what's needed here.

Example: sometimes the right answer is — yeah, it sucks, it isn’t going to get better anytime soon, so go find a gig elsewhere. Or, perhaps, you're doing this to yourself, so stop. That kind of honesty.

Most of the interaction should be question-based. You ask, they answer, you refine and probe a bit more, until you get to that hot nugget of truth where the issues really might lie. It takes some time, and doesn’t come quickly.

Once I find that nugget, I don’t want to attempt to tell people what to do. That’s not going to work. I do feel that I can share with them the important decisions I’ve made, why I’ve made them, and how it worked out. Including the mistakes.

Frequent Themes

Sometimes people have trouble visualizing what the road forward might look like. So I take them through scenarios: imagine if you did this, here’s what you might expect sequentially going forward, both positive and negative.

One frequent example is someone considering getting their MBA. Sounds great on paper, but is it? This decision requires enormous sacrifices — financial, time and otherwise — to pick up a recognized degree.

The thinking is invest now, benefit later.

But what if those expected benefits don’t materialize? Plenty of MBAs out there who thought their degree was a golden ticket to a successful career, and it didn’t quite work out that way. So I ask them: would you invest in an MBA even if there wasn’t a guarantee of a better career? Would the experience itself be rewarding enough for you?

If the answer is “yes”, you should do it.

Very often, people aren’t exactly clear on what makes them special, unique and valuable. It's simple: we can’t see ourselves as others see us. I always try to make sure I share with them what I see as their strong positives — as well as some areas to work on.

A self-aware person makes better decisions regarding their career.

Just to be clear, this mentoring thing is on them, not you. If they lose interest, etc. — no biggie. You’re just making yourself available as their needs dictate, and your schedule allows.

And, of course, everything must be kept in the strictest confidence, including the existence of the mentoring relationship itself. If you’re tempted to talk to someone about something — just to be helpful — be sure to get explicit clearance from the person being mentored.

A Rewarding Experience

I can recall several occasions in my career where a great conversation with a mentor really helped to set me straight. To each of you (and you know who you are), I will be eternally grateful for the unselfish gift you gave me.

I’d like to think I’m doing a decent job of paying it forward. I’ve had more than a few experiences where the other person had a clear “aha!” moment, and knew what to do going forward.

Sometimes, that’s the highlight of my week :)

Navigating through a career isn’t easy. We all can use impartial and sage guidance from time to time.

A memo from Captain Obvious: flash continues to change how we think about storage.

For me, the party started way back in 2008 when EMC introduced an enterprise-grade flash drive for Symmetrix. Fast forward to 2014, and the party is still rocking strong …

Up to now, most of my customer discussions have been how to best use flash with precision: surgical strikes on key areas where IO response times are a problem. The approach makes a certain sense: flash drives are more expensive than the magnetic spinning variety.

But in the last six months, it’s not unusual to meet a customer that sees themselves going “all flash” either now, or before too long.

The mindset has clearly begun to shift from "use flash to manage performance problems" to "use flash as the default". We're not talking exotic use cases, it's showing up in bread-and-butter enterprise IT settings.

And the motivations are quite interesting.

The Basics Of Flash

If we’re going to over-simplify, it’s all about the need for speed.

Yes, flash capacity is more expensive than disk capacity. But when it comes to performance, it’s an outright bargain. For most IO profiles, flash drives clearly offer more performance bang-for-buck than their mechanical counterparts.

Since most folks think in terms of capacity, there has long been this mindset of using flash selectively where it’s justified.

Flash as a cache to speed up disk drives. Flash as a tier of storage in your hybrid array. All-flash arrays targeted at especially demanding applications.

Since flash has always been considered much more expensive than rotating rust, it’s driven a mindset of scarcity vs. plenty.

Yes, flash prices are coming down quickly — but so are disk drive prices. Dedupe and compression adds additional efficiency with flash, but it’s the same story for disk drives. Bottom line: there will be a substantial cost differential between flash and disk for the foreseeable future.

But the cost delta isn’t what’s being questioned here. At some point, do prices fall to a point where it becomes “worth it” to abandon the storage media mix-and-match philosophy, and simply go all-flash?

The answer seems to be "yes".

A Note Of Reality

Now, to be fair, when I hear a customer talk of going "all-flash”, there are some agreed exceptions.

Few seriously consider flash as cost-effective for general purpose backup and archiving. Sequential data streams may be far better served by scale-out designs using mechanical disks. And there’s plenty of “junk capacity” around the data center: file shares, management environments, etc.

More precisely, by “all-flash” they mean using flash as the default primary storage for all production as well as test/development, e.g. user-facing applications.

And when you listen to their logic, you understand where they’re coming from.

It’s All About Managing Performance

Time is money. Users value consistently snappy performance — just ask anyone with a flash drive in their laptop. Disk defrags are a quaint memory, like dial-up modems. Just try and get anyone to go back to rotating rust.

A lot of upfront work goes into performance sizing, optimizing configurations, leaving headroom, etc. — but things have a way of changing. And the ongoing work -- monitoring, tuning, reconfiguring, etc. -- is non-trivial.

Here’s the net-net: with good all-flash storage, everyone always gets consistently snappy performance, and -- as a result -- there’s almost no need to get involved with storage performance topics. One less thing to worry about.

Blast millions of IOPS consistently at a server farm, and you’re not going to be very concerned about storage performance — either at design time or during ongoing operations. It just isn't that big a deal anymore.

Not to mention, you’ll have deliriously happy — and ostensibly more productive — users.

And that’s worth something.

The Simplicity Of All-Flash

The all-flash IT crowd says the whole provisioning thing gets really simple — there’s no decision to make about where to put a data set: it always goes on the fast stuff.

No managing multiple pools of capacity — it’s all one uniform pool. And no arguing with people about where their data lives.

All storage conversations basically boil down to a capacity discussion — how much do you need? --and performance is simply assumed.

These folks also say that they don’t need to invest in storage performance expertise — either within their own organizations, or by working with their vendors. It’s just not a big area of concern.

Putting on my VMware hat for a brief moment, more than a few customers have raised their hands and asked for an all-flash version of VSAN.

Keep in mind, VSAN customers already value simplicity -- that's what the product is all about -- so this shouldn’t be surprising.

An all-flash VSAN (or any other storage product) is going to be notably simpler than a hybrid approach.

However, regardless of whether you’re talking external or software-based server storage, one thing is clear: the benefits evaporate unless performance is consistently predictable. Unpredictable performance means that we’re back to closely managing things again — which negates some of the value of going all-flash.

Subtle Shifts In The Storage Vendor World Result

It’s interesting to think through what the implications might be for the storage vendor community if this trend continues apace. In many regards, their world gets a bit simpler as well.

Historically, being a storage vendor meant being prepared to have a complex and nuanced performance discussion with each and every customer. That meant investing in smart performance people. And creating smart performance tools. And building large performance testing labs.

You also had to invest in your customers getting smart about performance: white papers, reference architectures, best practices, consulting, etc.

Much less need for all of that in an all-flash world.

But Make No Mistake, The Pendulum Will Swing … Again

You might think that — in an all-flash world, we’re done with being overly concerned with storage performance.

I mean, when you go from a shared storage platform that delivers ~30K IOPS to one that does >1M IOPS, you’d be tempted to think that's that -- time to move on.

Yes and no.

Since the dawn of computers, we’ve consistently managed to consume whatever magical technology advances come our way, and end up demanding even more. And flash storage should be no exception.

Flash storage devices already fall into multiple bands of price/performance/availability, and this differentiation should continue. At the same time, flash is well along its inevitable march closer to the CPU: from IO channel to PCIe bus to motherboard. Memory technologies are evolving as well — already being used as a cache, but growingly attractive as a more persistent storage tiered using clever software.

If you’re a server manufacturer, you’re playing a very demanding game.

There’s a never-ending parade of new technologies you’ve got to rapidly adopt. Customers expect both flawless execution as well as steadily declining prices. And you’ve got some pretty aggressive competitors as well :)

In this category should be software-based storage products (VSAN, Nexenta, ScaleIO, Scality, Maxta, etc.) as well as HDFS (basically a storage system for big data) as well as the newer databases, etc.

Any software that stores and retrieves data, provides persistence and protection, etc. would be part of this discussion.

Indeed, even IDC -- the scorekeeper of the storage biz -- has introduced a new tracking category: software-defined storage platform, or SDS-P. While this IDC-named bucket is only responsible for a small portion of the market today, it's growing rapidly.

An obvious quibble: IDC's taxonomy is imprecise -- they are counting any storage product that is software-based, regardless of whether it supports APIs, can dynamically compose services, etc. Their new category would be far better named as "software-based storage platforms". A smaller quibble: NFS is included, HDFS is not? But it's a start ...

Regardless, it looks like server vendors are seeing the demand for storage-optimized servers start to grow.

Where We Came From

For the last two decades or so, most server designs seemed to assume that the majority of storage would be external: SAN or NAS.

You’d see very little enclosure room for disk devices. Limited power and cooling for all those spindles. IO controllers and HBAs were presumed to live on a PCIe bus, and you’d need several — so not much in the way of onboard IO capabilities.

Perhaps the zenith of that philosophy was the popularity of blade servers — a form-factor optimized for external storage. Independently scale compute/memory and storage — what could be slicker?

And then the world changed.

Things Are Moving

When Intel announces a new processor (such as the new Xeon E5-2600 v3, code-name “Grantley” — this one with 18 cores per socket), it drives a refresh cycle in the server vendor community. Not to mention DDR4 and 12Gb SAS now being table-stakes.

To take a quick look at some of the recent server product announcements, it’s pretty clear there’s a new design point being considered.

A prime example: the new Dell R730xd. In a slim 2U, there’s room for up to 24 2.5” drives. Look deeper, and you’ll see even more storage-centric features. Two dedicated slots for 12Gb SAS controllers. Support for the new NVMe standard for up to four PCIe flash devices.

Someone had software-based storage in mind.

A similar picture can be seen from the new IBM x3650m5: 24 drives, two dedicated IO controller slots, etc. The same picture can be seen with the new Lenovo RD650, as well as the HP Gen9 DL380. And a few others as well.

Let’s do the math.

Using 24 1.2 TB 10K Seagate 2.5” drives, you’d get ~29TB raw, ~14TB protected (using two copies of data) behind a two-socket server. That’s very reasonable for many use cases. Yes, those are pricey drives (today), but that’s not going to last for long.

If I were to consider VSAN, an 8-node typical server/storage cluster would support over 100TB of protected, great-performing storage. Much more than enough for most use cases. All in 16U of rack space.

Gives you a sense of where things are going, doesn’t it?

What About Blades?

More than a few IT shops have committed to blade architectures for good reason. At a certain scale, there’s an attractiveness to being able to scale compute/memory and external storage independently.

But given (a) the rising popularity of software-based storage solutions, and (b) the attractive density and economics in some of the newer server designs, the economic pivot points are starting to show signs of change.

If you’ve made a big commitment to blades, maybe it’s worth your time to re-run the numbers.

Clever server vendors will figure out how to cram even more small drives into 2U and 4U enclosures -- and manage power, cooling, vibration, etc.

Storage IO controllers will get simpler, more powerful, and start to lose unneeded features like on-board cache and RAID. They won't mask key drive and bus status information. Maybe, just maybe, they'll end up on the motherboard.

Flash devices will come off the PCIe bus, and end up on the motherboard as well. Hopefully they will learn to report errors in a standard way. And while we're pining away for standards, maybe we could get to some sort of standard with enclosure management? One can hope :)

No surprise, I’m out in gorgeous SF for the annual gathering of the vFaithful — VMworld 2014.

As before, it’s quite an experience, even for a jaded event veteran like me. Like a desert blooming after a rainstorm, Moscone Center is transformed into a fascinating technology experience, and — then — we wait until next year for it to happen all over again.

I’m not capable of giving you a detailed report on everything that happened at VMworld — there’s way too much to cover.

Instead, I thought I’d share with you what stood out for me personally.

What A Great Place

Many of you have been to Las Vegas for big industry events. Let me just say it — San Francisco has its special charms: food, culture, people, weather, scenery, etc. I’m always reminded by what a nice place it can be if you just let it wash over you.

Unfortunately, there’s not a lot of time to be a tourist.

For most people, VMworld is an all-day slog from Sunday through Wednesday, and perhaps Thursday. If you’re involved in IT infrastructure, this is the big show of the year.

Want to meet all the storage vendors, large and small? They’re all here. As are the networking vendors, server vendors, etc. And no shortage of smaller vendors with cool niche products.

Given the technical nature of the show, it's pretty easy to pierce the marketing veil and talk to someone who knows their stuff.

I like that.

The event can take on the surreal nature of a circus occasionally. Smaller vendors in particular pull out all the stops to get noticed, and there’s no shortage of attention-getting theatrics at hand.

The People

This is the Big Event for many people who work in IT infrastructure.

They come to learn about new technologies, brush up on their knowledge, hone their skills, and network with others who do pretty much the same thing for a living. My informal poll revealed that VMworld is most likely the only such event they’ll travel to all year.

These people can get deep quickly, because they use the products every working day.

Casual conversations quickly enter a black hole of specific version numbers, modules and behaviors from which there is no easy return.

But there’s more to it than that.

These people work for companies who have bet big on VMware technology. They’ve made a substantial personal career commitment as well. Many of them are part of various technical communities, where they help their peers.

The depth of engagement and passion you’ll see can be breathtaking. And it’s surprisingly easy to strike up a conversation with just about anyone you’ll meet.

Do you remember the old joke about eggs and bacon? The chicken was involved, but the pig was committed? These people are most definitely committed.

VMworld has historically been where VMware (and other vendors) have made big announcements, so there’s usually plenty of news and new technologies to ingest, debate and evaluate. Geek food for the intellectually hungry.

Sometimes you can eat too much. Attendees put in long, hard hours at this event. After three technical deep dives and a HoL (hands-on lab) or two, people can look pretty dazed towards the end of the day.

And, despite evening entertainment, they’re up bright and early for that 8AM session. Nobody wants to miss out on the opportunity to cram in just one more session.

There are so many rich and diverse choices of things to do that figuring out your schedule is itself a major undertaking. And, for people like me, there were some tough decisions about where to spend my scant free time.

The Vibe

The theme of the show was “No Limits”, which — as event themes go — pretty much captured the mood in the crowd. People were generally ecstatic what they’d been able to achieve with the technology, and saw that they could do even more with what’s becoming available.

Not that there weren’t criticisms and “helpful suggestions” aplenty on what VMware could be doing better/faster. As VMware’s headquarters is just down the road in Palo Alto, just about every VMwarer who could make the event was there to listen and learn from customers and partners, including many of my colleagues. It’s all-hands-on-deck.

Nothing like a development engineer getting direct feedback from someone who’s using their product.

The Tech

As a long-term industry watcher, I do have to admit my employer is doing an exceptional job of staying ahead of industry trends. The real religion at VMware is “help customers” — even if that means setting off in somewhat new directions.

We all know that converged infrastructure is rapidly growing in popularity. In exchange for constrained choices, the customer gets a better (and more predictable!) experience. First VCE and its competitors, and then single-module vendors such as SimpliVity and Nutanix, among others.

Based on the observed reaction at the show, there’s no doubt in my mind that VMware will be a major player in this appliance-ized category before long.

EVO:RAIL is best described as a re-integration of VMware technology around known hardware and an extremely simplified experience. Server partners build specific hardware for EVO:RAIL, add EVO software, and take the resulting product to market.

Literally overnight, several server vendors now have a direct answer for things like Nutanix. You need to see the installation and management demo vids to get a real appreciation on just how simple the team has made the experience. Hey, even I could run this stuff. And don’t underestimate the inherent appeal of an all-VMware environment.

The hardware examples on display at the EVO Zone were very cool — a 2U crammed with 4 server blades and 16 disk slots. If the sales and support model executes as expected, I think it’s going to be a big winner.

And there's much more to discuss when EVO:RACK becomes available.

On To Storage — vVols

As vVols are now in beta, you saw a number of vendors demoing and running sessions on the new capabilities. Every session I saw on the topic was packed, so there’s definitely strong interest. I can only hope that — as before — strong interest precedes strong adoption.

After a few conversations with folks, the reactions to vVols split clearly into two camps.

For those organizations who had converged roles (storage, server, network, virtualization, etc.) they saw themselves being able to adopt the vVol model reasonably quickly — and planned to go in that direction.

However, those IT teams that segmented along classic lines clearly could see the potential difficulty from an organizational and operational perspective.

Fortunately, there seems to be no shortage of IT groups who have either adopted the converged model, or are well along in the process.

VSAN?

On one hand, VSAN was old news at this year’s VMworld. Everyone now knows about it, etc. — it’s not the new, shiny thing. That being said, for a storage product that went GA last March, there was a visible groundswell of undeniably solid support.

We ran a number of user feedback sessions, and the results were pretty consistent. Great product. Does what it says. Radically simple user experience. Surprising performance. Amazingly robust and solid. Planning to use much more of it, etc.

And pay close attention to your hardware choices, especially the IO controllers :)

On top of that, we ran a number of roadmap sessions, and people could clearly see what would be available before too long. The development team is moving fast; building on its proven architecture. All good.

Final Thoughts

Over a hundred years ago, the First Transcontinental Railroad transformed the US economy, and created an infrastructure that opened up a vast new territory: the US west.

Thanks to labors of tens of thousands of hard working people, the country was changed for the better.

I can't help thinking about the parallels when I look at the people who attend VMworld.

Tens of thousands of hard-working IT professionals, just quietly going about their daily business of creating infrastructure that transforms the world around them.

Operational cost for storage is less about capacity, and more about distinct applications: how many, and how demanding? A thousand different applications that use a gigabyte each will take far more operational effort than a single application that uses a terabyte.

Another useful exercise: how many application workloads were you responsible for five years ago? How many today?

Don’t forget end user computing (VDI), test and development, etc. The number of virtual machines you’re running is a good proxy. Notice how fast it’s growing.

#3 -- Automation is the next frontier in operational efficiency.All will succumb, it’s only a matter of time. The broader the scope of process automation, the more efficient it can potentially be — the familiar bottoms-up vs. tops-down automation debate.

That implies that storage automation won’t productively be considered in isolation; it will be thought of in concert with adjacent IT disciplines: compute, networking, security, etc.

So, the primary problem we’re trying to solve here is rather simple to state: the need to create a far more efficient, highly-automated storage operational model — converged with other IT disciplines — in the face of rapidly growing application landscapes, and rapidly growing capacity.

Current Industry Practice Isn’t Exactly Helpful

If that’s the demand side of the equation, what does the supply side look like? Not encouraging, I’d offer.

#1 -- Array proliferation. Storage vendors with broader portfolios have been promoting the idea of “right workload, right architecture” for a while.

Use an all-flash array for your latency-sensitive databases. Use object storage for a managed content repository. Use a filer for user data stores on VDI. Use an enterprise array to get rich data services. On and on.

All of these storage arrays behave and manage very differently. Not exactly a boon to creating an efficient operational model, is it?

#2 -- Data service proliferation. One could argue that the value in storage is shifting to data services: snaps, replication, tiering, encryption, deduplication, compliance, caching, etc. The vast majority of these data services are tightly bound to specific external storage arrays.

Needless to say, all of these data services behave (and are managed) quite differently. Another structural barrier to creating an efficient operational model for storage.

#3 -- Application ignorance. If storage management complexity is being driven by application proliferation, shouldn’t storage “understand” individual applications — their boundaries, what they individually need, etc.?Historically, that hasn’t been the case. Storage arrays generally offer up large, homogeneous and static storage containers, and are largely unaware of individual application requirements — be they static or dynamic.There’s no direct logical coupling between what an application owner wants, and what the storage team provides — unless you schedule a meeting to discuss it. Maybe a few meetings. And that’s not operationally efficient.

#4 -- Storage now includes servers. A final observation: by now, everyone generally accepts that flash is an important part of the storage landscape. Flash is all about low latencies, and that means it is inevitably moving to the server: first on the PCIe bus, and then to the motherboard where it can be closer to the applications it serves. It’s simple physics.

“Storage” (at least for important, performance-sensitive applications) now includes a healthy and critical server-based component, which most certainly blurs the boundaries between what is storage (an external array) and what is server.

All of this — array diversity, data service diversity, no visibility into applications, flash moving to the server — creates formidable obstacles to envisioning a vastly improved storage operational model.

This, then, is the case for software-defined storage: the growing need to establish a far better, application-centric model for storage. Without such a capability, storage will continue to fall behind compute (and networking!) in terms of operational effectiveness.

Sooner or later, any decently-sized landscape will hit the storage wall. It's only a matter of time.

Now We Need A Definition, Don’t We?

Regular readers will know that I locked in on a precise and unambiguous definition a while back. Since then, I believe in it even more strongly than at the outset — it has survived rigorous debate and formidable challenges from people smarter than I.

There is much implied by this terse definition, once you start unpacking it.

A good starting point would be the notion of “application policy” — a manifest of what the application needs at a given point in time: performance, availability, capacity, protection, etc. Ideally, a single such application-specific policy could be used to drive compute, networking, security, etc. as well, and not be just about storage requirements.

This is a key abstraction. Here is what I — the application owner — want. Now go figure out how to do it, and don’t bother me with the implementation details. Don’t talk to me about RAID groups, copy-on-first-write snaps, async replication, caching block sizes or any of that, please.

Next, consider the word “compose”. That means that I — as an application owner — don’t have to make a hard choice between 3 or 4 standardized options established eons ago by the storage team. Maybe I need stellar performance, but don’t care much about data protection. Or access rates are low, but the data always has to be there when requested. Or some other combination from literally an infinite number of possibilities.

From there, consider the word “dynamic”. As an application owner, I should be free to re-specify my requirements: quickly, easily and with a minimum of fuss. Maybe that’s on a long-term cadence, e.g. months. Maybe I need a change *right now*.

Digging Deeper

That’s what you get when you take the definition's words at face value. Look a little deeper, and much more is implied.

This particular definition implies that storage is entirely under programmatic control, presumably using an API. Anything a human can do can (and should) be done under software control. That’s what good operational models are all about — reducing the need for human involvement.

This is not about pushing paper faster.

It also implies that storage is provided in application-specific containers, ones where service levels can be adjusted without impacting its neighbors — or having to do a manual and hence cumbersome move to another location.

This view also demands that policies can be set — and easily monitored for compliance. Here is what I have asked for — am I getting it? If not, why not — and what can be done about it?

There is also a strong implication — but not a mandate — that storage data services will behave similarly, and independently of what back-end device is actually storing the data: a decoupling, if you will.

A snap should be a snap should be a snap — unless you’re after something really special. Ditto for replication, dedupe, tiering, caching, encryption, etc. That strongly points to a model where data services run in server software, and aren’t tied to a specific vendor’s hardware array.

Finally, there’s a strong desire — but not a mandate — that this software-defined operational model for storage be achieved without having to lock in on a specific storage architecture, or a specific storage vendor’s offering.

If we’re all headed to something that sounds vaguely cool — like software-defined storage — it makes sense to ask pragmatic questions, e.g. “where are we going, and why are we going there?”

We’re driving to a vastly improved operational model for storage — as part of a larger journey to software-defined and SDDC.

The reason is clear: more IT organizations will inevitably hit the ‘storage wall’ when they realize their current storage operational model is no longer sufficient, and start to cause real organizational pain. More applications, more data, more desire for better automation — those will be the forcing functions.

Some are already there today. More will be there before long.

And a very few can look ahead, and start planning for the likely future today.

As a younger version of my present self, I was supremely confident in what I knew to be true — especially when it came to all things enterprise IT tech.

There was generally accepted wisdom about many things; to disagree with certain tenets made people question your expertise.

But with age and experience, I realized that many of the “obvious truths” on which I had built my belief system were quickly becoming fallacies.

Time for a new religion?

No, not really — more of a never-ending appreciation of how the world really works. That means modifying old frameworks, and creating new ones.

I often meet younger versions of myself, also supremely confident in their current beliefs. I don’t try and challenge them directly, but I do make the point that “obvious truths” in this business have a very short shelf-life indeed.

Enterprise IT Matters

Since I’ve spent most of my career working for IT vendors, it was natural for me to assume early on that I was selling to the most important people on earth: enterprise IT organizations.

Imagine my disappointment when I started to realize that — in many situations — IT is just another supporting function, like HR, finance, marketing, facilities, etc. And usually populated by mere mortals.

Sure, it’s important in that IT has to get the job done at reasonable cost, but that’s usually about it. IT is special, just like everyone else is special.

You can tell a lot from an org chart. In most companies, the CIO (or director of IT) reports to either the CFO or the person in charge of operations. Yes, it's important that the IT stuff works, but hardly strategic.

No wonder IT is one of the first organ donors when it’s time for cutbacks.

The important exception — and hope for optimism — is when a business enters the new world of digital business models, and is thus coerced to view IT as a competitive differentiator.

And then enterprise IT matters very much indeed :)

The Best Technology Wins

We geeks love to debate, sometimes for the love of debating itself. I clearly remember many heated discussions about the superiority of this new technology or that one as compared to alternatives, usually focusing on arcane implementation details.

The history of my poor choices is considerable …

I thought token ring was “better” than nasty ethernet with all of its collisions. I thought the OSI networking model was inherently superior to this academic TCP stuff. I believed the Motorola microprocessor architecture was far better at UNIX than anything Intel could muster.

I was so wrong so often because I was focusing on the wrong things.

The best technology is the one that is widely adopted.

That, in turn, is driven by a wonderfully complex equation revolving around ecosystem, competitive differentiation, ease of adoption, and — ultimately — the importance of the problem being addressed by the new technology.

Now I look at new technologies through a very jaundiced — and pragmatic — eye. And I’ve gotten much better at picking winners and losers.

Nobody Ever Got Fired For Buying From A Big Vendor

Not exactly. Most often, people get fired for failing to deliver. Sometimes that means going with a big vendor, sometimes it means something else. Results matter.

I’ve mostly worked for somewhat large IT vendors. I used to think that — as a category — they had inherent advantages over their smaller counterparts. For one thing, they can bring enormous resources to bear in support of their customers — an extremely positive attribute. On the other hand, with size comes the challenge of battling entropy and inertia as markets shift.

I can list several big-name IT companies that — at one time — I thought they had it all going on: IBM, DEC, HP, Sun, Microsoft, Cisco, et. al. Size appears to be no guarantee of long-term success.

My new view: good companies are just that — good companies, regardless of size. They excel at what they do, and — if there’s a requirements fit — customers do well with them.

And the best continually reinvent themselves for the modern world.

Enterprise IT People Want To Make A Difference

I have one of those “go big or go home” personalities. If you’re going to do something, make it count. I like taking intelligent risks. Life isn’t a dress rehearsal.

Naively, I thought most folks in the IT business were the same as I. Wrong. Historically, risk-taking hasn’t been exactly rewarded in enterprise IT settings -- it's a trait that's de-selected for in career paths. Safe and steady is preferred.

Until the status quo isn't working, and then the game changes considerably.

But there’s more. I was very disappointed to learn that there is no shortage of people who are happy with just getting by, punching the clock and working for the weekend. The status quo is just fine with them, thank you.

I’m not judging here, just stating the facts.

The good news: I meet plenty of bright, passionate and motivated people who work in IT. “Good enough” isn’t for them.

They want to be part of the change, and not business as usual. As a group, I admire them greatly.

Open Standards Matter

Early on in my career, I became embroiled in various industry standard efforts. Progress made continental drift look speedy.

While you can still find folks in enterprise IT who get very passionate about vendor-neutral industry standards, far more are pragmatic about the topic: just give them realistic options and the flexibility to choose.

Consider my car. I can change many aspects of it (tires, engine, electronics, etc.) thanks to interoperability and de-facto standards. I have yet to take advantage of many of these options, though, preferring to stick with what the manufacturer supplies.

But it’s nice to know I can if I need to.

Formal industry standards have their role, but they’re not a panacea. In many cases, they hamper innovation instead of accelerating it, e.g. we're all waiting for the standard to be approved before moving ahead.

As the world moves to software-defined infrastructure, programmability matters. APIs are rapidly proliferating and evolving, with few formal standards in sight. Functionality matters more. As it should be.

Doing It In Hardware Is Better

Early on in my career, I was seriously impressed by what good hardware engineers could do: FPGAs, ASICs and all of that. The resulting products were faster, better and cheaper as a result.

Why do things in software when you could spend a little more and do them in hardware?

My, how the world has changed. Merchant silicon gets better as predictably as death and taxes. Implementing functions in software allows more functionality, delivered faster to the customer — and at a lower cost.

As a twenty-year storage guy, that was a difficult bit of religion to revisit.

Many of my brethren in the storage industry will still argue that purpose-built hardware has all sorts of important advantages over software running on standard servers.

But their case will be harder to defend with every passing year.

Pat Gelsinger has this popular mantra about “being on the right side of the technology curve”. His point is that technological trends take a while to evolve -- but they inevitably do evolve in predictable ways.

And that’s a useful part of any belief system.

The Thoughtful Pragmatist

The notion of "religion" -- as it applies to enterprise IT topics -- is nothing more than a convenience to avoid having to endlessly revisit familiar topics. Claiming "everyone knows" shortens the discussion, and allows progress to be made.

But these are not ordinary times. The more I go back and question my long-held beliefs, the more I gain a deeper understanding of just how much the world has changed, and how little I truly understand.

I do admit I had a moment of temptation, but then realized that I don’t like demeaning titles, and assume the same for most of you as well.

But I will ‘fess up to a certain level of frustration results when discussing advanced IT concepts with my colleagues and customers.

The natural temptation is to put whatever’s being discussed on the operating table, and surgically dismembering a big concept into constituent technologies, which are then individually criticized. Frequently, the forest is lost in a critique about individual components of the trees.

Can I attempt a big picture?

IT consumption continues to balloon exponentially with no end in sight. Between commodity hardware and open source, the traditional ingredients are predictably getting cheaper and cheaper — but not outracing demand, it seems.

Which leaves us with a renewed and intense focus on operational efficiency: primarily the effort required to get something done. While new shiny technologies are always interesting, the new lens should be “how does this new thing enable a new operational model?”

Because how we do things is becoming much more important than what we use to do them.

Examples Abound

My first in-your-face exposure to this viewpoint came in 2009, when I started to become enamored with All Things Cloud. Working for a technology company, of course “cloud” had to be all about shiny new technology.

It didn’t take long before I realized the fallacy of this perspective. After all, much of the technology was the same as before; what had really changed was how it was being used.

When I came to understand “cloud” as a new operational model, it all made perfect sense. Given a minimum scale, many IT organizations could choose to invest in the new model, or simply consume it from an external provider who had made the required investment in a cloud operational model.

Today, when we talk about private clouds, hybrid clouds, etc. in the enterprise — the single biggest barrier to adoption is the daunting challenge of evolving a traditional operational model to an entirely new one. It isn’t really about the technology.

Fast forward a few years later, and I’m now immersed in All Things Big Data. It would be easy to make it all about Hadoop, HDFS, YARN, etc. — but I was slightly smarter the second time around.

I quickly realized that — at its core — big data was a new operational model for gaining insight and making decisions around available data, supported by a handful of newer technologies.

Fail to evolve your operational model, and you’re simply using new tools to do old things.

I still retain a strong interest in how cloud interacts with enterprise IT models, and am starting to get more intrigued in topics like next-gen security, enterprise mobility and similar.

Each implies a radically new operational model to be fully appreciated, and that’s hard pill to swallow for a lot of people.

A Software-Defined Storage Example

I’m fortunate that I get to present VMware’s ideas around software-defined storage on a regular basis. And, no surprise, I’ve found that there’s a right way — and a wrong way — to do this.

The not-so-good way: start bottoms-up with individual technologies, and (hopefully) end up with a picture of what the new operational model looks like. Save the most terrifying bit for last.

That is, if I ever make it that far.

Because everyone in the room will either (a) pepper me with dozens technical questions, or (b) zone out on their mobile device. And, if I’m fortunate to make it to the end in the allotted time, the inevitable reaction: “that’s not how we do things today!”

The better way: start with making the case for a vastly better operational model. Show how software-defined storage makes that achievable. And then move to the supporting technologies — today and tomorrow.

Put the overarching goal first; then show how it’s done.

I’m only a little embarrassed that it took me a while to figure that out — again.

If you work in enterprise IT, you’re staring at the very face of change and uncertainty — not only for your own career, but for your colleagues as well. Best to not deal with it — at least for today.

If you work for a vendor, you usually prefer sticking to happy-face topics that don’t cause discomfort and are perceived as a barrier to adopting whatever it is you’re selling today.

And there are plenty of product propositions in IT that don’t require a change in your existing operational model: plug it in, do things as you’ve always done them — but don’t expect anything more than incremental benefits.

The Big Kahuna

When it comes to enterprise IT, the 800 pound gorilla looming on the horizon is policy-based IT automation — using application-centric policies to provision and monitor all IT service delivery. It implies a radically different — and more efficient — IT operational than is the norm. And this will be its largest barrier to adoption.

Take a quick look at an average IT org chart. Fully considered, virtually all the familiar IT disciplines will need to embrace a new way of doing things to gain the required operational efficiency. Architects, operations, business ops — no stone will be left unturned before long.

Once again, the technology will be here far in advance of most organization’s willingness to organize around it.

Big Changes, Big Benefits

I have witnessed many scenarios where a strong leader came into an environment, did an assessment, and immediately started working towards a new operational model. The processes changed, the roles changed, the tools changed — and, often, some of the people had to change as well.

The change wasn’t immediate, but it was patiently persistent. Before long, there was an entirely new way of doing things.

That’s what good leaders do.

Because when it comes to any organizational function: IT, marketing, sales, engineering, customer service, etc. — you’re only as good as your operational model.

Many of my VMware colleagues and I are seeing the same thing, time and time again.

We’re doing a general update on things VMware — a smorgasbord of different topics. We get to the segment on vCHS — VMware’s vCloud Hybrid Service. Frequently, they either haven’t heard about it, or don’t know much about it.

That's an opportunity waiting to happen.

Because within 5 minutes, they’ll quickly grasp how a hybrid cloud is different, and how they might use it. Heads nod, light bulbs go off, the discussion gets animated — all that.

Given the inherent skepticism of many enterprise IT groups towards All Things Cloud, you might not expect that sort of positive reaction. But true hybrid clouds aren’t like other public cloud services — and that’s what makes them so darn appealing to so many enterprise IT groups.

Understanding Hybrid Clouds

The vast majority of public cloud services are built from a self-contained perspective. They have their own suite of services, their own management tools, their own processes, their own behaviors, etc.

We’ve seen enormous differentiation and maturation over the last several years: and there are many good public cloud services out there to choose from.

But no two are identical. And — generally speaking — they’re not operationally compatible with what’s being already being done within the data center.

The rationale behind vCHS is different — it is designed and marketed as an operationally-compatible extension to an existing vSphere environment — an add-on, if you will. It prides itself as having many of the same services, tools, processes and behaviors that you’d see in an on-premise vSphere landscape.

As a result, most anything you can do with enterprise vSphere you can do on vCHS — the same way, often with the same tools, and expect the same results.

If you're running vSphere, that's a big deal.

The resulting reaction is rather binary, depending on audience. If you’re not at ground-zero in an enterprise IT setting (say you’re an analyst or clouderati or similar), you don’t grasp the importance of that approach.

But for real-world enterprise IT practitioners, they get it — and get it big.

An Analogy?

We’re all familiar with how organizations use a mix of employees and non-employees: temps, contractors, consultants, etc. When there’s more work than hands, and the case can’t be made for adding full-time heads, we go looking for temporary outside help.

But we also know that bringing in outside help has its own challenges. People from the outside might be unfamiliar with how things actually get done. Language differences might present additional challenges. And you won’t actually know how people perform until you see them in action.

So what do we do?

We tend to re-hire outside help who have a track record and experience working within our organizations. If I’m being cynical: we prefer the devil we know vs. the one we don’t. At the very least, these re-hires are “operationally compatible” with the organization.

Thinking Of A Hybrid Cloud As “Outside Help”

Let’s say you work in enterprise IT, and a new application project comes your way.

If it’s like most, there’s a great deal of uncertainty when it comes to infrastructure requirements. No one is quite sure what the precise requirements will be, or when exactly they might materialize.

Heck, maybe there’s doubt as to whether the project will be run to completion, or be “de-prioritized” (or perhaps cancelled) down the road.

Having to lay out a bunch of capital (and labor) to stand up new infrastructure isn’t an easy decision. The expense needs to be justified, configurations created, orders placed and tracked, shipments received, equipment configured and validated, etc.

It’s a classic damned-if-you-do, damned-if-you-don’t proposition.

If you’re behind the application team’s curve, you get justifiably yelled at, as infrastructure is holding up progress. If you’re ahead of their curve, the investment will likely stand idle, waiting for work to do. A different kind of yelling results, from a different source.

Wouldn’t it be great to get some great “outside help”?

You’d go looking for a public cloud service, and — ideally — it would be easy to get on, and easy to get off. “Easy to get on” is shorthand for a long list of desirable attributes: meets requirements, reasonable costs — and (critically) operates and behaves in the same manner as your in-house environment: nothing new to learn, no conversions or migrations, straightforward to set up and integrate, etc.

One that’s operationally compatible with your existing environment.

Down the road, when the new application is up and running (and well-understood!), you now have attractive options: stay with the external service if it meets your needs, or set up additional internal capacity if that’s what you want. The timing is entirely up to you.

Moving your applications back in-house: apps, data, processes, etc. — all straightforward, as there’s a great deal of compatibility between the existing external environment and the internal one.

Easy to get on, easy to get off.

Just to be absolutely clear, you *could* consider using other (operationally incompatible) public cloud services, but it would be an entirely different proposition.

Just figuring out the costs and effort associated to determine if these services were “easy to get on” and “easy to get off” would take extensive research, experimentation, etc.

And you're looking for good outside help, not a science project.

The Power Of Operational Compatibility

Many folks in the broader IT industry think of enterprise IT as a collection of technologies, which is perhaps true at some level. A more productive view emerges when one considers enterprise IT as a collection of people and processes.

Any technology that is inherently compatible with existing people, skills, roles, workflows, etc. will be far more attractive than something that isn’t.

Investing in new people, skills, roles, processes, etc. is always an expensive proposition, and — for so many IT organizations, it’s usually a zero-sum game: investing in the new thing usually means de-investing in the other things.

You don’t have to go far to find many arguably good technologies that withered on the vine, simply because they demanded a significantly new operational model. And public cloud is turning out to be one of those in so many cases.

Back To vCHS

From what I can gather, the service has become very popular, and showing strong growth rates. Trial runs are becoming substantial commitments.

Of course, there’s still a roadmap of service enhancements to be delivered by the vCHS team, but there’s apparently more than enough there to motivate a broad range of IT shops to quickly adopt it.

While I could go on about the management model, the network model, application compatibility, tool compatibility, security model, etc. — I could summarize all of that with a single phrase: it’s operationally compatible with existing vSphere environments. It presents as a simple extension of the existing environment.

That means that there’s somewhere around ~500,000 IT shops around the globe who could potentially view vCHS as “easy to get on, easy to get off”.

Several years ago, it became clear to me that the next aspirational model for enterprise IT was “IT as a Service”, or ITaaS.

At its core was a simple yet powerful idea: that the core IT operational model should be refashioned around the convenient consumption of IT services.

Under the ITaaS model, most everything IT does is now presented as a variable service, marketed to users, with supply driven by the resulting demand.

IT becomes the internal service provider of choice.

Now, several years later, that once-controversial idea has clearly grown deep roots, with many examples of progressive IT organizations embracing this perspective. Some have made the transition, some are mid-journey, others have yet to begin. The IT world has moved forward.

So, it’s fair to ask — what might come next? I have a strong suspicion as to what the next operational model will be.

Automation: The Ultimate IT Productivity Lever

"Give me a lever long enough, and a fulcrum one which to place it, and I will move the world" -- Archimedes

When it comes to continually improving IT productivity, automation is that lever. It's the gift that keeps on giving when it comes to IT outcomes. Progressively improved automation means progressively improved capex and opex efficiency, fewer errors, more responsive reactions — done right, everything gets better and better.

It’s not just an IT thing: you’ll seem the same continuing automation investment patterns in manufacturing, logistics, consumer marketing — any endeavor where core processes are important.

Broadly speaking, there are two approaches to how one goes about automating IT. Many think in terms of bottoms-up: take individual, domain-specific repetitive tasks, and automate them — perhaps in the form of a script, or similar.

The results are incremental, not transformational.

During the early days of telephony, switchboard operator productivity was limited by the reach of the operator’s arms. Someone came up with the idea of putting wheels on the chairs. Clever, but only modest productivity gains resulted — what was needed was a re-thinking of the problem at hand.

We’ve got the same situation in IT automation: we’re not after mere incremental improvements, what we really want is a sequence of order-of-magnitude improvements. And to do that, we need to think top-down vs. bottom-up.

Starting At The Top

Since IT is all about application delivery, applications logically become the top of the stack. Approached that way, automation becomes about meeting the needs of the application, expressed in a manifest that we refer to here as “policy”.

We need to be specific here, as the notion of “policy” is so broad it can conceivably be applied almost anywhere in the IT stack, e.g. what rebuild approach do you want to use for this specific disk drive?

Indeed, listen to most IT vendors and you’ll hear the word “policy” used liberally. To be clear, policies can nest — with higher-level policies invoking lower-level ones.

For this conversation, however, we’re specifically referring to top-level policies associated with groups of applications.

The Big Ideas Behind (Application) Policy

The core idea behind “policy” is simple: policies express desired outcomes, and not detailed specifications for achieving that outcome. Policies are a powerful abstraction that has the potential to dramatically simplify many aspects of IT operations.

In addition to being a convenient shorthand to expressing requirements, policies are also an effective construct to manage change. When application requirements shift — as they often do — a new policy is applied, which results in the required changes being cascaded through the IT infrastructure. Or perhaps model what a change in policy might do.

Finally, compliance checking — at a high level — becomes conceptually simple. From controlling updates to monitoring service delivery: here is what the policy specifies — is it being done? And if not, what is needed to bring things into compliance?

You end up with a nice, unambiguous closed-loop system.

Stepping outside of IT for a moment, we’ve all probably had personal experience in organizational policies being handed down from above: travel policies, hiring policies, etc. Not a new idea.

The ones that work well seem to be the ones that outline broad objectives and provide guidelines or suggestions. The ones that seem to cause friction are the ones that are overly specific and detailed, and hence constraining.

Simple Example #1

Let’s take an ordinary provisioning example of an application. At one level, you can think of a policy as a laundry list of resources and services required: this much compute, memory, storage, bandwidth, these data protection services, this much security, etc.

So far, so good. Our notion of policy is focused more on what’s needed, rather than how it’s actually done. But, since we’re presumably working against a shared pool of resources, we have to go a bit further, and prioritize how important this request might be vs. all other potential requests.

Let’s arbitrarily designate this particular application as “business support”. It’s somewhat important (isn’t everything?) -- but not as important as either mission-critical nor business-critical applications.

It needs reasonable performance, but not at the expense of more important applications. It needs a modicum of data protection and resiliency, but can’t justify anything much more than the basics. It has no special security or compliance requirements, other than the baseline for internal applications.

The average large enterprise might have hundreds (or perhaps many thousands) of applications that fall into this category.

Under normal conditions, all requests using this policy are granted (and ideally paid for) as you’d expect. But what if resources become constrained?

Yes, your "business support" application will get the requested vCPUs and memory, but — if things get tight — it may not get what you wanted, perhaps temporarily. Here’s the storage requested, but if we come up short, your application may be moved to cheaper/slower stuff and/or we’ll turn on dedupe. Here’s the network connectivity you requested for your app, but … you get the idea.

Our expanded notion of application policy not only can be used to specify what’s required, but also how to prioritize this request against other competing requests for the same shared resources. Why? Resource constraints are a fact of life in any efficiently-run shared resource (or cloud, if you prefer).

Let’s take our idea a bit further. The other scarce resource we can prioritize using this scheme is “IT admin attention”. Since our business support application isn’t as critical as others, that implies that any errors or alarms associated with it aren’t as critical either.

What about the final situation — a “special event”, such as hardware failure or software upgrade? No surprise — lower priority.

Just to summarize, our notion of application policy not only addressed the resources and services desired at provisioning time, but also gave guidance on how to prioritize these requests, how closely it should be monitored, how tightly the environment needs to be controlled, etc.

All in one convenient, compact and machine-readable description that follows the application wherever it goes.

Now To The Other Side Of The Spectrum

Let’s see how this same policy-centric thinking can be applied to a mission-critical application.

Once again, our application policy has specified desired resources and services needed (compute, memory, storage, bandwidth, data protection, security, etc.) but now we need to go in the other direction.

If this particular mission-critical application isn’t meeting its performance objectives, one policy recourse might be to issue a prioritized request for more resources — potentially at the expense of less-critical applications. Yes, life is unfair.

When it comes to critical services (e.g. high availability, disaster recovery, security, etc.) we’d want continual compliance checking to ensure that the requested services are in place, and functioning properly.

And, when we consider a “special event” (e.g. data center failure, update, etc.), we’d want to make sure our process and capabilities were iron-clad, e.g. no introducing new software components until testing has completed, and a back-out capability is in place.

What’s atypical is the top-down automation of all aspects of IT service management, using a centralized application policy as a core construct to drive automation.

With this approach we don't have to limit ourselves to a few, impossibly broad policy buckets, like "business critical". We can precisely specify what each and every application might need, separately and independently.

It seems to be a truism that IT spends 80% of their time on 20% of the applications -- mostly because their requirements are unique.

When I consider the previous transformation from typical silo-oriented IT to ITaaS it’s often proven to be difficult and painful. When you undertake to change IT’s basic operating model, it demands strong, consistent leadership.

It’s not just that new approaches have to be learned, it’s just that so much has to be unlearned.

And the ITaaS transformation isn’t just limited to the IT function. Not only does IT need to learn how to produce differently, the business also needs to learn how to consume (and pay for services) differently.

But for those who have already made this investment — and you know who you are — the next step to policy-based automation is comparatively easy. Indeed, in many ways it will be a natural progression, resulting from the need for continually improving and impactful automation.

To achieve this desirable outcome on a broader scale, there are more than a few hurdles to consider.

First, all participants and components in a policy-driven IT environment need to be able to react consistently to external policy.

This, in many ways, is software-defined in a nutshell. Indeed, when I'm asked "why software defined?" my knee-jerk response is "to better automate".

Servers need to react. Networks need to react. Storage needs to react. Data protection and security and everything else needs to react. All driven by policy.

Second, serious process work is required to formally document actionable policies in machine-readable form. So much of IT operations is often tribal knowledge and accumulated experience.

As long as that knowledge lives in human brains — and isn’t in machine readable form — automation productivity will be hampered.

Third, the resulting IT organization will likely be structured differently than today, overweighted towards all aspects of process: process definition, process measurement, process improvement — just as you would find in non-IT environments that invest heavily in automation.

And any strategy that results in refashioning the org chart brings its own special challenges.

Creating That End State Goal

So much of leadership is painting a picture for teams to work towards. When it comes to IT leadership, I think policy-based automation needs to be a component of that aspirational vision.

A world where virtually all aspects of IT service management are driven by application-centric, machine-readable policies. Change the policy, change the behavior.

The underlying ideas are simple and powerful. They stand in direct contrast to how we’ve historically done IT operations — which is precisely what makes them so attractive.

To say there were a few personal worry-bubbles when we took it to market would be an understatement. What would our customers think? How would storage people react to it?

Now that the first full quarter is behind us (and largely public information), I can safely declare: VSAN shows every sign of being a winner.

The Public Information

As part of VMware’s earnings call, VSAN was identified as a “growth product”, also “well exceeded internal plan” and, in a later interview had “over 300 paying customers”. NSX — another growth product which has been out for bit longer — is now at a $100m run rate.

Not too bad for VSAN’s first full quarter out. Heck, just getting a mention in the earnings call is a big deal :)

But there's more to the story than those sparse tidbits ...

Strong Initial Appeal

Thanks to VMware’s incredibly broad and engaged customer base, just about everyone I meet knows what VSAN is, what it does, how it works, etc.

Personally, I find that quite impressive, and I that’s a testament to VMware’s surprisingly powerful brand in the IT marketplace.

During the extensive beta (~11,000 downloads), most people found that VSAN did exactly what VMware said it would do: great performance, solid reliability — and a hard-to-beat simplicity factor.

The Metrics That Matter

From my days as a product manager, I’ve learned that — once you get past the revenue numbers — only two things really matter: customer experience and customer enthusiasm.

The best way for me to understand customer experience is to look closely at customer support data: how many support requests, what kind, how serious, how long did it take to resolve, etc.

I'm very pleased that there’s not much to report here — simply because there aren’t that many support calls, at least, not in proportion to the number of customers in production.

When it comes to customer support calls, no news is good news.

Customer enthusiasm is harder to quantify — what you’re looking for is one IT pro enthusiastically recommending a product to others. Here, I have to troll on Twitter, various tech forums and blog posts, but — by all indications, VSAN enthusiasm is substantial and growing.

The consensus is clear: VSAN is a well-engineered product that does what it says it does, and offers a unique value proposition when compared to all other alternatives.

Evolving Reactions From Storage Pros

By now, I’ve presented VSAN dozens of times in front of combined IT teams: server people, storage people, etc. And it’s been fascinating to watch how their reactions have evolved — even in a very short time.

The first reaction (early on) was a bit of confusion: where’s the storage array, where are the LUNs, what external protocol does VSAN use, etc. Some patient explaining was required at the time — VSAN is not storage as most are familiar with.

The second reaction was a feature comparison: where is X, Y and Z? Some of those things are in VSAN today, some are coming, others are farther out. More patient explaining that VSAN (at least initially) isn't intended for use cases that required sophisticated data services -- it was more about efficiency and simplicity.

The third reaction was the most surprising, but it finally came: that “aha!” moment when the storage team realized that — since VSAN was under the control of the virtualization team — those server admins could pretty much take care of their own day-to-day and less-critical storage requests, freeing the storage team to focus their time on critical stuff like big databases and mission-critical applications.

Deploying VSAN could free up a whole lot of time for the storage team — not to mention saving a bunch of capex. And there’s no real argument when both sides see a win.

VMware Horizon with View running on VSAN is turning out to be surprisingly popular. There’s great management integration (low opex) and — for most situations — it’s surprisingly cost effective.

VSAN performance is very good, approaching the territory of all-flash arrays when running a variety of benchmarks. Recently, a very thorough reference architecture was published -- with all sorts of measured performance runs and great detail — a great read to help understand both the power of the combined solution, or just appreciate VSAN’s exceptional performance when co-residing on the same VDI servers.

And Now On To VVols …

Part of the enthusiasm around VSAN is the utter simplicity of the management experience: set high-level policies on a per-application basis, and VSAN takes care of the rest.

Between VSAN and VVols, we’re clearly moving to an entirely new management model for storage: one where application policies are used to drive storage behavior -- the underlying essence of software-defined storage. It may take a while for this model to be fully embraced, but it’s inevitable from my perspective.

What do we gain by preferring a singular, consistent and integrated view of software-defined disciplines — and what do we lose by considering them individually, using a traditional lens?

Let’s Start With Compute …

Wrapping your head around software-defined anything can take some serious effort, if my personal experience is any guide. I do what I can to help explain the key ideas, and why they are important.

A good starting point for wading into the deep end is our familiar server virtualization — something we all have experience with. Seen through a “software defined” lens, we could better describe it as using application policy to dynamically compose compute services.

Here’s an application. Here are the server resources and services I want it to have: memory, vCPUs, HA, priority, etc. I express my desires to the hypervisor using a policy, which then dynamically allocates and manages the resources and services on my behalf.

If my requirements change, I simply change the policy — the hypervisor takes care of the messy details of reallocation, rebalancing, etc.

This is not exotic stuff — it’s how hypervisors are used around the world today.

And Now On To Networks …

The “killer app” that drove server virtualization was consolidation: cramming more work onto physical hardware. A very tactical need for efficiency resulted in a very strategic outcome.

The first “killer app” for network virtualization (or software-defined networking) appears to be micro-segmentation — the ability to easily define isolated networks behind the enterprise perimeter, and manage the rules by which they interact — using policies.

This is a big deal, and is worth a short explanation if you're not familiar.

In the last few years, large flat internal networks have proven to be enormous security risks. Attackers can use a footprint gained on one device to probe the entire enterprise network at leisure.

Since no one can keep all the bad guys out all of the time, the logical answer is to segment the network to contain the damage. A desktop in accounts payable shouldn’t be trying to access source code in engineering, as an example.

But as one tries to implement that idea in a larger setting, reality can bite when using traditional technology. Now you have dozens (or hundreds!) of network segments, each with its own need for a firewall and rule set. Enterprises are dynamic entities, so trust boundaries and desired interactions are constantly changing.

However, using a software-defined network makes this particular challenge very approachable. Application policies are used to dynamically compose network services. Change the policy, change the services. Indeed, more than few NSX implementations have been driven by this need.

And, once again, a very important tactical requirement (improving security) results in a very strategic outcome.

And Then To Storage

There’s no clear consensus on what the “killer app” will be that drives the first round of software-defined storage consumption, but I have a strong suspect: private cloud deployments.

Private clouds only thrive when highly automated — it’s all about efficient ease-of-consumption. By definition, there’s no good way to predict what storage services will be desired. Change is constant: moving service levels up and down, consuming more or less of resources, etc.

And inserting human beings in the middle of each and every storage workflow sort of defeats the whole purpose.

What you’d like to be able to do is have the application express desired storage policy, which is then used to dynamically compose the requested services. Change the policy, change the services.

Once again, a very tactical need (efficient ease-of-consumption) may result in a very strategic outcome.

Making The Case For A Singular Software-Defined View

Note that in each case (compute, network, storage) services are dynamically composed in response to an application’s specific policy.

So — at least at one level — the raison d’être for software-defined anything is giving applications what they need.

But that still doesn’t make an ironclad case for an integrated software-defined model vs. considering them as traditional standalone functions.

I think the argument for the former has three components: dependencies, optimizations and consistency.

Yes, this is an argument for VMware’s vision of SDDC. It's not that I'm opposed to alternative viewpoints, but -- as of this writing -- there aren't a lot of serious alternatives to consider for the enterprise IT crowd.

Examples Of Cross-Domain Service Dependencies

Many application services are dependent on others. Clearly, provisioning compute necessitates provisioning network and storage resources, but there’s more to the picture than that.

One example from the network world might be logging. If I request a logging service, that’s going to take storage — maybe a lot. If I want those logs automatically analyzed, that’s going to take compute as well. The network service catalog will need to be able to autonomously provision storage and compute services, if it's all going to work smoothly.

While some network services might be considered in isolation (e.g. firewalls), the more useful ones will be composed of storage and compute services underneath.

Similar examples emerge from the storage world. Most application-storage communication involves a network of some sort, which must be provisioned and managed. Various forms of data protection (remote replication and backups) are dependent on network bandwidth. Deduplication — if requested — is CPU and memory intensive. Ideally, storage services could request compute and network services as needed.

Again, while some storage services might be self-contained, the more interesting ones will be cascade to network and compute services.

Here’s the point: in our emergent picture of software-defined, we have to allow for services in one category (e.g. networking) to request supporting services (e.g. storage, compute).

That sort of integration going to be hard to achieve unless each of the support software-defined disciplines have been designed to consume other services presumed to be present.

Examples Of Cross-Domain Service Optimizations

When you go looking, all sorts of interesting new optimizations are possible when we consider an integrated software-defined view vs. separate disciplines.

Let’s say you’re replicating over a remote link, and things are getting slow. Yes, you could allocate more bandwidth — if available.

But there are other options to consider.

For example, you could crank up compression (requiring compute and memory), or perhaps simply throttle the offending party if that’s an application policy option.

Another example: storage is getting pummeled with read requests, and existing storage read cache is getting overwhelmed — such as you might see with a boot storm. Rather than starve other storage consumers, one optimization scenario might be to dynamically allocate a chunk of RAM as read cache (e.g. CBRC in VDI deployments) until the storm passes, and then reallocate.

Here’s the point: when policy objectives aren’t being met, there’s usually more than one way to respond. Many of those responses involve substituting one kind of resource for another, and that will require a high degree of integration and awareness across functional domains.

Examples Of Cross-Domain Service Consistency

Getting everyone “on the same page” can be difficult in an organization where people have to work closely together. In highly automated and optimized IT infrastructure, it’s the same situation.

Yes, we can laboriously specify externally to storage, network and compute precisely what these policy concepts might mean, but a more attractive alternative would be to leverage a reasonably consistent set of definitions across all three.

This desire for underlying consistency becomes magnified when one considers building hybrid clouds using external service providers. Consistent concepts and behaviors make the job of orchestrating predictable behavior across clouds that much easier.

A related perspective emerges when one considers monitoring, alerting, reporting, etc. across the integrated whole. While integrating disparate components is possible, it’s a lot of work that seems to never end.

The Observed Adoption Anti-Pattern

While I’m sure there will be some forward-thinking enterprise IT architects that see software-defined as an integrated and consistent construct across disciplines, the likely adoption patterns are most clearly pulling in the opposite direction.

The compute/server team makes a hypervisor choice, and figures out how to best automate the things they are responsible. The network team starts considering network virtualization, makes a technology choice, and starts automating their bits. And, finally, the same story repeats for the storage team.

Three teams, three potentially different choices.

Each team is embracing software-defined concepts. Each has identified the “win” for their part of the puzzle. Each is acting in a logical and pragmatic manner — albeit from their functional perspective.

Pity the poor infrastructure architect who has to glue those choices together in a single, cohesive whole.

In the hardware-defined world, we attempt to solve this multi-technology challenge with long, detailed and constantly-changing interoperability matrices (from each vendor!), a substantial and ongoing investment in interoperability and regression testing, plus after-the-fact integration for key workflows for things like provisioning, monitoring and reporting.

Indeed, much of the appeal of converged hardware infrastructure (e.g. Vblocks, Flexpods, et. al.) lies in re-creating the integrated nature of products that were designed and implemented independently. And addressing that need has quickly become a fast-growing multi-billion dollar market segment -- the demand is obviously there.

I feat that what we saw in the hardware-defined world, we will inevitably see in the software-defined one.

Because — while the technology has certainly changed — we humans really haven’t.

I stirred up a real hornet's nest with this one. On one side, the cloud-focused types who believe that "enterprise IT just doesn't get it". On the other side, a vast army that retorts "you just don't get enterprise IT".

It's inevitable that various vendors and analysts will start using a term without a crisp set of defining attributes. I'm a storage geek, so this is a big deal to me. Here's my list. So far, I haven't seen a better one.

A few years back, I became deeply fascinated with the consistent patterns I found when IT organizations started to transform. I wrote a lot on the topic, the collection you can find here.

For me, IT transformation to an ITaaS model is as inevitable as adolescence -- sooner or later, everyone is going to go through it. Just not at the same time. This particular article gets a lot of traffic: if you're interested, the entire collection is worth reviewing.

Parents of my age faced an interesting challenge: we were brought up in one world, but were raising our children in a very different world -- one with no clear guidance on how to deal with various issues.

As I re-read this, I realize that today's parents might have a far better handle on these topics than my generation did at the time. That's goodness.

One of the time-honored rituals in the IT industry is the periodic publishing of Gartner’s Magic Quadrants for different enterprise technology categories.

While some inevitably poke fun (e.g. the Mystical Quadrilateral, the Prescient Parallelogram, etc.), most any IT vendor will tell you it’s very serious business indeed.

Years of effort are spent trying to painfully nudge our particular dot upwards and to the right. Gartner doesn’t hand out free passes.

Once you get your product into a favored position (e.g. one of the leaders), the goal then shifts to create as much distance as you can between your dot and everyone else in your category.

This is the fifth year Gartner has recognized VMware as one of the leaders in the hypervisor category. Per Gartner’s rules, vendors can’t claim to be the singular leader, just one of the plural leaders.

No shock, Gartner identified two MQ leaders in hypervisor technology: VMware and Microsoft. Fair enough. It wouldn’t do to have just one :)

But as I read through the Gartner analysis and criteria for judging, it felt almost historical in nature. How Gartner has historically viewed hypervisor choice — and how I think enterprise IT groups now view hypervisor choice — have clearly separated.

Having stared at Gartner MQs for many years, the first thing that jumps out is the wide separation between the two leaders. Most Gartner MQs you’ll see have the leaders tightly clustered around each other, with only infinitesimal separation requiring maximum zoom to discern.

The visible implication here is that the two “leaders” are in reality not all that close — which would be my personal observation as well.

Notice that Oracle VM peeks over the horizon as a “challenger”. I don’t think Gartner could have placed their dot any lower in that quadrant.

All other hypervisors are banished to the “niche player” category.

Now, On To The Details

Right off the bat, Gartner claims ‘at least’ 70% of x86 workloads have already been virtualized. So, you might ask, what’s left to be virtualized?

One healthy component would be servers in very small settings — server closets and the like — the “small” part of SMB — where you’ll still find a lot of yet-to-be-virtualized servers.

Personally, I believe this niche is where Microsoft is seeing some success with their Hyper-V feature — these undemanding environments typically have very limited IT staff (e.g. 1-3 people) that already need to support Microsoft operating systems and tools anyway, so using the Hyper-V feature included as part of Windows makes a certain sense.

If you read through the commentary, the Gartner analysts seem to think Microsoft primarily has a sales and marketing challenge, and not product issues. Perhaps this is a result of feature-checklist comparisons. I would disagree with Gartner’s assessment — I think there are significant and meaningful differences in the respective technologies.

The people I know who use vSphere tend to routinely push it to its limits. I have yet to meet (or hear of) anyone who’s pushing Hyper-V hard — or trying anything particularly ambitious. Those that do often end up as VMware customers.

I would think this is a material difference between the two technologies.

Gartner cites their own December 2013 survey, where more than 90% of the respondents consider vSphere their primary hypervisor, and 48% named Hyper-V as their secondary hypervisor. I find this curious — what, exactly, does “secondary” mean?

Given the enormous pressure for IT organizations to standardize on common automation, technology stacks, etc. — I have to openly question what significant role — if any — a “secondary” hypervisor would play in most enterprises.

While running production on multiple hypervisors might sound useful in theory, I don't meet many people working in IT who would consider that an ideal state of affairs.

Where Gartner Might Be Missing It

In all fairness, Gartner’s perspective is on well-defined and generally-accepted historical industry categories, with x86 hypervisors being one of those. That’s what analysts do.

But things have moved on. Maybe we could have had an isolated “who has the best hypervisor?” discussion back in 2008, but the world is a different place now.

Back then, the major attraction was server consolidation: more logical servers on less physical hardware. That’s still somewhat important: one can still differentiate on how many logical servers can be loaded up, performance, efficiency, features, etc.

But for many, I think the focus has moved on to bigger goals. These days, I believe there are two important strategic vectors in play when it comes to hypervisor choice.

The first goal is automation — the choice of hypervisor greatly impacts the degree of automation sophistication that can be achieved, and how quickly. Server consolidation was all about capex efficiency, advanced automation is all about opex efficiency.

Yes, there are a great many automation tools available today that are positioned as hypervisor-agnostic, but — for most time-pressed enterprise IT groups — the natively-integrated automation tools (e.g. vCAC for vSphere and other hypervisors) hold great appeal.

The second strategic vector is that there’s more to the hypervisor discussion these days than familiar server virtualization. Software-defined networking (and software-defined storage) is a real thing.

Should these be considered stand-alone functions, or — ideally — seen as extensions of the server hypervisor?

The idea of taking three independently-conceived and engineered “software-defined” stacks (compute, network, storage) and integrating them into a single software-defined data center strikes me as an unnecessarily difficult (and inherently unoptimized) path.

Certainly if one wants to automate converged workflows across all three disciplines.

My belief is that — once people get their head wrapped around the desirability of these newer forms of infrastructure virtualization, there will be strong appeal in native integration vs. after-the-fact interoperability.

Not Everyone Sees It This Way

Sampling bias: I spend the majority of my time with larger and/or progressive IT organizations. My views are shaped by theirs. That being said, there are plenty of IT consumers that are neither large nor progressive. Pragmatism rules: people do what makes sense for them.

Anything beyond simple automation and UIs is probably not a priority for this crowd. They want native, intuitive and built-in workflows that work the way they do.

Similarly, software-defined networking and storage will only be appealing to them once it’s neatly baked into the hypervisor they’re using.

I think VSAN is a good example of this.

Humble Yet Purposeful

You might think the latest Gartner results would encourage VMware to boast proudly and loudly as many vendors would be tempted to. Not the case -- an obligatory matter-of-fact press release, a blog post, and not much more.

The mood is humble yet purposeful. The people I work with are incredibly demanding on themselves and their colleagues. Nothing is “good enough” for them. Their consensus is clear: we’ve only scratched the surface on what can be achieved. For most, IT infrastructure is still unnecessarily complex and inefficient.

While VMware has certainly achieved much, there is so much more that needs to be done.

The individual infrastructure components are straightforward enough when considered individually; getting everything to interoperate smoothly is far more burdensome.

As hardware prices fall — and IT capacity balloons — the focus has clearly shifted to improved operational efficiency: doing more/faster/better but without adding more people. “Simple” is the new killer app in enterprise IT infrastructure.

All three approaches to integrated infrastructure — converged, hyper-converged, hypervisor-converged — have that singular goal in mind: making things simpler.

But each approaches the problem in a different way — and with different results.

With Best Intentions

Most IT infrastructure vendors invest heavily in their products so that they can be used with others: providing external interfaces, testing for interoperability, writing lengthy reference architectures, and so on.

But despite best intentions, the resulting environments end up being notoriously difficult to architect, integrate, support and manage efficiently on a day-to-day basis.

As Philip K. Dick said “exactly what the powers of hell feed on: the best instincts in man” — a purgatory many IT professionals know well.

Form inevitably follows function: look at most traditional IT organizational charts, and you can see familiar server teams, network teams and storage teams. Since it’s all infrastructure, workflows often bounce back-and-forth between the silos like a pachinko machine.

It’s inefficient and it’s wasteful. And just about every IT group is under enormous pressure to move forward -- on to something better.

Enter "Convergence"

The problem with using any term in an IT discussion is that definitions are imprecise -- exactly the situation here. There are no standards bodies or arbitrators, so we make do the best we can.

At its essence, convergence is a simple idea: make disparate components act as a whole.

Perhaps the best example of convergence is the modern smartphone, which — in addition to making plain-old voice calls, is also a camera, music player, GPS, web browser, video player, gaming console et. al.

As a result, you can drive amazing workflows thanks to convergence on smartphones.

One example: my daughter took videos of a recent concert, edited it, added her voice-over commentary, and posted it on a half-dozen social sites.

While she was catching a ride back from the show.

Good thing I subsidize her data plan :)

Not A New Idea

Through my jaundiced eyes, convergence in IT infrastructure isn’t really a new idea. During the golden age of minicomputers, these early examples of converged infrastructure came with their own purpose-built hardware, networks, storage, databases, etc.

The good news? Everything usually worked together as you’d expect — and were very operationally efficient as a result. The bad news? Functionality was limited to what the vendor provided, and your choices were always very constrained.

The pendulum swung: UNIX, Windows, client-server — and complexity increased as a result.

Now it appears to be swinging back.

#1 -- Converged Infrastructure

In my eyes, the first modern take on converged infrastructure was the VCE Vblock — something I was a big fan of during its inception. VCE — a separate company — delivers a pre-integrated product — a Vblock — built on technologies from VMware, EMC and Cisco.

VCE invested heavily to create a singular product experience: from sizing to delivery to installation to ongoing upgrades and support — and it continues to be quite successful.

NetApp quickly followed with FlexPod — an alternative take that initially focused on a reference architecture: assembled, delivered and supported through partners.

Shortly thereafter: HP, IBM, Hitachi and others followed. EMC offered up VSPEX which allowed more choice on server and network — but using an EMC array, of course.

IDC defines the category as “integrated infrastructure” — as opposed to “integrated platforms” that are designed to run one type of workload (e.g. Oracle, Teradata) vs. many.

That's approaching a $10B category that didn't really exist just a few years ago.

All of these converged offerings have a similar premise: integrate familiar infrastructure components to dramatically simplify the sizing, design, acquisition, installation, support (and to a certain degree, management) of IT infrastructure.

However, your choices — while good ones — are necessarily limited. For Vblocks, it’s the vSphere hypervisor and management stack, Cisco servers and networking, with EMC storage and data protection. For FlexPod, the same generally schema, except with FAS arrays. Similar for HP, IBM, Hitachi et. al.

Despite the appearance of the dreaded lock-in, more than a few people are willing to accept a few reasonable restrictions in exchange for a far simpler experience.

#2 -- Hyper-Converged Infrastructure

A few startups in (SimpliVity, Nutanix and just recently NIMBOXX) have taken the idea a bit further — by using software to eliminate the need for external storage. All sell pre-integrated modules that are easy to order, install and — presumably support.

While IDC does not track this category (yet!) —- and the companies involved are not public — it’s hard for me to get a bead on what’s actually getting sold vs. evals, POCs, etc. I can only assume that all are enjoying modest success -- but nowhere near the converged infrastructure category.

Understandably, there are tradeoffs involved.

For each, there’s only one primary storage choice: the software stack that is provided by the vendor — no external storage as you’d find using a hypervisor-based approach. And, of course, you’re limited to the specific server choices that are OEMed by each.

That being said, I’ve seen situations that would be well-served by simply buying a few modules and just getting on with it. And I'm sure we'll see more offerings in this category before too long.

While these solutions are arguably “more converged” than the previous category, calling them “hyper-converged” is a bit of a stretch. Network functions are still largely separately managed, so there’s that. Since two of these offerings use vSphere, you’re presented with at least two (or more) distinct management experiences — the one from the vendor, and the one from the hypervisor.

And there’s only so much you can do with plug-ins :)

Hypervisor-Converged Infrastructure

To be clear, I’m speaking on my own here — VMware has not widely adopted this term, except when describing VSAN. But -- when used more broadly -- I'm finding it a useful term that accurately describes what makes software-defined data center (or software-defined converged infrastructure?) so compelling.

At a conceptual level, the hypervisor sits in a very strategic position between applications (and their operating systems) — and the underlying infrastructure. It can abstract compute, network and storage. It can provide a powerful control plane over each, as well as additional functionality.

And — because it can abstract all the underlying resources, the hypervisor can converge the operational management experience in a way that’s almost impossible in other ways.

Sure, prior to smartphones, I could have shot a movie on a video camera, transferred it to my computer for editing, recorded a voice-over, and individually uploaded it to multiple sites. But it would have taken many more steps, taken a lot longer, and problems would have inevitably cropped up along the way. Converged is easier.

Convergence — deep integration across normally disparate functionality —- occurs in the hypervisor. As a result, you’re now operationally managing the hypervisor’s abstractions, and not the underlying subsystems.

The resulting workflows can be natural and seamlessly cross boundaries: optimized around form vs. function. It’s an approach to convergence that's designed to be automated — it’s not an afterthought.

To be clear, “hypervisor-converged” doesn’t mean all the functionality is provided by the hypervisor alone -- just that functionality is abstracted -- although software-based functionality will certainly be an option for some.

For example, VMware’s NSX can effectively exploit underlying functionality on a wide range of networking gear, including Cisco’s. When vSphere VVols (virtual volumes) become generally available (now in public beta), the same can be said for external storage arrays. And, of course, vSphere has been exploiting physical server hardware for a long time, and does it well.

Contrasted against other forms of convergence, one thing is obvious — a hypervisor-converged environment can run on just about any reasonable choice of gear an IT shop wants: server, network and storage. That's attractive to many. But at the same time, all those choices can create complexity, especially with regards to sizing, procurement, integration, support etc.

We’ve already seen some “appliance-ization” of hypervisor-converged infrastructure (e.g. VSAN Ready Nodes), and I think it’s safe to say we’ll see a lot more before too long.

A hypervisor-converged solution that can either be consumed on an appliance-ized module, or run side-by-side with roll-your-own hardware will be very, very attractive to many, I think.

More Open Than You Might Think

And then there’s the concern that some express regarding potential hypervisor lock-in; specifically vSphere as it's so widely used. Note that the significant majority of converged and hyper-converged products are already built on vSphere today. Since the workloads themselves are eminently portable to other hypervisors and public clouds, it's really the operational model that deserves the focus.

Although I think there’s far more choice and openness here than generally recognized.

As a starting point, it’s long been the case that there’s a rich choice of management tools and third-pary add-ons -- in addition to the ones that vSphere provides as part of vCloud Suite. Arguably far more choices than any other hypervisor. Not to forget, the VMware-provided management tools are quite adept at managing non-vSphere environments.

What’s more recent is the great out-of-the-box integration today with OpenStack. As an example, check out this eye-opening video showing mix-and-match interoperability of NSX and vSphere with KVM-based hypervisors managed by Openstack Horizon. That's right: mix-and-match hypervisors -- although not deep integration, which is required for convergence.

Certainly more to come here.

Convergence Is Here To Stay — And Grow

Every IT shop on the planet craves a far simpler environment — one that lets their team do more — and do it better — with far less effort.

Maybe I’m biased, but I see the evolution here as a logical progression: converged, hyper-converged and ultimately hypervisor-converged.

In case you missed it: the beta for vSphere "next" is live, available here:

It's an open beta -- just about anyone can participate. That's cool.

If you're a storage geek like me, you'll find the vVol bits are there, and -- depending on your storage vendor -- you can start to put the new vVol storage management model through its paces. If you're not familiar, more on that here.

If you're into VSAN, you'll find the beta of the next version as well. Big fun.

For those of you who were worried you had nothing to do during the upcoming holiday weekend in the US, here's your project.

Change the circumstance to a crisis footing (or the perception of one), and — all of the sudden — the ball moves much faster down the field, obstacles diminish and magic can often result.

Those of us who aspire to be change agents should take ample note of this useful human foible, and always be on the lookout for a good crisis that can be used to advance the agenda.

Or, failing that, perhaps fabricate your own.

It’s Evolution

Not all that long ago (biologically speaking), we were hunter/gatherers who spent most of our days looking for food. Not infrequently, we’d be seen as food by other, larger species.

Those of us that could summon deeper abilities on short notice (faster thinking, a quick escape or putting up a good fight) survived to live another day.

Genetic selection being what it is, most of us have this ability to “turn it up” when we feel threatened. Our vision narrows, time seems to move slower, the brain jumps into a higher gear, blood pumps, etc.

It was a useful survival instinct back then, and still is useful now.

In The Modern Day

When grappling with how best to drive change in an organizational setting, it seems that creating that “sense of urgency” frustrates many.

The way I usually get into this topic is when talking about IT transformation with IT leaders. They can clearly make the case for change. They often have good ideas about how to do it. They are usually making clear steps in the right direction, but it’s all moving way too slowly for them.

How do you go about lighting a collective fire underneath your team? Create that sense of life-or-death urgency that focuses the mind?

That is, within socially acceptable norms :)

Grim Realities

Back when I was doing a lot of IT transformation work with EMC’s customers, I had the opportunity to meet with over 200 IT organizations that were interested enough in the topic to bring me in and have a deep discussion.

It was a fascinating journey into the psyche of modern IT organizations and how they think about things. It also gave me an entirely new perspective on the interactions between vendors and their best customers.

Those that were making the best progress either had (a) an exceptional leader and leadership team at the helm, or (b) were dealing with an exceptional crisis that could become an existential threat to either the IT organization, or the business itself.

Mere mortals and business as usual? Slow, methodical and unsatisfactory progress at best.

Examples Everywhere

So, what constitutes a good crisis when discussing IT transformation?

A large, established retailer with declining results brought in a new CEO to drive a new agenda. The CEO, in turn, lost no time in methodically firing a bunch of execs, and bringing in a new leadership team.

All of the sudden, the IT leadership team was finding new religion in a big hurry.

A mid-sized bank got acquired by a much larger organization. The inevitable battle had begun within IT for control and domination. For the IT group that got acquired, it was clearly an existential threat, and they were looking for advice on how best to play the game.

They ended up doing quite well for themselves, it seems.

A specialized retailer/distributor was clearly getting Amazon’d. No one — execs, IT — really knew what to do about it, nor seemed particularly motivated to take the required radical action. I walked away from that meeting thinking “you’re going to be dinner for something bigger”.

And that’s what happened.

But you shouldn’t have to be on the precipice of doom to harvest the potential of a good crisis. After all, a good crisis can be manufactured if needed :)

The Manufactured Crisis

What do I mean by this? Taking an event and/or documented data and amplifying it to instill fear, anxiety — and motivation — in the team.

Yes, you’re being manipulative. Get over it. That’s what good leaders do.

In IT, massive system failures are legendary in driving rapid change in organization, process and often vendor choice. Nothing like blood on the walls to make things happen in a big hurry. But it would be preferable to find something a bit less destructive.

I’ve seen a number of clever examples of this, some worth sharing.

One IT leader paid for an extensive “customer satisfaction” survey among internal users. The results came back predictably awful, with no shortage of authentic, negative -- even caustic -- commentary.

The message? This is what your co-workers think of you and the work you do. He made it very personal. With that hammer in hand, he was able to drive much greater change at a rate much faster than before. Six months later, he did it again, and had a brand new hammer to wield.

Over a two year period, things changed substantially, and the survey showed it.

Another IT leader was able to work with finance to determine how much the organization overall was spending on IT services (internal and external), and just how much of that was going to internal IT.

Plotted over five years, it showed a very grim picture, with a neat linear extrapolation that IT would essentially be out-of-business by 2020!

His message: who wants to be here in a few years to turn the lights out? Very emotional stuff.

A third leader created a win/loss scorecard, detailing all the cool IT-related projects that were theoretically up-for-grabs at one point, but went to either an external agency, consultant, service, etc.

Clear rationale was presented for the decision — in each case, the internal IT team was judged not up to the task.

No one likes being a consistent loser.

His message: you want to work on cool stuff? We’ve got to earn the right, and here’s how we’re going to do it.

Note the emphasis on storytelling: people don’t engage with the facts unless they’re wrapped in a well-told story line.

Because we all like a good story.

The Same Thing Happens Inside IT Vendors

IT vendors are companies, much like other companies. The same laws of organizational physics are evident here as everywhere else.

The things that tend to drive the most visible and progressive changes are (a) one or more highly visible losses to a competitor, or (b) letting a customer down badly for one reason or another.

When wrapped with a well-told narrative — and broadcast widely — these tend to drive accelerated change in the presence of a good leadership team.

People often want to do the right thing — it’s just a matter of creating the right motivations. You, as a leader, can help with that.

A promising young start-up (Code Spaces) was held up for ransom by an intruder who broke into their AWS account and took control. The digital kidnapper wanted a payoff, or else …

A sad posting says that — basically — all their customer’s data is gone, and they’re done for. That’s it. There’s no coming back for them. Not to mention the pain inflicted on their trusting customers.

In a not entirely-unrelated story, in the US, the IRS (the tax agency) is in serious hot water because they can’t produce emails in the context of a congressional investigation. The excuse? The emails were on a personal hard drive (??), which failed, and has long been disposed of.

While the IRS is not out of business (after all, they’re a government agency), they’re certainly seriously impacted by the incident, making doing business more difficult. No, I’m not going to try and claim the same with my personal tax records …

And with every tragedy, there are lessons to learn.

Do You Have A REAL Backup?

IT professionals know that a REAL backup is one that’s completely separate and isolated from the original data source as many ways as possible: separated logically, separated physically, stored on different media technology, different access credentials, etc.

The more kinds of separation, the better the protection.

I have taken ridicule for this position before (e.g. people who consider a simple snap a backup), but I’ll stand my ground. All those snapshots aren't doing Code Spaces much good now, are they?

If losing data permanently and irretrievably would be an unmitigated disaster, then extra precautions are needed.

What’s Changing

What’s become popular recently is a new breed of “digital kidnapper” — someone who extorts ransom to avoid the loss of your data.

We all know (or should know) about the recent spate of malware that encrypts your personal hard drive. If you derive your livelihood from your personal computer (as many of us do), this can be a life-altering experience.

If you didn’t have religion about real backups before, you’ll certainly have it now.

The Cloud Angle

Code Spaces appears to have run entirely on Amazon’s AWS — primary data, backups, etc. In my book, that’s dangerous — if AWS has a bad day, you have an even worse day. And everyone has a bad day, sooner or later.

All access was through their control panel. The bad guy got access, and he was in business. Not being deeply familiar with AWS, I’m now very curious about how access control is set up for AWS’ control panel.

An awful lot of valuable data is stored there — think of it as a huge bank — and one now has to ask questions to see if it could happen again, and what steps would be necessary to prevent that.

A related question: was there anyone at AWS they could have contacted to help out? Amazon’s model is highly automated; when a customer has a crisis of this magnitude, I would guess they’re not set up to respond quickly, if at all. The service did what it was designed to do.

In hindsight, if Code Spaces had been making simple lazy copies to anything else — a home computer, a server elsewhere, etc. — the effects of the attack could be somewhat mitigated. They’d be in business, after a stretch.

That’s the value of a real backup: when something bad happens, you’re injured, but you’re not dead.

Shifting Tides

Not all that long ago, most business processes ran on a combination of paper and digital. If the computer lost data, you could always go back to paper records, and attempt to recreate things.

Not anymore. There’s no paper trail. Lose the data, it’s gone. Although, in the case of the IRS, I bet those emails are somewhere :)

Information is the new wealth, the new repository of value. That’s going to attract bad guys — if not for IP theft, then for ransom attempts.

Just like you can get your bank account cleaned out, you can get your cloud account cleaned out — with similar disastrous impacts.

This is not a criticism of clouds, or AWS, or anything else — just that the world has changed, and we must think and act differently to protect our information.

So many people enamored with the brave new world of web-scale IT. You see them online, at conferences, etc. — brimming with passion, enthusiasm and excitement. I call them the clouderati — their heads are clearly in the clouds.

It seems that every few weeks this group brings us a new shiny meme that captures a lot of attention: containers, devops, OpenStack, NoSQL databases, Open Compute — even machine-learning algorithms for energy efficiency.

I’m certainly not immune to the attraction. So many innovative concepts to learn about, evaluate, debate, etc. If you’re looking for intellectual stimulation, there’s no shortage.

But, in many ways, it’s world unto itself — this world of the clouderati.

A shiny bubble, floating over the vast and often grim landscape of real-world enterprise IT. No one is debating the coolness of whatever the latest idea might be; it’s just that it’s not consumable by the vast majority of enterprise IT shops.

If it’s not consumable, it’s not particularly interesting to this audience.

Judging by many of the conversations I’ve had, I believe that many of these enthusiastic new web-scale tech people might not have a gritty appreciation of the enterprise IT terrain. So, in an effort to promote a more productive dialog, here is my quick primer on just a few of the harsh realities of enterprise IT.

#1 — Business Model Matters

If you’re a web-scale company, IT doesn’t just support the business, IT is the business.

Most web-scale outfits have business models that prioritize making IT differentiated, so they invest like crazy in getting better at it. They have large teams, they hire PhDs, etc. As a result, they can experiment, they can write code, etc. — all part of their model. As it should be.

By comparison, most enterprise IT organizations work for businesses that don’t prioritize IT differentiation. So many executives see IT as little more than a cost of doing business, rather than a source of potential business innovation. Thankfully, that’s starting to change.

When comparing web-scale IT and enterprise IT, the application profiles are very different. Most enterprise IT organizations have to deal with wildly diverse application workloads — each with unique requirements vs. the relative conformity of most web-scale environments.

One-size-fits-all doesn’t.

Yes, Google does it. But enterprise IT is not Google.

Fun exercise: if you have the opportunity, ask a web-scale company where they run their boring-but-critical applications like financials, HR, CRM, etc. Hint: it usually isn’t on their cloud.

#2 — Perspective Matters

The vast majority of people I know who work in enterprise IT have a difficult and often thankless job. They work long hours. They rarely get to go to conferences and seminars unless they’re local events.

They live in a rough world of office politics, budget constraints, dodgy environments, conflicting priorities and co-workers who — politely put — may not be bringing an “A” game to the table. Job security and career advancement is frequently optional.

I’ve found that the vast majority of enterprise IT people don’t pay attention to most of what’s being talked about in the cloud world — simply because it doesn’t relate to them and their world.

#3 — Legacy Matters

In most enterprise IT environments, they’ve already made huge investments in certain technologies: infrastructure, software, applications, etc. Not only have they spent a boatload of their scarce IT dollars on this tech, they’ve invested considerably in training their people how to use it effectively.

While it’s popular to disparage decisions made in the past, they are a fact of life. They are as much a part of the IT landscape as a mountain range — they just can’t be conveniently ignored.

Mainframes, anyone?

I encourage people to present their new tech in the context of how it works with what people already have, instead of assuming a convenient fresh start. While it’s true that a few fortunate IT groups have the ability to stand up legacy-free test beds to try out new things, that tends to be the exception rather than the rule.

#4 — Skills Matter

IT organizations are basically collections of people, each with a certain skill set. Indeed, instead of looking at the technologies a group has invested in, consider the portfolio of skills needed to make it all work.

Talk to any CIO group, and they’ll quickly acknowledge that managing the skills portfolio is a huge challenge.

Certain propositions that demand a critical set of very new and hard-to-obtain skills (e.g. devops, big data analytics, etc.) are simply non-starters for so many IT organizations, as they present a skills hurdle that seems to be near insurmountable.

Look at it from an IT leader’s perspective.

They’ve got only so many headcount. They’re not going to get any more anytime soon.

If you ask them to invest big in a new skill set, that means de-investing in something else that’s needed to keep the lights on.

And that’s a big ask.

#5 — Supportabilty Matters

It’s safe to say that successful enterprise IT organizations are incredibly risk-adverse. Trust me, they do that out of necessity, and not because they are genetically paranoid.

When something breaks (and it always does), it’s an incredible fire drill between upset users, limited IT staff and the supplying vendors. A well-established and well-supported technology will always be strongly preferred to one that isn’t — even if the new shiny thing is better, faster, cooler, sexier, etc.

That’s just one of the reasons larger IT vendors have such a clear advantage when it comes to enterprise IT — they understand just how important this is to customer.

As an example, being “open” is often less important than having the ability to support your enterprise customers when the going gets rough.

#6 — Relevance Matters

“If it ain’t broke, don’t fix it” — words to live by in an enterprise IT setting. I find that so many of the newer propositions just aren’t all that relevant to what enterprise IT people consider important.

For me, relevance breaks down into two categories.

The first category is “your users are demanding it”. Put mobile, big data, etc. in this category. Sometimes that new requirement comes with budget, sometimes not.

The second category is “we have to continually improve at what we do” — speed, costs, quality, etc. — and usually precious few resources available to invest in continual improvement.

Unless whatever is being proposed clearly falls into one of these two categories — and doesn’t come with a bunch of external complications and costs, it’s probably a non-starter in the enterprise IT world.

Enterprise IT Isn’t Going Away Anytime Soon

There are those that will endlessly argue that “it’s all going to the cloud”. And that might potentially be true — in the very long run.

While we’re waiting for utopia to materialize (built of unobtanium, naturally), we are left with an enormous opportunity to help enterprise IT organizations get better at what they do.

Building Bridges, Not Walls

So many of the new web-scale tech vendors seem to be putting up walls to adoption by enterprise IT organizations. Maybe that’s intentional (e.g. knowing your target market), maybe that’s inevitable. Maybe selling to enterprise IT organizations just isn’t sexy enough for some people. I don’t know.

There are certain IT vendors (my employer VMware included) that think in terms of building bridges between the existing enterprise IT world, and the emergent world of web-scale IT.

The idea is to give people technology that makes their existing world a better place (e.g. virtualization) and then give them a defined path to walk to a new web-scale world — when they’re good and ready to, that is.

... But Please Don’t Stop

As I’ve said initially, all the new tech that’s being created as web-scale emerges as a distinctly differentiated form of IT is very cool, and is an impressive engine of innovation. I’d hate to live in a world where we didn’t have the benefit of all those cool ideas to consider and understand.

Just don’t expect them to be adopted in most enterprise IT shops anytime soon :)

I had a good introduction to the Hindu pantheon at an early age, and I keep thinking of cloud as Shiva The Destroyer -- knocking down the old IT world order, and creating the opportunity for the next.

Simply put, cloud changes all three aspects of traditional IT: the technology model, the operational model and the consumption model.

Everything we know is being remade, and will be remade again and again.

Anyone who claims to have this all figured out is either being presumptive, or is far smarter than I.

You can appreciate my fascination :)

Private Clouds Are Here To Stay -- And Apparently Growing

One of the more vitriolic debates amongst the clouderati has historically been the role of private clouds.

There were many who just couldn't grasp why IT organizations weren't getting out of the business of owning IT assets, and instead consume 100% of their IT needs externally.

For those of us who've spent their career working with enterprise IT organizations, we know the reality: there are very strong structural motivations for private clouds: economics, control, etc. -- and those aren't going to change anytime soon.

I'd like to be able to point to some reasonable industry analysis that backs up my assertions, but I can't. Since there is no generally accepted model for what is a private cloud (and what isn't), we're left with reading tea leaves that act as real proxies for the action going on here. My informal sampling of customer groups is right around 90%: just about everyone has a part of their environment they refer to as their "private cloud" or similar.

One indicator might be virtualization adoption. Hypervisor technology within the enterprise is now almost ubiquitous. I even meet customer groups that are somewhat embarrassed that they're "only" 80% or 90% virtualized. Who would have believed that just a few short years ago?

Another proxy might be purpose-built infrastructure. VCE appears to be setting the pace for data-center-class converged infrastructure (now a >$1B business) with both HP and IBM hot on their heels. Moving down a bit, the reference-architecture builds (FlexPod, VSPEX, et. al.) are now a mainstay of the industry, routinely called out during earnings calls. Moving down even further, the pre-fab node model continues to be popular to many (think Nutanix, SimpliVity and perhaps the oft-rumored Marvin?).

All can be considered reasonable proxies for private cloud adoption. And, collectively, they represent many billions of dollars of IT spend, and appear to be growing quickly.

But there are better proxies: automation software for example.

For me, cloud is primarily an operational model, and that can't be achieved without a non-trivial investment in new workflows and tools.

I was pleased to see a recent IDC report that stated the data center automation software market grew an impressive 22% from 2012 to 2013, reaching a healthy $1.8 billion. I think it's safe to presume that most of that is going towards automating newer, cloud-like environments, e.g. private clouds.

Unsurprisingly, VMware emerged victorious with a #1 spot of 24.7% market share and an impressive 65% growth rate. That's momentum. However, a quick look will tell you it's still a very fragmented market indeed, with the next three vendors (IBM, BMC, HP) responsible for around 35% of the market.

Much Ado About Amazon Et. Al.

While there are some truly good industry journalists out there, some of them swim in the shallow sensationalist end of the pool. A recent example was Amazon's offering of a vCenter plug-in to enable easier provisioning of AWS services using in-context tools.

You would have thought a simple piece of plug-in software represented a mortal blow to VMware, given some of the coverage.

As many readers will know, provisioning on AWS and moving a handful of simple VMs across has long been relatively straightforward. The bigger challenge has been with the other 98% of the model: monitoring, moving stuff back, etc. -- all the gritty operational stuff that no one really understands until they've had to deal with it.

And nothing has really changed there.

By comparison, the defining attribute of VMware's vCHS (vCloud Hybrid Service) is that all-so-important operational compatibility. If you're running VMware tools, it runs as an extension of your existing environment.

Avoiding the over-used word "seamless", it's pretty darn close.

I've seen customer reaction as uniformly positive to the existing vCHS offering, and there's a boatload of cool enhancements in the roadmap if you can get someone to share it with you.

The public cloud business is the essentially same as any other: understand your customer and what they really need. The vCHS team seems to have done that.

One of the unverified memes floating around inside VMware is illustrative: if we can get 10% of our installed base to use vCHS in some fashion, it will be the industry's second-largest public cloud.

By comparison, my compatriots at HP have announced a new cloud strategy, one that appears to be challenged from the outset.

Part one is a massive and public investment in OpenStack. That's a good thing. Part two is to create an alliance of independent IT service providers to use HP's modified OpenStack software tech (Helion) to build their own services, often referred to as the "arms dealer" model.

That one isn't so good -- and I speak from direct experience.

When I was at EMC, we had a similar game plan. The first incarnation (VMware's VSPP) was only modestly successful in achieving its original goals. There were a number of issues that were difficult to overcome, but the biggest one from my perspective was that every service provider wanted to change the recipe.

Maybe they thought they could do better on their own, maybe they thought they had to differentiate from all the other services, or something similar. Some have become successful (CSC's BizCloud comes to mind), others less so.

During that period of time, I could see that customer reaction wasn't strongly positive. Yes, all the service provider partners were using essentially the same tech, but the resulant customer experiences were all over the map.

Say what you will about McDonalds, but when you walk into one you know exactly what you're getting. That's what customers have told me they wanted: consistency and compatibility of experience -- wherever they go.

I wish HP the best of luck.

Where Does OpenStack Go From Here?

Before anyone feels obligated to flame me, I'm a big OpenStack fan. My company -- VMware -- sees OpenStack as a clear opportunity to sell their software tech using a different integration framework. My previous company -- EMC -- has been genuinely enthused by the platform and the interest.

That being said, few would take the position that OpenStack-based products are ready for prime time. Not many IT organizations are prepared to invest the engineering muscle required to make OpenStack production-capable.

The current situation is analogous to Linux in its early days: everyone can see the potential, but it's just not soup yet.

Two things to keep in mind as we consider the proverbial OpenStack "tipping point".

First, the pressure on IT organizations to cloud-ify their environments is nothing short of intense these days. People want results, and they want them sooner than later. And few can afford to delay heavy investments in tech and process waiting for OpenStack-based products to mature. Once their environment is operational (hopefully with a majority of VMware products), it's tough to make a case to come back and do it all over again anytime soon.

Second, we're likely going to see far more "productization" of OpenStack code in the near future -- mixes of open source and proprietary software, with the usual constraints on what's supported, and what's not.

Building software products costs real money for real engineers, which will inevitably negate some of the perceived economic (and "open") advantages some are hoping for today.

Will we see more OpenStack-based private clouds in enterprises in the future? Bet on it. But it's going to take longer -- and perhaps be more moderate -- than most people expect.

And On To PaaS

It was a distinct pleasure to follow the recent Cloud Foundry Summit online. A thousand people showed up -- folks who are passionate about what CF can do for them today, and in the future. I found many of the stories impressive. And the momentum appears to be accelerating.

The big idea? Perhaps the most important thing PaaS (and Cloud Foundry) does is abstract application from infrastructure in a meaningful and useful way. Developer productivity accelerates, significant applications can be deployed far more quickly, and there's far less dependency on the specifics of the underlying infrastructure.

In a post-cloud / multi-cloud world, it's the next frontier.

Not everyone does significant application development, but many do. And, as business models change in a digital world, more and more organizations will find themselves in the software development business.

That means considering a strategic investment in your software "factory".

From my biased perspective, it seems that -- right now -- it boils down to two choices for modern enterprise PaaS: Cloud Foundry ... or -- ??? -- well, I really don't know. The Cloud Foundry Foundation strikes me as an elegant governance model to ensure the direction remains aligned with stakeholders, and it's garnered substantial support (34 members making sizable contributions) in its first few months.

What I really liked about following the event was that there were plenty of real-world examples of the tech being used today to deliver eye-popping results. Let's hope the strong momentum continues.

It's Still Early Days In Cloud

There are plenty of commentators who are willing to claim "game over" when it comes to cloud.

Horsefeathers. It's as short-sighted as claiming "game over" for the internet, mobility, etc.

Just when you think you're beginning to understand something, it morphs and begins again.

We're in the opening chapters of what will prove to be a long and exceptionally interesting story.

Anytime a new concept enters the marketplace, the inevitable Definition Wars begin.

Understandably, it takes a while for the industry to collectively wrap its head around a consensus perspective regarding a new subject.

When it comes to one of my personal hot topics (software-defined storage), I do try to patient, but I fail. I despair as I encounter many of the flaccid definitions currently in vogue.

Maybe my standards are too high?

Making matters more irritating is the seemingly endless army of chirpy storage marketing types looking to slap a fresh label on familiar products and technologies — inserting even more noise into an already weak signal.

In this post, I’m going to resist the temptation to assert a black-and-white definition, and disclaim all others. The reality is that software-defined storage is a cluster of related concepts: some fundamental, others more optional depending on requirements.

Depending on your situation, good enough may be good enough.

As I walk through the list here, I’d encourage the motivated reader to assemble their own personal list of attributes he/she considers important in their emerging environment.

Essential: Programmability Of Behavior

If nothing else, software-defined storage should be capable of being entirely under external programmatic control.

That’s a long-winded way of saying that anything that can be done by a human can be done via an external program, presumably using an API.

This is the essence of software-defined: behavior is determined by external software control — typically an automation framework. You 'll find essentially the same concept in software-defined networks, and software-defined compute (e.g. server virtualization).

Without this one key attribute, there won’t be the basis for effective coordinated automation, and that’s one of the primary goals here. Even better if a single API framework could be used to control many flavors of storage vs. one per vendor.

Frustratingly, there are “software-defined” storage vendors today that offer little-to-no API capabilities. When pushed, they say their homegrown management tools are all anyone needs — and that’s software, isn’t it?

Make it past this initial (and essential!) hurdle, and we can start to differentiate better (and more useful) forms of software-defined storage.

Very Desirable: Dynamic Vs. Static Resource Model

The vast majority of storage products today use essentially static resource models underneath the covers. The storage administrator must determine — well in advance — what different applications might eventually need, then buy a bunch of stuff, then carve it into pre-provisioned resource pools with different service levels (capacity, performance, protection, etc.), and then advertise it for consumption.

You can see where problems arise.

If an application’s requirement doesn’t fit nicely with one of the pre-established storage service levels, that’s a problem. If an application’s requirements move up or down outside the range provided by the pre-defined storage resource pool, that’s a problem. If actual aggregate demand doesn’t line up with the buckets that have been forecasted, that’s a problem.

Far better if we can pool resources, and use them to dynamically create (and adjust) storage service levels as actual demand materializes.

Very Desirable: Storage Services Aligned On Application Boundaries

IT is all about applications, and that’s how users see the world: through the lens of their applications. Ideally, we’d be able to tailor and adjust services (including storage) along precise application boundaries.

More for this application, less for that one.

Today’s storage has scant knowledge of applications or their boundaries; it sees the world as LUNs and filesystems, each one of which offers up a combination of capacity, performance, protection, etc.

A better model would be one where storage containers are aligned to application containers, and storage services can be specified for a given container and no more.

Very Desirable: Automation Driven By Application Policy

We all know there’s a big difference between saying what you want, and having to explicitly spell out exactly what’s required to achieve a goal.

In today’s storage world, we typically have to be very explicit: use this much cache/flash, tier under these conditions, protect using a combination of RAID 5 and remote replication, etc.

Better yet if that same application policy could serve as a manifest for compute, networking, security, etc. and not be unique to storage.

Desirable: Works With External Storage Arrays

There are many who see “software-defined storage” as synonymous with “only runs on commodity servers”.

That’s far too restrictive in the real world: storage arrays will be with us for a very long time indeed, and they represent the vast majority of storage spend these days.

Definitionally excluding them doesn’t make logical sense, nor is it a pragmatic view. To the extent a storage array can support the functions described here, it should be seen as full participant in any software-defined storage environment.

Desirable: Runs On Commodity Servers

That being said, there’s enormous interest in software-only storage products that use familiar servers to offer storage servers. It’s an important option (just like external storage arrays) that our ideal software-defined storage environment should support.

Perhaps now would be a good time to call out the distinction between software-based storage (runs on commodity servers), and software-defined storage.

As an example, there are more than a few software-based storage products on the market that don’t even begin to qualify as software-defined storage (e.g. no APIs!) using the guidelines presented here.

Ideally, software-defined storage would support both traditional external arrays as well as newer, software-only storage stacks — and use a consistent framework for both.

Somewhat Desirable: Works With Older Storage Arrays

You might find yourself arguing that this attribute should be essential, but let me defend myself: storage gear has typically had a useful life of 3-5 years, meaning that in larger shops there’s a conveyer belt of new stuff coming in, and older stuff going out.

In the majority of cases, the stuff you have sitting on the floor wasn’t designed to participate as part of a software-defined storage environment — at least, not without an additional storage abstraction layer.

Somewhat Desirable: Supports Existing Workflows

A big benefit of software-defined storage is that it can support converged operational workflows: where all resources are managed by an application-centric policy. That’s the big win with software-defined data centers.

While all that sounds great, the reality is that most of the storage world today runs on manual workflows and tools that have been around for a while. During the adoption period for SDS, it’s somewhat advantageous to support some of the existing processes while experience is being gained with the newer ones.

That being said, there seems to be no shortage of greenfield environments being stood up with clear breaks from legacy approaches, hence me believing this attribute is somewhat optional.

The VMware Perspective

For those of you who know my situation, you might think I’m simply reciting the VMware party line when it comes to software-defined storage: VSAN, vVols, SPBM, vCAC et. al.

You’re right, I am.

I guess I’m fortunate that my personal perspective and my employer’s perspective overlap. That’s the big reason I work where I do.

In my eyes, VMware “gets” software-defined: for compute, for networking — and now for storage.

I’d love nothing more than to spill the beans on what’s being worked on over the next few years. We’re not just talking about incrementally better tech; we’re talking about entirely new models.

All I can say is that I’ll have plenty to blog about for years to come.

If You’ve Made It This Far ...

... you’re probably intrigued with this stuff, just as I am.

Now is a good time to create your own informed opinion on what software-defined storage might mean to you and your organization. At this point, though, you’ll have to be good at filtering noise, and amplifying the important signals.

There's some history here that's worth considering. As we all know, what it meant to be a “server expert” changed dramatically when virtualization became prevalent.

I felt I owed everyone a post about an illustrative incident that happened over the last few days.

As we move to more software-based storage stacks, there are some interesting questions arising about the new responsibilities of the various parties involved: software vendors, hardware suppliers, partner and end users.

There’s a new model emerging for building storage solutions, and we now have a well-documented example of some of the inevitable bumps along the way.

What Happened

Last Saturday morning, as I was drinking my morning coffee, I saw that Twitter had a number of tweets referring to a formidably titled Reddit post “My VSAN Nightmare”.

For the TL;DR crowd -- Jason added new hardware as he was replacing a failed server, and the cluster became IO saturated for many hours as it rebuilt unprotected objects. After working through its backlog, the cluster became usable again, with no data loss or corruption. But it was a rough stretch there for a while.

As I tell people, there's no bad day in IT quite like a Bad Storage Day.

Time To Mobilize

As I work closely with the VSAN team, I immediately started an email chain, but I wasn't the first. The good news is that VMware’s support organization (GSS) had already engaged near-flawlessly and got the customer back up and running. We also had access to a good sampling of logs and traces — essential when you’re trying to figure out what’s going on.

One of our lead engineers (Dinesh) spent his weekend with others getting to the bottom of what happened. By Sunday morning, we had our unequivocal smoking gun: the IO controller being used (a Dell H310) couldn’t keep up with the load generated by the rebuild. VSAN first tried to throttle the rebuild, and then started to throttle application loads — to a degree that made the cluster very unresponsive.

The official response is here on Reddit — again, worth reading if you’re interested.

Now What?

A bit more background will help with the picture. First, Jason was running a moderate configuration: not a lot of disks, and not a lot of IO. He said he was quite delighted with the performance of his VSAN cluster when things were running normally. All good.

More to the point, the choice of IO controller really matters when it comes to sustained IO performance. Even if you have a flash drive caching — and enough spindles — everything comes through the IO controller. These controllers manage their work using a queue for outstanding IOs, which are then dispatched as needed to various storage devices.

In a nutshell, when it comes to sustained IO, queue depth matters. Low-end controllers may have a queue depth of only 25; better ones more than 1000. The deeper the queue, the better the IO controller does at keeping all the storage devices busy. Queue depth issues won’t manifest themselves unless you’re trying to drive a lot of IO — say, during a node re-protect?

The VSAN software did exactly what it was designed to do — again, almost flawlessly. When the new storage capacity appeared, it waited the proscribed 60 minutes, and then start to reprotect objects that had a FTT=1 (failures to tolerate) policy. During a reprotect, VSAN may want to copy a *lot* of unprotected objects — potentially many terabytes — and do so relatively quickly.

But the hardware has to be there to support that.

When the IO controller got saturated, VSAN first backed down the reprotect (as designed), but insisted on minimum forward progress. That requirement for a minimum rate of progress is a desirable design goal, as the storage objects are now unprotected and completely vulnerable to a second failure. In this case, when insufficient forward progress was being made, VSAN then started to throttle production applications — again, as designed.

And that’s when the problems started.

What Really Went Went Wrong

On the VMware VSAN HCL, we list the Dell H310 card Jason was using. After all, it works fine — up to a point. As Jeramiah Dooley points out, there’s a difference between “it works” and “it meets my needs”.

In a completely separate document, we state that VMware recommends a minimum queue depth of 256 for “optimum performance”. There is no information published by VMware on the queue depth of the Dell H310 we list, or an explicit warning regarding the implications of using this entry-level card.

Making matters worse, IO controller vendors don’t routinely disclose the queue depth of their products: the usual procedure is to either scour the ‘net or run a specific utility. As an example, Duncan Epping has the beginnings of a list here.

You have to know what to look for — and also know what might happen if you don’t.

And, as a final twist of the knife, it seems that you can re-flash this card with the original LSI firmware, bringing the queue depth all the way up to 600 :-/

In my opinion, that’s a lot to ask from people who might be new to storage.

Jason — like perhaps many others — took a look at the VMware VSAN HCL, found a suitable (and inexpensive) IO controller card, built his cluster — and was quite pleased with the application level performance. He was unaware that — during an extended rebuild operation — the IO load would increase dramatically, and it would potentially bring his lightly-loaded cluster to its knees.

That’s not good.

The same kind of problem could potentially arise from — say — having insufficient IOPS (spindles) in a disk group. While the flash cache is fast enough, all those writes made during a reprotect need to end up on disk sooner or later. And those 4TB NL-SAS drives aren’t exactly speed demons.

That being said, if someone knows exactly what they’re doing, using low-end IO controllers and/or low-performance storage devices should be supported. That’s one of the appeals of software-based storage — choose your own hardware.

What To Do?

The most restrictive solution would be to limit the HCL to only devices that offered sufficient application and rebuild performance, and have some sort of exception process if desired. But that feels overly restrictive to me.

A better solution would be to clearly mark low-performance HCL entries (maybe with a turtle logo?), explicitly warn people that these devices may not offer enough performance for applications and (especially) rebuild scenarios, along with a link to more information: queue depths, why they matter, VSAN’s reprotection behavior and impact on sizing guidelines, and so on.

Growing Pains

Coming as I did from the array side of the storage business, this sort of thing doesn’t usually happen. In addition to sizing for capacity and application performance, recovery and rebuild scenarios are fully considered as part of the storage sizing exercise.

That being said, I have been witness to a few spectacular incidents when the guidelines weren’t followed, and similar applications-on-their-knees outcomes resulted.

At the end of the day, the hardware resources have to be there to do the work required — and that includes failure scenarios.

What’s fundamentally changed with the software storage model is that the system builder (e.g. customer or partner) is now responsible for sizing their environment correctly — and not the hardware vendor.

And now we as storage software vendors need to arm everyone with information and warnings to help make the right choices, and protect them from making poor ones.

Much more work to do here.

My heart goes out to Jason Gill for his decidedly unpleasant experience. We all want to thank him for thoroughly documenting it for others to learn from.

It looks like most decent-sized enterprise IT organizations have either begun their transformation to an IT-as-a-service model, or are well along their path. Maybe it’s sampling bias on my part, but it’s rare these days to meet an IT team who isn’t doing at least something with the concepts.

At some point along the way, the quintessential subject of automation emerges as a Really Important Topic. And — sad to say — there is no “automation pill”.

A few weeks ago, I was asked to share my perspective on the different “cloud automation” packages that are out there. As a VMware employee, I would have been quite justified in singing the praises of vCAC, and dissing all alternatives.

But I didn’t take the easy bait.

Instead, we got into a great discussion around the desirability of having an “automation philosophy” that could help IT leaders guide their choices and approach.

The Setup

I see the move to an ITaaS model as I do adolescence: sooner or later, it happens to everyone.

Yes, it’s awkward at the beginning, but things do get better as you mature :)

With ITaaS, IT refashions itself as the IT service provider of choice, a builder/broker of services the business wants to consume. Measurement systems change, core processes change, roles and responsibilities change — it’s a dramatic refactoring of familiar IT.

Automation is the secret sauce that makes ITaaS work. It is not an afterthought. The more you automate (and re-automate, and re-re-automate), the faster/cheaper/better IT service delivery becomes. New process insight drives new automation, which delivers better results.

It’s the gift that keeps giving and giving.

If you’re in the manufacturing business, for example, continual process engineering and its attendant automation is a way of life. The same appears to be true for enterprise IT these days.

So, at some point, the IT team starts investigating different “cloud automation” products. That’s where the fun starts. It’s not hard to come up with a list of 30+ vendors, each with their own marketing pitch, terminology, positioning, etc.

Things can get pretty confusing very fast, unless you’ve got some guiding principles in hand. And that’s what I mean by a “automation philosophy”.

Towards An Automation Philosophy

Here are the questions I’ve been using when discussing this topic with IT organizations. Feel free to think of these as a starting point — extend, modify and edit to your heart’s content:

1. Do you see yourself as a heavy automator?

Think out ahead three years or more.

Do you see continually improving automation as a really big deal in your environment? Or do you see yourself covering the basics, and then moving on to more pressing challenges?

Consider the familiar trope of provisioning a server. The difference between five days, five hours, five minutes and five seconds might not be relevant in your world. Or extremely relevant, depending.

There’s a strategic choice to be made between creating processes that are “good enough” that they get the job done — and continually investing in process improvement and the required automation.

For some, the answer will be “good enough is good enough”. Fair enough. For many others, that won’t be an option. You don’t have to be Google to appreciate the payoffs that can come from increasingly sophisticated levels of automation.

2. Do you see automation as extension of your existing model (improving existing processes and workflows), or working towards creating new models (as well as new processes and workflows)?

Another key decision point. Incremental improvements can certainly be achieved simply by automating existing processes and workflows, with no significant model change.

With this approach, everybody does what they’ve always done, but simply use better tools to get their job done.

My personal perspective is that the best automation packages do both: they support existing models and workflows, but also create the opportunity for entirely new converged operational models when desired.

If you’re enamored with the breathtaking potential of software-defined IT, you probably realize that significant model changes are demanded to reach its full potential.

And that aspect — if relevant — should be part of your automation philosophy.

3. Are you willing to change your organizational model to get better automation?

Look at any traditional IT org chart, and you’ll see the silos (or cylinders of excellence if you prefer) around classical IT disciplines. Without changing the organizational model, the best you can hope for is automating the silos, and not how they interact to create services.

Technology in IT is becoming the easy part; re-engineering IT organizations to function more effectively is where the real heavy lifting occurs. And you should decide at the outset — are you and the team up for this?

Show me any IT organizational org chart, and I can quickly assess (a) where they are in their transformational journey, (b) what level of automation they already have in place, and (c) exactly what kinds of pains they are experiencing when it comes to improving the situation.

4. Do you know enough to make intelligent long-term choices?

When I started seriously playing in bands a few years back, I didn’t have the right equipment, and I didn’t know squat about what I needed.

So I scraped together a basic, inexpensive rig knowing full well that — as I did it more — I’d better understand (and invest in) what I really needed to do the job. Everything I bought initially I thought of as “disposable” — an investment in me getting smart about what I really needed.

When I did finally pull the trigger on an expensive keyboard rig, I knew exactly what I was doing, no regrets.

As I worked with IT organizations starting to move towards ITaaS, I found more than a few who had adopted a similar “disposable tool” philosophy at the outset.

The thinking was simple: invest in learning at the beginning, which will in turn lead to better (and longer-term) choices. Most often, this was done by a centralized automation team that spanned traditional technology silos. Maybe they didn't stick with particular products over time, but their knowledge and learning was certainly recyclable.

No major investments, no long-term commitments — until you feel you’re smart enough to make a wise decision.

No surprisingly, every software vendor would prefer you to bet, and bet big. And, in many situations, that can be advantageous. But if you’re just starting out on this whole ITaaS automation thing, you may not know enough to make an informed decision.

So invest in getting smart :)

5. Will you eventually see automation as software development?

If you’re going to be serious about automation, you’re going to be writing code in some form. If you’re really serious, you’ll be writing a lot of code. And, depending on your automation philosophy, that’s a good thing.

That’s the whole #devops thing in a nutshell: infrastructure automation as code.

Viewing automation as a software development activity makes you consider things like code re-usability, testing environments, release management, documentation, and all the other disciplines that are part and parcel of a software development model.

This view also implies that you’re hiring infrastructure process engineers with a software development background. And that’s not exactly a garden-variety skill set.

6. Does this whole open source thing really matter to you?

Nothing brings out the passion in industry types like a good debate on the pros and cons of open source models. Open source automation tools are no exception — people get pretty worked up.

I tend to be pragmatic in my views.

When it comes to automation, what you’re essentially doing is encoding process know-how so that it’s reusable and extensible. If you’re taking this whole automation thing seriously, you’re talking about a repository for a continuing investment of many person-years (previously known as man-years).

A close analogy might be ERP, or SAP in particular. An environment like SAP can be viewed as a repository of business process know-how: lots of standard templates extended with unique secret sauce. SAP is clearly proprietary software (and wildly successful), but it’s extensible, portable, integrate-able, etc.

I tend to focus on how well the tool does the job, rather than how the tool is licensed.

I do meet very specific IT shops that are fully committed to using as much open software as possible. The ones who are successful bring two important things to the table: a long-term perspective, and lots of software engineering types to make things work.

However, large amounts of time and people aren’t always an option …

7. Is there a place where you can get started?

It’s often the case that we think about automating everything. And, since all the legacy pieces have to move together, it’s a painfully long journey. Not to mention intellectually challenging.

But many IT shops have been successful in standing up a next-gen environment that’s a clear departure from the legacy: new processes, new tools, new roles, etc.

I’m quite enthused when I meet more than a few groups that are on their second (or third!) iteration of their vSphere-based “cloud”. That’s progressive.

If you have such a portion of your environment, consider yourself fortunate — that’s where you can learn and move quickly on ramping your automation muscles.

But if you have no such environment, your approach to automation will be dictated by legacy concerns, meaning slow and incremental progress vs. substantial breakthroughs.

And -- it has to be said -- it's devilishly hard to automate environments that weren't designed to be automated :)

Putting All The Pieces Together

We in the IT biz are famous for our strongly-held opinions. Throw a bit of red meat out there, and you’ll be impressed by the reactions.

When it comes to automation, I believe IT leaders need to arm themselves with strong opinions.

Building an IT delivery environment that is continually improving (based on continually improving automation) is a central strategy issue, and deserves careful and deliberative thought.

Having a well-thought philosophy on the topic can help in quickly navigating the inevitable decisions and distractions.

My parents thought it an important milestone towards growing up, being more responsible, etc. I guess I didn't get that memo, as I was hoping for a slingshot at the time.

As I saw the wallet as completely useless, it sat in a drawer for many years gathering dust.

Today, I was buying gas.

I reached into my wallet to pull out a credit card, and — dang!! — the entire contents fell into the Crevice Of No Return: that slim gap between the driver’s seat and the center console. Down on my knees, fishing out small plastic cards, I became seriously exasperated.

And I was once again reminded of how utterly useless a men’s wallet should be in this day and age.

- two dozen plastic cards, each with my name and a set of digits encoded

- paper currency from a few places, depreciating slowly as all currency does

- taxi receipts I haven’t yet expensed — some are actually readable

- outdated pictures of my family — from a distant, sepia-toned era

Physically, all of this makes for a thick package that must be relocated from its traditional location on my hindquarters to another location when I’m sitting for a while: plane rides, longer drives, extended work sessions, etc.

Quite literally, it’s a pain in the rear.

Each of these physical artifacts could (and should!) be stored on a personal digital device, such as my smartphone. But the world has conspired in such a way to make this a near-impossibility in my lifetime.

And I am not alone. I would bet that — even the most annoying digerati among us — still has a collection of physical artifacts that must be kept on one’s person at all times and in all places.

I started thinking — what would it take to get rid of my archaic wallet, and all of its contents?

The Basics

Simply capturing all of this as digital information — as an app, or set of apps — is relatively straightforward.

There’s nothing on that list above that doesn’t already exist as a digital entity, with the possible exception of currency (bitcoins, anyone?).

My phone has a PIN. I use two-factor authentication to access my corporate network, and the secure content container that’s installed. By comparison, my wallet and its contents are physical entities and rarely require any authentication to use.

So don’t make this about “security”, please.

It’s Not Me, It’s You

The challenge, of course, is getting people on the other side of the transaction to accept a digital equivalent of what has been historically a physical entity. They need a clear incentive to change their ways -- and, in so many situations, that's missing ...

Example #1: I walk into my doctor’s office, and present my health insurance ID card. The receptionist takes it from me, laboriously enters its information into an application, and then asks me a few trivial questions to (presumably) help prove it’s really me.

Trust me, I’d be more than happy to fire up an app and have a QR code scanned and enter a PIN.

Example #2: I go to buy gas. I notice that there are magnetic readers for digitized wands on the pumps, but they’re branded and specific to a single vendor.

Won’t work for me, as I prefer to buy gas when and where convenient, and I don’t want to carry around a magnetized stick just for the privilege.

Example #3: I’m on a long road trip. I get pulled over by a friendly officer of the law, and asked to show (a) my drivers license, (b) proof of vehicle registration, and (c) proof of insurance.

Well, in my state you can’t get registration unless you’re insured, so there’s never a need to show proof of insurance. But I’m not in my state.

Much hilarity ensues.

Example #4: I’m boarding a flight. My QR code (coupled with a physical ID) gets me past security and onto the plane. I notice many people who are still clutching many pieces of paper.

Now, if we can just figure out a way for me to ditch the physical ID …

Example #5: My relationships with my preferred retailers (Amazon et. al. ) are completely digital. There are no physical artifacts associated with me doing business with them, other than the occasional need to dispose of voluminous packaging materials.

Example #6: I bought a new house not long ago. I had to go out and buy a multi-function printer because the various entities insisted on communicating with me via fax. You do remember fax machines?

It's my money, can't I choose a provider that knows that it's 2014?

You can see the problem — until there’s near-universal acceptance of the digital form, we’ll all be forced to carry around physical artifacts that basically encode digital information. And that change depends on some sort of forcing function.

Ancillary Issues

Yes, if I lose my phone, I’m basically screwed for awhile. But the same thing happens when I lose my wallet — no real difference, except that the re-instantiation of my phone’s contents is infinitely easier than re-instantiating the complete contents of my wallet — assuming I’ve backed things up!

It should be common knowledge by now that there are big wins for those companies that provide all-digital relationships; better branding, reduced fraud, less churn, rich data to analyze, meeting consumer preferences, etc. After all, it's 2014.

More difficult are the near-monopolies where there is no competition, or any strong incentive to meet changing consumer needs. Most government entities fits into this category (law enforcement, airport security), as do captive relationships such as with my health care provider, certain financial services, or any situation where switching costs are high.

I’m Cynical

I’d like to think that society can move ahead in the presence of a better way of doing things, but the observed evidence doesn’t support my optimism.

Obvious measures take years, decades or sometimes generations to implement. Sigh.

Although it's a terse definition, it quickly unpacks into a very large and powerful set of of concepts. Similar definitions can be constructed for software-defined networking, software-defined compute, et. al.

Three important disclaimers:

Just because storage is implemented in software doesn’t make it software-defined.

Just because storage software comes packaged in an array doesn’t mean it can’t be part of a great SDS environment.

Just because a vendor uses the term liberally in their promotional materials doesn’t mean they have the slightest clue (many examples!)

As long as the key attributes of the SDS definition are satisfied, I would argue that -- regardless of specific technology -- you’re moving in the right direction.

The First Question: Is SDS Right For You?

Let’s face it: SDS is new tech and new process, and not everyone may be in a position to move ahead at this time.

I meet enough IT organizations to fully appreciate that not everyone is in the same position.

While virtually everyone aspires to better ways of doing things, the opportunity to do so is usually constrained by the pragmatic: the crisis du jour, a higher priority project, no funding, no resources, toxic politics, etc.

Server virtualization made clear sense for many years before it was widely adopted, as not everyone was in a position to move ahead at the same pace.

And, depending on your situation, the timing and situation might not be ideal for you to move ahead with software-defined storage.

That’s OK — things have a way of changing before long.

Check Your Motivations

No joke: doing something for cheaper is a powerful motivation when adopting new IT technologies. But exactly what are we doing cheaper here?

If your primary motivation is simply paying less for raw storage capacity, that benefit might be there for you — or not.

After all, parts is parts :)

However, if your motivation is more along the lines of a new operational model that’s more efficient and agile than today’s approach, that’s clearly there.

What’s “cheaper” with SDS is delivering better storage services, more quickly, with less effort *and* at a lower cost.

If your goal isn’t a better operational model (more agile, more efficient, less waste, etc.) I fear that you’ll ultimately be disappointed with the growing crop of SDS products.

Decide: Top-Down — Or Bottom-Up?

At this stage in the market’s development, there are two clear motions when it comes to SDS.

One is bottoms-up.

The idea is to create a software abstraction over existing traditional storage assets, and turn them into a more streamlined service that can be consumed for a variety of use cases: physical, virtual, etc. EMC’s ViPR clearly fits this mold. And, depending on your situation, this may look very appealing indeed.

But there’s another viewpoint — and that’s top-down.

In these situations, IT is building a cloud-like platform — maybe their second or third. They now fully appreciate the primary importance of automation and the associated control plane. Everything runs in a VM. The legacy isn’t a primary concern. And the new environment is presumed to be reasonably homogeneous.

For these people, the richness and expressiveness of the control plane matters, and (repeating myself here) the ability to dynamically compose storage services, aligned on application boundaries, driven by policy. VMware’s VSAN — and forthcoming VVols for external storage — reflect this line of thinking.

Start Small …

As I look at where VSAN is finding early traction in enterprises, it’s most typically a small piece of a much larger environment.

When I talk to these people, their motivations are clear — they want to get hands-on experience with a real software-defined storage product that’s designed to work with what they already have, e.g. vSphere and everything they’ve built around it.

There’s no substitute for hands-on: to understand how it works, what’s involved, key differences, where it fits, etc. As the initial investment isn’t unduly exorbitant (for VSAN, it’s a cluster of three or more servers compatible with the HCL + software licenses), the primary barrier ends up being investing in the time required.

… And Think Big

One of the more eye-opening exercises I was involved in years ago was tracing the flow of a storage service request for a specific company — from its origins in a business meeting where someone says “we’re going to need more space”, all the way to that storage service being stood up and ready to consume.

The surprise wasn’t really a surprise: something like 85+ discrete steps, spanning 9 different functional groups with an average elapsed time of 100+ days. Not to mention rampant over-provisioning.

Sounds like how we used to do server builds. And people wonder why public clouds are so popular …

Mapping out that same process in detail within your own organization might be a useful exercise. You’ll likely find that the full organizational cost of processing a request often exceeds the cost of the resource itself. Now, multiply that by the number of service requests in a year, and you’ll likely come away with a very big number.

If you’re thinking big — that’s the prize: building a better model. Satisfying storage service requests — quickly and precisely — using an efficient application-centric workflow — and consuming optimized resources. Sort of like Huffman coding for storage service delivery :)

Planning Your Journey

If you’ve stuck with me through all these long-winded articles, you hopefully have a new appreciation for what’s quickly becoming possible with software-defined storage.

Like server virtualization that came before, software-defined storage has the potential to transform how we think about storage in all of its facets.

The technology is becoming available; all we have to do is figure out how to apply it our individual environments.

The master craftsman Daedalus built wings from feathers and wax to escape Crete with his son Icarus. He told Icarus to not fly too close to the sun, as the wax would melt. Icarus — of course — ignored his father's warning and fell to his death.

Gravity wins again.

Without being unduly over-dramatic, I see the same scene played out countless times for those of us that work as IT vendors. Many of us aspire to be truly consultative to our customers and clients — but the gravity of having to sell specific products and services (with assigned quotas) inevitably drags us down to where we started.

If we fly too close to the sun, we can fall very far indeed.

But it's somewhat paradoxical: the IT industry is changing fast. Simply having a better widget isn’t any guarantee of success; what really matters is your customers’ ultimate success. And that demands a consultative approach.

We often talk the talk, but walking the walk is proving to be much more difficult.

Things Are Changing In The IT Biz

These days, IT transformation is becoming the norm.

The “third platform” (or cloud-mobile if you prefer) is now a real thing. The game is on: slash costs on the legacy, and invest like crazy in the new stuff.

At the same time, the fundamental enterprise IT model has to be re-invented: to a builder-broker of value-added IT services delivered with a healthy dose of relevant business context. The new IT business model drives a re-thinking of the organization model, which in turn drives the selection of technology and vendors.

During times of transition and transformation, most IT shops aren’t exactly clear on what they need, or why. And confused customers don’t buy stuff.

A quick story?

Many years ago, I realized it was finally time to get serious with my personal finances. I knew I should be thinking about investments, portfolios and all of that, but wasn’t quite sure how to go about it.

I found a financial planner who patiently walked me through different models, and how those models translated into specific financial products. We came up with an executable plan.

And, yes, she ended up selling me a bunch of stuff — because I knew exactly what I was buying, and why.

This demand for exec-level consultative advice isn’t lost on the better customer-facing people at most IT vendors. They can sense that their clients are looking for a model, a strategy, a plan — something that can be executed — which then later translates into specific requirements for products and services.

Because the IT companies they work for aren’t meeting this need well, they may set about industriously fashioning their own consultative wings from feathers and wax.

And — like Icarus — at some point gravity wins, and they find themselves falling back to pushing products.

The Other Side

I’ve worked for several IT vendors over the last 30 years, and know a lot of people who are doing the same. It doesn’t matter where you work, there’s always a clear metric on how your company is doing: the stock price.

That, in turn, is driven by revenue growth and future prospects for more of the same. Fail to feed the revenue beast for even one quarter, and the stock market will punish you accordingly.

Get punished multiple times, and you’re now M&A bait for a larger company. As most senior management are also shareholders, the incentives should be clear. Investors aren't really into delayed gratification.

It’s not that field-facing organizations are blind to the consultative relationship their customers crave; it’s just that the measurement system works so strongly against it. I can’t tell you how many sales planning meetings I’ve been in where the two, incongruous thoughts are shared at the same time: “We need to be more consultative” and “how are we going to make the numbers this quarter?”.

Many IT vendors have attempted to close this gap by establishing a professional services arm. If the customer desires a consultative relationship, here are the people in our organization who do that sort of thing. And some of them I've met are very good at what they do.

That being said, these people can’t escape gravity either.

Consultants are measured on utilization and billable hours: they’re pushing a product, and the product is their people.

Professional services organizations embedded within IT product vendors face an additional gravitational pull: the sales team wants to close a product deal as soon as possible, and may often perceive a consultative engagement as an unnecessary delay.

And pity the unwary IT vendor consultant who naively makes a different recommendation than what’s already been proposed by the product team ...

Who Will Close The Gap?

You might think that the larger consulting and system integration firms are well-positioned to close this gap for enterprise IT organization.

And they do — after a fashion.

If I’m being critical, they often aren’t as deeply knowledgeable about the technology (past, present, future) as the vendors who build the stuff.

More importantly, I think they aren’t exactly incentivized to transfer important skill sets and intellectual property, which — of course — is precisely what the customer ultimately wants.

Another View

One way of looking at the IT industry landscape is to categorize the various players in terms of the intellectual property they bring to the table.

Vendors tend to understand product technology: what it is, how to use it, where it’s going, etc.

Consultants understand business process and analysis: where are you today, where do you want to go, how will you get there?

And service providers understand operational process — how to run IT like a business, and offer services that people want to consume. All useful and necessary.

You can see the gap from a CIO perspective — who can explain how these three disciplines should come together around a specific enterprise IT context? All three are important, but it’s the combination that is desired.

I'd like to have an answer, but I don't.

However, I do know that the IT vendor(s) that figure out how to do this effectively — and sell a lot of product or service in the process — will likely escape gravity altogether.

In the early days of cloud, we were beset by competing (and confusing) definitions, often delivered with generous helpings of vitriol and rancor.

When it comes to software-defined storage, things are surprisingly genteel: I yet to witness a good #twitpiss on the topic.

Maybe it’s too early?

As part of this series on SDS, I think it’s worthwhile to share and critique other definitions besides the framework we’ve established here. While there seems to be no shortage of frustratingly shallow marketing pieces that glom onto the term; I have been laboring to find a reasonable alternate framework to compare and contrast.

Fortunately, someone sent me a link to a short PDF produced under the auspices of SNIA — the Storage Networking Industry Association. Although it’s only a working draft, it does show that reasonable people can get together and come up with a definition that’s quite different than the one presented here.

A Bit Of Background

If you’re in storage, you know SNIA has been around for a while. While not exactly a monumental force in shaping the industry, we can thank them for key contributions such as SMI-S, CDMI as well as some great training.

As I was reading the document, I found myself disagreeing with many of the tenets and definitions. Although there’s a form to provide feedback, I thought I could do a better job in a blog post.

Anything produced by SNIA is largely a collaborative effort from vendor employees who have donated their time.

And you know us vendors: we don’t agree about much — which shows in this document.

Minor Quibbles Abound

It didn’t take long for me to get critical: the document opens with a long list of attributes that may (or may not) be associated with software-defined storage.

If you’re looking for a crisp definition from SNIA, it isn’t here.

The following are attributes of SDS that are typically seen in the market:

May allow customers to “build it themselves,” providing their own commodity hardware to create a solution with the provided software.

May work with either arbitrary hardware or may also enhance the existing functions of specialized hardware.

May also enable the scale-out of storage (not just the scale up typical of big storage boxes).

Nearly always includes the pooling of storage and other resources.

May allow for the building of the storage and data services “solution” incrementally.

Incorporates management automation.

Includes a self service interface for users.

Includes a form of service level management that allows for the tagging of metadata to drive the type of storage and data services applied. The granularity may be large to start, but is expected to move to a finer grained service level capability over time.

Allows administrators to set policy for managing the storage and data services.

Enables the dis-aggregation of storage and data services.

If we go back to the definition of software-defined storage presented here (dynamic provisioning of storage services, aligned on application boundaries, driven by policy), you might see a few areas where the descriptions overlap, and many where they don’t.

If I’m being positive, I’ll note the use of the “policy” word, although I suspect they’re thinking of it very differently. The pooling and disaggregation notions are useful, but not essential to a good definition.

If I’m being negative, I’ll point out that there’s no mention of dynamic storage services, nor any concept of aligning those services on application boundaries.

And then we get into metadata … the SNIA definition proposes using metadata to implement the control plane for SDS.

Ruh roh!

The Metadata Trap

Metadata is simply data about data. Imagine a set of descriptive tags associated with a storage object (such as a file) that lists what it is, how it is created, how it should be handled, protected, etc. No need to figure out what to do with a piece of information — just read the metadata!

I had a long and tempestuous affair with metadata concepts a decade ago, and it didn’t end well. It’s not hard to arrive at the heady conclusion that metadata could solve the world’s storage and information management problems!All we have to do is teach the world to generate, manage, store and interpret metadata effectively. Simple enough!

When it comes to software-defined storage, the SNIA team has apparently fallen into the metadata trap. I don’t blame them — I’ve done it as well. But I did emerge from the experience a bit wiser and certainly more jaded.

What the SNIA definition gets right is that the application should be the boundary for defining storage service requirements.

Where I think they went wrong is their choice of mechanism: the conveying of requirements via attaching metadata to storage objects — presumably using the CDMI standard offered by SNIA.

While there is nothing inherently wrong with tagging storage objects, it’s not a dynamic mechanism. Labels are presumed to be static descriptions; their interpretation is contextual. The proposed mechanism doesn’t acknowledge the real world of rapidly changing requirements, nor the lack of precise semantics.

A simple example?

I, the application developer, create an application that uses a data store, and I tag it as “my application’s data store” or similar. Maybe I even populate it with some initial values specifying capacity, performance and protection.

But in reality, the set of storage services my application needs (performance, protection, etc.) are entirely dependent on the context: are we in development, are we in beta, are we in production, is the application now more important, is it the end-of-quarter, is it running slow, are we retiring the application, etc.?

As an application owner, my only communication mechanism with storage is solely a set of metadata tags. Will I re-write the necessary labels as needs change, or simply demand a re-interpretation of existing labels? What if multiple applications are using the same information, but for different purposes?

Either way, you come away dissatisfied by this proposed mechanism — and that’s before considering an implied mandate to use object storage for all storage use cases.

What’s Really Going On Here

There are two ways to view software-defined storage: as an independent entity, or as an integral part of a software-defined data center, alongside networking and compute.

If one insists that SDS be a standalone, self-contained entity, it must then be in control of itself, and not dependent on an external control plane to make decisions. That implies that SDS needs data structures to record how to handle individual storage objects, which leads you down the path to metadata and an object storage model.

But if one rejects the standalone requirement, and allows that software-defined storage is simply one supporting component of a software-defined data center, an application-aware external control plane can (and should!) instruct storage on what to do.

No mandatory requirement for metadata, and no mandatory requirement for object storage. Sure, application-specific metadata can be accommodated and exploited as needed by an external control plane -- but it's not the basis for a primary control plane for defining and monitoring application-specific storage services.

Put differently, I see the the SNIA definition as an example of a traditional silo perspective — that storage must be its own, self-contained entity.

Fill out a form (e.g. metadata), and we'll get back to you.

And I’d argue that’s exactly the thinking we’re trying to get away from.

Problems With Pooling

Not to pile on, but digging in a bit further I found something else to vehemently disagree with: the use of pre-defined and static storage service pools — the familiar gold, silver and bronze.

While certainly the accepted norm in much of the industry, this model is far from ideal for either applications or the storage team.

Using pre-defined pools, applications can only choose from a small number of established categories, and can’t get the precise mix of services they desire. Nor are these services aligned on application boundaries. And if an application’s storage service requirements change, it means moving everything to another standardized pool.

With a static pool model, the storage team is now responsible for accurately forecasting the aggregate requirements of all applications, present and future, accurately establishing buckets, and pre-provisioning the required pools far in advance.

It’s the familiar “have a hunch, provision a bunch”.

The SDS model we’ve presented elsewhere by comparison is dynamic: individual applications request precise mixes of storage services, and those are dynamically provisioned by the supporting infrastructure on an as-needed basis.

If it helps, think of it as the difference between a just-in-time demand-driven manufacturing vs. a Soviet-style centralized planning model.

Final Thoughts

I do want to give all credit to the SNIA team for coming together and investing the time to commit words to paper. And I hope they interpret this critique as a contribution to the discussion, and not an attack. Frankly, I'd like to see more with knowledgeable perspectives weigh in on this discussion.

That being said, their result is emblematic of many of the philosophical debates raging in IT architecture today: static vs. dynamic, silos vs. integrated, the nature of control planes, and so on.

Clearly, the views presented here are very much influenced by my role at VMware — a company not traditionally viewed as a major storage vendor, and perhaps not hugely relevant to storage-oriented organizations such as SNIA.

From a very simple set of ideas, a cornucopia of choices now exist: each its own distinct flavor.

That’s the way technology matures: innovation, differentiation and speciation.

I was originally a UNIX guy. For years, I had that wall-sized chart that showed how UNIX evolved over the years. I’m sure one could create a similar wall-sized infographic showing the evolution of “cloud”.

And, yes, people still fruitlessly argue that some forms are “better” than others.

We’re going to see something very similar when it comes to software-defined storage. Many flavors of the same idea will compete for adoption.

And being faced with so many choices can make for hard decisions ….

Learning To Make Good Choices

Many of you are parents, and — as a parent — I wanted to teach my kids to make good choices, especially when I wasn’t around. Many times, there was a conversation around “here are your choices, here are the likely consequences of each”.

Before long, we’ll see many dozens of software storage options, all conveniently self-labeled “software-defined storage”. There will be "peer pressure" in abundance: vendors, analysts, co-workers, etc. IT leaders will have to make some difficult choices — and, as in life, there will be ramifications.

#1 -- Familiar — Or Evolved?

I can already see that is going to be a hard one for many folks.

Here is your familiar storage array. And over here is a software stack purporting to do the same things the storage array does, only faster/cheaper/better/whatever.

And no big change in the operational model.

On one hand, evaluating your choices becomes far simpler when you use the prior storage array model as a measuring stick. How does the software-based version compare on price, performance, features, etc.?

So much of the current industry commentary runs along these lines.

On the other hand, let’s reconsider the core definition I’m using for software-defined storage: the ability to dynamically compose storage services aligned on application boundaries. That’s something most storage products -- hardware or software -- don’t do today.

This particular definition mandates a serious change to the operational model: using application-centric policies, just-in-time provisioning, etc. And I would argue that the chief goal of software-defined anything is to evolve the model, and not simply recreate the familiar past using new technology.

A document editor is not just a better typewriter.

Put differently, just because storage is being done in software doesn’t make it software-defined storage.

#2 -- Bottoms Up, Or Top Down?

I have met literally hundreds of IT shops with basically the same challenge: rampant storage diversity. While diversity is a great thing in the social sphere (work, friends, opinions, food, etc.), it’s an enormous drag on IT.

If you go looking for root causes, you wouldn’t be surprised. IT strategy being driven by an out-of-control procurement function that buys whatever is on sale that month. Every group making their own technology choices independently. Aggressive M&A resulting in a cluttered toy box to be sorted out.

These folks are quite interested in a storage abstraction layer that normalizes all the differences, and provides a standardized consumption and operational model. I believe that EMC’s ViPR controller is the best example of a technology that meets this need.

For them, software-defined storage means working with what they have, and not starting out afresh.

That being said, there are plenty of cloud builders out there.

These cloud builders are rightfully focused on the desired operational model, and they make technology choices based on how well they support the end state.

They don’t feel obligated to recycle someone else’s choices — although occasionally they don’t have an option :)

Our working definition of software-defined storage appeals greatly to this crowd.

And — yes — today there are many products being marketed as “software-defined storage” that only have rudimentary northbound APIs at best. A simple test: anything a user can do, a program should be able to do.

Actually, a program should be able to do more than a user can :)

#3 -- Managed Separately, Or Converged?

Today, dedicated storage teams are the norm in good-sized IT settings. If we’re going in the direction of software-defined storage, isn’t SDS their gig to decide and manage? Well, yes and no.

This is a variation of the idea presented above, but speaks to how the technology function is organized.

While I love most storage professionals dearly, I find they continually struggle with converged technology and operational models. Mostly, they don’t see where they fit in — that is, based on what they’re doing today.

And that’s the point.

Poignant story: a few years back, I was asked to sit in on a review of a customer's cloud strategy. The server team reported out what they planned to do, the network team, the storage team, etc. Each team had a plan to implement "cloud" after their own image.

After the third report-out, it was clear that the teams hadn't done much collaboration :)

#4 -- Static -- Or Dynamic?

So much of current storage thinking is based around the notion of pre-allocated buckets of resources. Maybe the consumption is dynamic, but someone had to stand up (and pay for) all the choices in advance.

This weekend, my wife and I are hosting a party. This morning, we made a list of everything we'd need: food, beverages, etc. Basically, a bunch of educated (and expensive) guesses, with generous overprovisioning.

If this party is like past ones, we'll wake up in the morning to find that we came up short on a bunch of choices, and have tons left over in other categories.

There's just no predicting what people will gravitate to.

But that's what we largely do today: have a hunch, provision a bunch. The need for granular and dynamic service provisioning should be clear; getting comfortable with the notion will take some time.

We spent the last few years moving IT from hand-carving every request, to having pre-allocated service choices. We'll spend the next few years moving from pre-allocated services to dynamic composition of services from resources as demanded.

#5 — Start To Invest Now — Or Wait Until The Dust Settles?

Another hard choice for many.

Our somewhat isolated discussion around software-defined storage has to compete with other big things going on in the IT realm: mobility, big data, next-gen apps, security, etc. A quick survey gives the impression of a noisy, adolescent marketplace that has yet to fully mature.

Why not wait a year or more until things are clearer — and easier?

There’s a strong counter-argument, especially if you agree with the definition we’re using here.

The biggest win with SDS is the operational model — and it’s brand new. New operational models can take a lot of time to fully understand, implement and become comfortable with. It's always the long pole in the tent.

This observation has nothing to do with technology per se, it has to do with people and process.

Personally, I’m telling people that now is a good time to modestly invest in becoming familiar with the new operational model — and putting it to work in some small part of their environment: test and dev, maybe VDI, an OLAP server, a new instance of their private cloud, etc.

After a bit, it becomes obvious to all that the new model is light-years ahead of the existing one, which always creates more momentum to move ahead.

The Key Question?

When it gets down to software-defined anything, the real question is — what the heck are we trying to do here?

Is our goal simply to do what we’re doing today, only do it incrementally better?

Everyone who drops by the room (even to just say"hi") can grab a cool VSAN t-shirt for their collection. Until we run out, that is.

We have five 30-minute training modules. Each separate module you complete enters you in a drawing to win one of the Chromebooks we'll be using in the lab. We'll be announcing the winners at 5:30 on Wednesday. You don't have to be present to win.

Follow #VSANlab on Twitter for status during the show -- and prize winners!

I've worked with Chad and John in the past. I work with Christos and Richard now that I'm at VMware. By my standards, each one of them qualifies as a bonafide visionary on the subject. Trust me, they all have a lot to say, and don't necessarily always agree on things :)

There’s a big difference between saying software is hardware-agnostic, and that saying that hardware doesn’t matter. As anyone who’s been in a data center knows, hardware really does matter: cost, performance, functionality, environmentals and more.

Where the debate usually rages is around the use of standard designs and form factors vs. the desirability of purpose-built devices.

But for the purposes of our software-defined storage model, it’s not a relevant distinction.

As long as our proposed software-defined storage layer can dynamically compose storage services that are aligned on application boundaries, we can equally consider standardized server hardware as well as purpose-built storage arrays.

That being said, when it comes to storage, server form-factors will inevitably end up playing a very significant role … thanks to newer memory technologies.

From Disk To Flash To RAM

As part of EMC, I started to become familiar with the first round of enterprise flash drives way back in 2007. Even then, it was pretty obvious that flash deserved to be in a server.

By the time you take flash technology and package in a disk-compatible enclosure — and run through both a storage network and storage array to access it — you’ve taken a great technology and made it somewhat more costly — and slower.

Yes, putting flash drives in an array makes it easier to share across multiple consumers, easier to consume and managee, etc. -- but that can be seen as a function of software, not hardware.

Certainly, putting flash storage on a PCIe card makes sense. So does skipping the PCIe card completely, and putting flash right on the motherboard using something like ULLtraDIMM technology.

Roughly speaking, moving from a drive packaging format to a PCIe format gives us roughly 3x the performance at a significantly lower cost.

It’s too soon to tell, but it looks like moving from PCIe format to ULLtraDIMM will once again give us roughly 3x the performance, again at significantly lower cost.

Let’s not forget that there’s something even faster than flash, and that’s server RAM.

While not persistent in the same way that flash is, RAM can be quite attractive for read caching and non-persistent writes. A quick check of newegg.com tells me that server RAM is about $10 per gigabyte; enterprise-grade PCIe flash cards run about $3-5 per gigabyte.

The costs of both technologies are falling predictably. Yes, we’re constrained about how much of each we can currently stuff into a server, but that’s always improving.

If we look at our classic storage hierarchy, the changes are clear: the first two storage performance tiers are now clearly server technologies: RAM becomes the new cache, and server flash becomes the new “high performance” persistent storage medium.

As a result, it’s likely that servers physically morph into very fat motherboards, chock-full with terabytes of RAM, as well as 10s of terabytes of flash storage. Whether those motherboards end up packaged as rack mount or some blade format — we’ll see.

And - of course - we’ll need storage software to make all of this truly useful.

Moving Outside The Server

If the majority of performance-oriented storage tech moves to inside the server enclosure, where will that leave external storage arrays? Tackling cost -- and scale.

The disks themselves continue to get bigger (4TB->6TB and beyond), as well as predictable per-GB cost declines. Prices for disk media are now approaching the 5 cents per GB range, or around 1/100th the cost of enterprise-grade flash.

More concerning is that disk drive access densities continue to drop — disks are getting a LOT slower per usable gigabyte.

Note: access density is calculated by taking number of IOPS a disk can deliver, and dividing it by its usable capacity. The result is access density: the number of IOPS you can expect per gigabyte.

All things being equal, a 2TB drive has half the access density of a 1TB drive, and a 4TB drive has half the access density of a 2TB drive. As drives get ever-larger, they get ever-slower. The corollary is that IOPS are now always cheaper using flash.

While mechanical disks can certainly be put in rack-mount server enclosures, it’s not always optimal at scale.

Disk drives suck power, generate heat, don’t like vibration, need to be swapped out, etc. In general, purpose-built enclosures have done better in housing vast quantities of disk drives (many 1000s), and I would expect that to continue.

If I was really speculating, I’d expect see renewed interest in low-powered processors for array controllers. As most of the storage processing overhead will inevitably move to the server, there’s going to be much less of a case for using higher-powered CPUs in storage array controllers.

There is also an argument that external arrays are better positioned to provide certain data services, as an array is closer to the data, understands its layout, etc. While this may (or may not) be true -- it only can be potentially true for data that resides entirely within that array: not residing in server RAM cache, not residing in server flash, etc.

Connecting Server And Storage

So, in our near-term hardware model, servers have capacious RAM (used for cache) and even more capacious flash (used for persistence). External storage arrays get bigger and denser, and are predominantly comprised of high-capacity spinning disks.

What about the interconnect between the two?

While ethernet usually gets the nod for cost-effective connectivity, how we use ethernet in a storage setting is likely to change. Previously, an external storage array had to present data on a network in an externally consumable manner: blocks, files, objects, etc.

However, in our new model, this presentation can now occur entirely within the server. There’s much less need to expose a legacy protocol over the network.

Going forward, the interconnect between server-resident storage and array-resident storage is now a storage-to-storage conversation: bulk data moving up and down the wire. Remember: anything with low latency requirements has presumably moved to server-resident flash.

This means we can start considering different protocols that emphasize bandwidth and transfer efficiency, e.g. RDMA and others. Less point-to-point, more mesh. And much less need for SCSI, NFS et. al. to be exposed by an external storage array.

My only note of skepticism: fibre channel works well, and there’s an awful lot of it installed. One of the last things anyone wants to do is rip and replace a perfectly working fabric :)

The other obvious "wire" we need to discuss is to the proverbial cloud.

Many of us already use cloud storage to extend the capacities of our laptop flash drives, as well as improve access, protect against loss, etc.

As we continue our series on software-defined storage (or SDS), I think it’s time to turn a critical eye on the current state of affairs, and be completely transparent on many of the obstacles and challenges that inevitably lie ahead.

The move to a entirely new technology stack -- and the associated operational model -- can be a long journey. It's nice to know what to expect along the way ...

A bit of google-fu will show around 20 technology vendors who have used (or abused?) the “software-defined storage” term, in addition to a half-dozen industry analysts and countless hordes of bloggers, including myself.

I’ve done the best I can with this series to methodically walk through concepts, terms and definitions. I would only hope that others would do the same, based on their world views. When cloud was new, we saw an awful lot of “cloudwashing”, e.g. dressing up your same old stuff with a new buzzword.

If you’re watching the storage landscape, it looks like we're entering a "SDS-washing" period. That's unfortunate, but inevitable.

Just as we saw with "cloud", a noisy landscape doesn't exactly encourage people to move forward with confidence.

#2 -- The Technology Either Isn’t Here, Or Isn’t Fully Baked Yet

If you’ve run through my lengthy series of blog posts, you’ve probably come away thinking that “gee, not a lot of proven stuff in the market that works this way”.

And you’d be quite correct.

But the point of this whole SDS discussion is to illustrate where things are going, and not recap where we are today. The IT infrastructure industry is heading down the highway at high speed -- and storage is along for the ride -- so it's just a matter of time for the pieces to fall into place.

Put differently, your perspective is likely be quite different in 12-18 months time.

Even then, you’ll still see gaps (aren't there always?) but you will also see a much more complete picture in terms of technology you can actually put your hands on.

And some interesting pieces of the puzzle are available today, if you choose.

#3 -- There’s A New Operational Model Implied

For those IT shops who are moving to (or have moved) to an ITaaS (IT as a service) model, the leap to incorporating SDDC and SDS concepts isn’t all that enormous — the foundations for the required operational framework are already in place.

However, for those IT shops that are still using a familiar and traditional operational model, none of this will likely matter. Software-defined storage is operated as a service, not a silo.

With that, I fear we’ve lost a lot of people.

#4 -- And There Are Still Some Potentially Unanswered Questions

Like any new set of concepts and technologies, there will be areas that deserve deeper probing and questioning. One such area: the decoupling of data services from the data plane.

Let’s take familiar snaps, clones, etc. As implemented in a traditional storage array, the data service has intimate knowledge of metadata (track tables, bitmaps, etc.) and physical data placement, as well as the ability to directly manipulate cache.

The result are snaps that (usually) perform well.

An open question is whether this desirable property can be replicated when snaps are decoupled from the data plane. There are a few examples where this has been done reasonably well, but until we see more implementations, it's an interesting question.

A similar concern comes when considering stacking data services: snaps, replication, dedupe, encryption and similar. Certain combinations are obvious; others take a bit more thought around how the interactions should work.

#5 -- Any Abstraction Solves Problems — And Can Create New Ones

Take yourself back to the early days of server virtualization. Yes, we had a powerful new abstraction.

But now resolving something like a performance issue now had to be approached differently, as there was a new abstraction in the mix. It took a while for the tools — and the expertise — to catch up with what the technology could now do.

A valid issue, to be sure, but not insurmountable.

Software-defined storage is a new abstraction — actually, several abstractions — so it would be safe to expect much of the same experiences: same problems, but with a different approach required.

#6 -- There Will Be Resistance From Many Quarters

Let’s face it — today’s storage array technology basically does the job, and — properly implemented — does the job reasonably well. So did physical servers, back in the day ...

Why fix something that (ostensibly) isn’t broken?

I expect to be hearing this from many quarters: established storage vendors who haven’t invested in SDS technology, storage professionals who are justifiably skeptical, various pundits, etc.

By now, you know the answer to the rhetorical question: because much better is becoming possible. The world is a savagely competitive place, thankfully. Those that invest in the required technology and operational model — as the technology matures — will be far better positioned than those that wait until the dust settles, when the answers are obvious to all.

There’s another barrier to be considered: organizational friction. Good IT service delivery rests on a foundation of operational processes that are both mature — and inherently resistant to change.

Changing the technology is relatively easy; changing how people use it is far more challenging.

#7 -- A Long View Will Be Required

We’d all like to think we have the required long view.

And we do — for about 30 seconds — and then it’s off to the next meeting, handling the next crisis, responding to the next urgent request, etc.

To continually focus on achieving a higher goal — maybe over a few years — that’s a very difficult task for most.

But the path forward doesn’t require a major disruption -- there are small and reasonable steps that can be taken.

In a future post, I’ll offer my suggestions as to how motivated IT architects can move forward incrementally.