an insider's perspective, technical tips n' tricks in the era of the IT Revolution

Homebrew VMware

September 01, 2015

There’s lots that can be debated, many topics that can be debated. One thing that cannot be debated is that people are trying to make parts of IT that they view as important but low “differentiation” (for their business) SIMPLER.

This is the really basic observation that is behind the success of converged, hyper-converged infrastructure - packaged IaaS offerings, and all forms of cloud business models (Private, Managed and Public).

Since EMC is a public company*, how our business is working is wide open to the public every quarter. We’ve publicly noted that VCE was on a $2B run rate, and in Q2 of this year, we publicly disclosed a 30% revenue growth rate. Now - until recently - VCE was in effect a single “product” company with only “block” style converged infrastructure.

*People have recently noted that sometimes when a company is forced to publicly disclose revenues/burn rates… well just how much can be hidden behind stating “percentage growth” rather than absolute dollar terms. Something to consider with any startup (whether you’re looking to buy, or join) - don’t ask the person you’re talking to for how they are doing on growth, ask them for absolute bookings, cash position and burn rate.

Posting big numbers, and fighting the law of big numbers (every time the number grows, maintaining a high growth percentage just gets harder) is very impressive. It highlights something that cannot be denied: when for whatever reason customers cannot use or are not best served by a public IaaS cloud or a managed virtually private one - customers want simplified and converged infrastructure, and then on that, they want simplified “Cloud Personas”. This highlights the main “why” on all forms of converged infrastructure: “go faster, and get out of a low-value part of IT”. It also highlights why EMC views “winning all forms of CI” a strategic imperative.

But - there’s an inherent upside/downside of the “block” style of converged infrastructure:

Scale:

Block architectures are really only available in “medium/large”. Upside: CI Blocks are remarkably cost-effective from a CAPEX standpoint at those larger scale points. Downside: they are composited of traditional architectures assembled, engineered and packaged in a converged fashion, the smallest one is not inexpensive, and they scale to a certain point where you then need another big block. Think of a CAPEX picture with big, $750K steps in the step function.

Operation:

While they are delivered in a single large block - they are a composite of traditional components. Upside: CI:Blocks have the full set of data services and operational behaviors on which some traditional workloads depend. Downside: while the CI vendor takes on the responsibility of testing, engineering, updating - it’s not easy - and while there is some OPEX benefit of going CI, it’s not enormous.

Both of these points highlight why the hyper-converged category exists - is well populated by strong startups, and why EMC (and VMware) views hyper-converged as a critical market to win - over time, as important (perhaps more important) than “Block” traditional style architectures - which will continue to be popular at scale, and when building a “unified” stack that must support workloads with traditional data service requirements.

Q:So, what defines a hyper-converged infrastructure offer?

A: Here’s my list:

Uses lower-value industry-standard servers with local storage as the modular component (don’t try to build a ton of value into the server itself)

Uses an SDS layer to deliver the persistence capability - and grow in a linear way as additional nodes are added. SDS layers are wildly different.

Uses a management and orchestration (M&O) layer to automate a lot. How much is automated varies - and HOW this is done varies, but it’s a lot of system-level automation.

Q: Are all hyper-converged infrastructure offers the same?

A: No, they sure aren’t. While the all have the 3 above - their design centers are WILDLY different.

Let me ask you something. Do you think it’s possible to design something that NAILS the requirements of: A) an ROBO/SMB/SME customer looking to host a mix of VDI/general purpose virtual servers - maybe even a few thousand (a pretty big customer!); AND B) simultaneously with an Enterprise Datacenter (with a crazy variety traditional apps and some brownfield Cloud Native Apps) AND C) hyper-scale green field Cloud Native App focus?

No - that’s totally delusional.

The fact is that each of those CAN be met with a hyper-converged infrastructure approach - and they all will kind of “look” the same in the sense that they have: 1 (industry standard servers), 2 (SDS), 3 (system M&O) in the list above - but the design centre of the systems as a whole are WILDLY different.

People have started to realize this when they take hyper-converged systems that optimize around the SMB/SME customer bands and then push them into the other use cases. They LOVE the simplicity and low entry price they deliver, but then discover the economic and system level scaling gets less awesome as they scale up. And while they “look” like how everyone imagines a “hyper-scale Cloud Native App” infrastructure should look (and often they use some of these technologies internally) - the design center of the system is totally different.

This means something simple: there is no unicorn: there is no single hyper-converged stack (hardware and software) that can cover these wildly differing use cases.

Now - people will surely disagree with me (and I can imagine who!), but that’s why I think a portfolio is needed. The portfolio should have exactly the amount of variation needed - not an ounce too much, but not an ounce less.

Here’s the VCE portfolio - and these are all now available (the VxRack is in directed availability now, as VCE ramps up to make sure they can deliver the white glove experience people have come to expect from them).

Blocks: Converged Infrastructure built from traditional datacenter architectures - which makes them ideal for a broad set of workloads including the most mission critical monolithic apps with deep infrastructure and data service requirements. Comes in medium and large - not available in small. Has a big “step function”. Simplifies operation and lowers OPEX since it’s an engineered system - but only so much as the traditional architecture can support. Offer: VxBlock (and Vblock for people that don’t want NSX)

Racks:Hyper-Converged Infrastructure built using industry standard servers + SDS + Management and Orchestration layer. Optimized for Rack and multi-rack scales. Comes in small to VERY, VERY large - in small linear steps. Disaggregated system design for broad hardware flexibility. Simplifies operation and lowers OPEX since it’s an engineered system - beyond blocks because it does not have a traditional datacenter architecture. But - this means that they can support broad workloads - but in general not those that are the most mission critical monolithic apps with deep infrastructure and data service requirements (can only do what the SDS layer can support). Can support multiple software “personas” for wildly different use cases. Offer: VxRack

In this category, we know we need to be the simplest, most awesome turn-key product. It needs to start in low price bands - be feature rich, and nail the ROBO/SMB/SME use cases.

I’m going to make this simple - because most people know VSPEX Blue - and I will run out of space to update on VxRack.

The joint team has a singular mission: to completely win in the Hyper-Converged Appliance Infrastructure category.

There have been lots of updates: support for different licensing and support models, updates to the software stacks - and there are a lot more coming.

We know what works - and what we need to do to fill in gaps.

We are in it to win it.

We are going to accelerate our iteration in every vector.

I would not want to be one of our competitors in this space (who we respect fiercely) - because we are going to hit the gas.

Partners - learn about VSPEX Blue - and hang on to your hat.

Racks.

At EMC World in May, we talked about our early work on VxRack. It is now in Directed Availability - and will be generally available shortly.

People speculate wildly about the hardware in it (and I see competitors get it wrong). There are 3 node “types”, but each of those have a broad range of configs - it’s a disaggregated system.

What do I mean by “Disaggregated?” - well unlike appliances that generally are optimized to scale somewhat linearly - with a disaggregated rack-scale system - you can scale compute and storage in different dimensions. This is what I mean:

Looking at the above - you can imagine that this design could easily include things like DSSD in the future, or the crazy new interconnects that Intel keeps introducing.

Unlike appliances, it has a spine-leaf network architecture that is there always - because like all true rack-scale systems - you need to contemplate the network if you really want to scale. All of this means there needs to be a hardware management and orchestration layer that is MUCH more sophisticated than in the appliance category. This layer of tech needs to do fault, remediation, firmware, low-level hardware management and telemetry.

This is a place where there’s a lot going on - with more to come in the future. We’ve talked a little about the work that EMC has been doing (OnRack), VMware has been working on something called Hardware Management System (HMS) - and Intel has some really cool things also. Hint - stay tuned!

And lastly - if you want to support multiple “personas”, you need to have a system-level management function and a vehicle to keep the whole system current and linked to the system support tools - which is VxRack Manager and VCE Vision, which we showed at EMC World:

So - you now have an idea of what VxRack is - with the “bring your own persona” baseline.

Now, what will be the first pre-packaged “personality” that we will support on VxRack - the VMware SDDC of course! This where our partnership with the EVO SDDC Suite and EVO SDDC Manager comes into the picture.

Ultimately - no one just want Hyper-Converged Infrastructure - they want to DO something with it. Some people will want to build their own cloud IaaS stack - or deploy other applications. Many people will want to deploy the VMware SDDC in a way that’s simple and easy.

Thanks to the VMware EVO SDDC Suite team (thanks!), here’s a demo of the EVO SDDC Manager in action!

You can go to the EVO SDDC Suite booth at VMworld and see the EVO SDDC Suite and VxRack for yourself!

There’s lots of work to do still… VxRack needs to exit directed availability. We need to finalize the integration between the OnRack and VMware HMS elements - but the teams are working furiously down this path, and if you want the latest, reach out to your VCE team (or hit me on twitter :-)

Will we stop with the EVO SDDC Suite persona? No - expect to see “Greenfield Cloud Native App” personas in the future - lots to do there around the Photon Platform and Pivotal Cloud Foundry - and heck, even things for customers who reject the “full Federation" Cloud Native App stack.

I’m going to close out this post by re-emphasizing an important and simple observation.

HCI Appliances are: 1) Made from building blocks + 2) are designed to start small + 3) have a lightweight hardware M&O layer

HCI Rack-Scale systems are: 1) Made using a disaggregated system components + 2) are designed to scale big + 3) have a material hardware M&O layer

Trust me. Over time, Converged and Hyper-Converged Infrastructure will consume the stand-alone infrastructure component markets (which will still exist - but simply feed into CI and HCI offers from giants).

That means you will see a lot of craziness in this area over the coming years - challenge me, challenge us, challenge others - and think clearly - architecture-level thinking is valuable.

Keep reading for the new (at least for now!) record holder, from Adam Jones a Sr. TC, VCP (and more acronyms behind his sig than I've ever seen :-) in Kansas City with a dual core, x86-64 (ergo future ready), 8GB machine....

Folks, here it is - the "all-in" step-by-step guide for setting up Site Recovery Manager with nothing more than two ESX servers (which you can even do with home-brew servers, as noted here) and the Celerra VSA.

If this is your first read, you can either just click on the document below or start the 101, 201, 301, 401 journey here.

This guide isn't the first one of these out there. I think my respected friend Adam Carter at Lefthand did it first (Adam, hope you're enjoying HP!). The NetApp one is pretty good too. What I like about ours is the extreme completeness. It's also nice that the Celerra VSA has no timeouts, no limits (except that you'e not allowed to use it in production).

A quick shout out... This was absolutely yeoman's work by one of the newest members of the Global EMC VMware Specialist squad - Bernie Baker, with strong assists from one of the other newbies - Stephen Spellicy. Both of these guys started relatively recently, and this work is (IMHO) FANTASTIC - it highlights that this stuff can be EASY (and that they have what it takes to have a early impact). Bernie - you've set the bar high for the team and yourself - GREAT WORK!

Until the next major revision of the Celerra VSA (which will be linked to major feature releases), expect to see only one more "301"-level Celerra VSA post (downloading and using EMC Replication Manager for VMware for point-in-time, VM-consistent instant VM backup/restore) - we're warming up in the batter's box a series on Avamar Virtual Edition....

November 27, 2008

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post. I wanted to provide a quick "HOWTO" to help customers Replicate between two VSAs, for many reasons - one of the most fun being VMware Site Recovery Manager.

Celerra Replicator (I hope the Celerra Product team doesn't read this) is a RIDICULOUSLY inexpensive (particularly when compared with the competition) but very sophisticated remote replication capability. It can support cool stuff like:

1:N and also N:1 replication fan-in/fan-out replication relationships (like with any technology there are operating envelopes and parameters)

It can support cascading topologies (i.e. site one replicates to site 2 with a given frequency, and then site 2 replicates the data from site 1 to site 3 at a different frequency

It has pretty sophisticated QoS mechanisms (i.e. you can setup different bandwidth use for different parts of the day

You can replicate all sorts of configs - CIFS/NFS/iSCSI, and fully supports thin provisioning at the source or the target

And of course - most interestingly of all - it's fully integrated with VMware Site Recovery Manager (so you can use this to build your own SRM playground.

Of course - SRM doesn't support NFS yet, so I focused this HOWTO on iSCSI - but it's very easy to see how you would do CIFS/NFS (just hit the other radio button in step 3)

Before you do this - make sure you have a working VSA - just follow the Celerra VSA HOWTO 101 here...

I've updated the OVF - tightened it up a bit. You can get the new one here

Ok - have you got it? Have you followed the 101? Want to add more storage - just follow the Celerra VSA HOWTO 201 here.

Ok... then read on...

Quickly then here - in this HOWTO:

Configuring NTP

Correcting the Replication Database (post Clone or deploy from OVF the Celerra VM's serial number is changed) - BIG thanks to Himabindu Tummala and Santosh PasulaReddy in Celerra Engineeringfor helping me figure this step out

Configuring an iSCSI LUN that will be used as a replication target

Configuring Celerra Replicator

One of our elite-delta squad VMware Specialists is putting the finishing touches on an "all in" doc and pair of Celerra VSAs to make setting up SRM with two Celerra VSAs a breeze, but you have every ingredient you need now with these HOWTOs (and SRM itself is a walk in the park).

One quick note - remember that if you are looking for support - post on http://forums.emc.com, in the Celerra, Celerra Simulator forum.

Next up to plate is the HOWTO on Avamar Virtual Edition, and I'm also going to do a HOWTO post on how to use the super-cool Replication Manager with the VSA. Rep Manager - aka "RM" can actually be used with all EMC's primary storage platforms, for all sorts of replication/test and dev use cases - for Exchange, SQL Server, Oracle, and also VMware VMs and VMFS datastores - NFS datastores being supported shortly.

November 11, 2008

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post. I wanted to provide a quick "HOWTO" to help customers add more physical storage, so they have more to play with!

Before you do this - make sure you have a working VSA - just follow the Celerra VSA HOWTO 101 here...

I've updated the OVF - tightened it up a bit. You can get the new one here

Ok - have you got it? Have you followed the 101? Ok... then read on...

Quickly then here - in this HOWTO:

Adding physical storage to the VM

Configuring the additional storage to the Celerra VSA (and how to remove storage)

Next up to plate are the 301 (setting up Celerra Replicator and SRM), and the Avamar Virtual Edition.

October 21, 2008

I've been tearing my hair out with a future build of ESX for a couple of weeks. Now my system is decidedly NOT on the HCL. It's one of my home-grown ESX 3.5 u2 systems - an Intel G33 motherboard, Q6600 CPU, 8GB of RAM, bunch of Intel 1000 Pro GT NICs. To boot, I've been using an old-school QLA4010c (thanks qlogic for sending me the new QLE4062s and the cool 8042 10GbE converged adapters!).

During install, it would hang while loading the network drivers - I eventually isolated this to the Intel GT cards. PTs, OK. MTs, OK. Interesting. ?If I don't disable the onboard NIC, same deal - hang during install. They all use the e1000 driver, but mental note - I'm using MTs and PTs going fwd.

the internal SATA controller looked like it installed fine, but then looked like the MBR wasn't configured right - at the end of install, the HDD wasn't bootable. I found this long ago with non-supported controllers, so poked around with this.

So, the QLA4010c (and this seems to be true with the QLE4062 too) boot from iSCSI SAN wasn't working right (BIOS loads, drives found, but don't show in the install screen).

I poked, prodded. I tore my hair out. I was stuck with no bootable controller/drive combo, and was considering a trip to Fry's for another MB whose SATA controller I would know would work (an older nvidia MB for example).

Then, I had an idea.

I've used a bootable USB ESXi 3.5 image on my laptop, and suddenly thought - "why don't I just do that?"

A few minutes later with DD for windows and the dd image I had my ESX image on USB.

August 05, 2008

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post. I wanted to provide a quick "HOWTO" to help, and will publish a 201 (setting up Replication and a 301 - configuring SRM).

Most folks are doing well if they just RTFM, but I thought a little "hand-crafted walkthrough" might help others. The standard documentation is geared towards VMware Workstation, which is the officially supported target (BTW, you don't need to ask, we're making ESX OVF a standard package :-), so some people stumbled a bit when trying to use it with ESX.

I've updated the OVF (some of the problems people were having were a result of the actual previous OVF I posted - SORRY - sheepishly) . Make sure you download the new one if you're going to follow along (again, you can get that here)

In a little bit, I'm going to start posting bit by bit other EMC VMs (they are legion - ADM, Control Center, Avamar, Replication Manager, Networker 7.4.x, IT Compliance Analyzer, EMC Backup Advisor, and that's just getting started) with similar guides, so let me know if this is useful - BTW - they are also going through the formal release and VM Appliance programs, but I'm notoriously impatient, and think many of you out there are smart enough to get some fun out of these without too much hand-holding :-)

First - this is the view of the entry to the server closet, which is literally a closet with a "raised floor" - not for cabling and airflow, but rather so that when the basement inevitably floods, the gear is spared. It gets hot, so I put in an industrial thermocouple that acts as a cut-off of the dedicated 20A circuit I pulled in - for a shut-off just in case my sophisticated HVAC system fails.

This is the view of my "production servers" - an Intel Q6600 based cluster (8GB RAM each) and a AMD x2 3800+ based cluster (4GB each) - cookbook instructions below. I use the two bigger Dell PowerConnect 5324s for iSCSI and LAN/NAS traffic - the little Netgear switch is for VMotion. I could have used VLANs, but I like the physical topology, and had the switches kicking around (or just say "screw IP best practices", but what's the fun in that). The little host in beside the Intel cluster is my VC host. You can save some bucks and run VC as a VM of course (handy in some DR use cases!). I'm glad that VMware made that a supported use case around the VC 2.0.1 timeframe (can't remember the exact release)

This is the view of my "DR cluster" - an Intel E4300 based cluster (8GB RAM each). These are cheap as dirt, but kinda sucks that they don't have VT. I also just use a cross-cabled Cat5e cable for Vmotion. I use this for when I'm just playing around with Site Recovery Manager (how do I do that without arrays? READ ON!!!). The poster has one of my favorite quotes:

"Whatever can be done, will be done. If not by incumbents, it will be done by emerging players. If not in a regulated industry, it will be done in a new industry born without regulation. Tehcnological change and it's effects are inevitable" - Andy Grove

Slightly less inspirational (but only slightly) is the rolled up construction plans for my elaborate MAME-based arcade that I'll get started on any day now.....

This is a quick shot of my sophisticated HVAC system. This rig generates a LOT of heat, particularly when I spin up the arrays. I bought a cheap 300CFM bathroom fan, and then tore apart all the drywall and vent it out. the intake is baffled, so when I close the door, the whole thing is pretty quiet.

If this is making you eager to do the same thing - it's gotten ridiculously cheap, and ridiculously easy. I'm going outline how to build two ESX servers - one where you're looking for the CHEAPEST thing you can build, and one which is a little more pricey, but a great "bang/buck" ESX lab.

ESX 3.5 makes doing this a LOT cheaper - now that SATA drives/controllers are supported, Nvidia NICs are supported in ESX 3.5 bits, but there is a trick - technically the Nforce professional chipsets are the ones supported, but they use the same MAC as the cheapo consumer stuff - you just need to make sure that the motherboard doesn't use a Realtek NIC (or buy one of the Intel NICs)

General things to make sure you do

Get a CPU that supports 64-bit guests - this is generally an Intel CPU that starts with the letter "Q" not the letter "E" (or just check the specs and look for VT support). Any Athlon 64 or opteron works. Can you go cheap - yeah, but that often costs you 64-bit guest support on Intel. If you're going cheap, I personally go AMD.

Get a motherboard that supports a minimum of 4 GB of RAM - 8GB is nice (all ESX servers are generally constrained by RAM)

Get a decent (but still super-cheap) GigE switch - something that supports VLANs so you can create configs that work with less physical NICs - it's crazy, but you can get an 8-port switch with full support for 802.1q, p, and everything else you could possibly need.

Make sure you have a motherboard that has onboard VGA - you don't need a good graphics card, but you need something for initial config.

UPDATE (Jan 5th, 2009) - one of my colleagues sent me a new "record cheap" dual core 8GB config, and I've done a post on that HERE - you might want to start with that, as technology moves pretty fast - heck, some of the older stuff below you can't even BUY anymore :-)

AMD ESX configuration (as cheap as it gets, but you have everything you need) = $337This config leverages the fact that ESX 3.5 supports Nvidia NICs - and there will only be one NIC for VMotion, network, and IP storage. Name of the game = how cheap can you go

Generic ATX Motherboard - based on the 430, 6100 or 6150 chipsets - just MAKE SURE it has the Nvidia NIC, not a Realtek NIC = $54 (http://www.newegg.com/Product/Product.aspx?Item=N82E16813157108) NOTE - THIS ONE HAS A REALTEK NIC, so you need to by an additional intel NIC. I use an old ASUS A8N-VM CSM socket 939 motherboard, and it has a Nvidia MAC and works great - but you need to find an older Athlon that fits that socket 939 form factor.....

If you're not sure what MAC the NIC uses on the motherboard, or just want to be safe - add 1 Intel GbE NICs (these are a trick - you need specific ones for the Intel e1000 driver that comes with ESX 3.5 to work - hard to find, and DON'T buy the server MT versions - find the cheapo desktop GT PCI or PT PCIe versions - hundreds cheaper and work fine) = $42 http://www.allstarshop.com/shop/product.asp?pid=16016&ad=pwatch

Intel ESX configuration (a super cheap quad core, 8GB, lotsa GbE powerhouse) = $695This config leverages the fact there are ridiculously cheap multi-core CPUs and RAM these days. the NICs on Intel motherboards are usually based on older Intel or Realtek chipsets, (no driver support in VMware) - so you need to find some fancier (but still cheap) NICs. Name of the game here = how cheap can you build a powerhouse that you can run 10 VMs at once without breaking a sweat?

You will need to buy two of whatever model you get - for VMotion, VM HA, DRS, Storage VMotion, etc... (so AMD total cost = $674, Intel cost = $1390)

You will need ESX Server and Virtual Center - within EMC, we have a VMware/EMC ELA (remember - VMware operates independent of EMC as the parent!). You can, of course, download the software from http://www.vmware.com/ - they have 60 day eval timeouts.

What the heck to use for shared storage? Well, I have a CX300i, an EqualLogic PS100E, a Storevault S500, and have an Openfiler box - but I'm a freak, and have a very supportive wife. There is another simple option (that I actually use more than any of the others): use a Virtual Storage Appliance - these turn the DAS storage in the ESX server into iSCSI LUNs or NAS. EMC offers a free, unlimited, no time-out Virtual Celerra, which just runs on ESX, and is otherwise a fully functional Celerra - anyone who wants one, head over to this post...

Disclaimer

The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Dell Technologies and does not necessarily reflect the views and opinions of Dell Technologies or any part of Dell Technologies. This is my blog, it is not an Dell Technologies blog.