If you have done a failover cluster on Windows before, the process is a bit different so don’t just dive in as I did. So what does my environment look like:

SSO has been already deployed and working

A management vCenter is running; you will need this or some other means to clone the first virtual machine after installation

So wait why are you clustering vCenter if there is already a vCenter you ask? Many reasons, but primarily our availability of our management vCenter is less of a concern. The clustered vCenter is being deployed to support vRealize Automation so end users will rely on this vCenter to be able to request catalog items. Availability was more of a concern for this purpose than strictly management.

Start with only a single Window 2012 R2 64-bit virtual machine (not 2) as you will later clone this virtual machine to act as the 2nd node

I placed the original, and clone on two separate physical hosts

Each virtual machine has a single 60GB (C) drive for the OS

2 additional volumes will be added which, in my case, are XtremIO volumes presented as a physical RDM. This should also work using in-guest iSCSI for example

1 of the 2 additional volumes is a 60GB (D) drive which vCenter will be installed on and the other a quorum disk for the failover cluster

Each virtual machine has two NICs – one for production/client access the other for cluster communication

The Windows Failover Cluster will have an IP address, as well as the vCenter Service role which you will create; in total this is 6 IP address

An AD account was created for the vCenter services, added to the local administrators group and given permission on the SQL server as required

The directions have you install the vCenter Web Client, Inventory Service, and vCenter services to the D drive. There is a known bug that causes the web client to not function properly when installed to a non-default location (though it seems more that it doesn’t work when not installed to the C drive). You’ll need this KB article which walks you through creating a symbolic link, after implementing this the web client operated as expected. Also, once installation is complete and working on the primary node, you’ll need to failover to the secondary node to create the sym link (well at least I did, would it let you create a sym link to a drive that didn’t exist? hmmm)

So, with that out of the way there is a few things to define before you bring up your first virtual machine – specifically the names and IP addresses of both virtual machines, the Window cluster, and the vCenter cluster. For example:

This is important, and I misinterpreted this step the first time I did this: When you create the first virtual machine – give it the name and IP address of what will ultimately be the vCenter cluster – using the example above you will name the computer VC2, with an IP address of 192.168.1.100 and join it to your domain. After the initial install this will be changed.

Create the virtual machine, with 2 NICs and the RDMs. Mount one of the RDMs as D and one as whatever letter makes you happy, for my OCD that would be Q for quorum. Create your system DSN as you normally would, log in as your vCenter service account and perform a custom installation (not simple), installing each of the components to the D drive. During the installation process note that the name being added to SSO is the name that will ultimately be the vCenter cluster.

Before removing the RDMs, make sure to note their original file name, volume ID, and SCSI controller; they need to be added back in the same order.

These steps are pretty straight forward in the guide, change all of the vCenter services to manual, shutdown the virtual machine, remove the RDMs, and make a clone of the virtual machine. One item not clear was when to re-add the RDMs, I chose to play it safe and kept them out of the virtual machine for now. Once the clone is complete, power on the cloned virtual machine and rename it to the secondary vCenter node hostname and IP address. Power on the original virtual machine, unjoin it from the domain, rename and IP it with the hostname for the primary vCenter node, and rejoin the domain. Now you can power off the virtual machines, re-add the RDMs to the primary node, then the secondary as you typically would, making sure the SCSI controller is set to physical sharing.

Power on the virtual machines and install the Failover Cluster feature on each. Once complete, create a new cluster on the primary node – during the creation you will be asked for a cluster name and IP address – use the Windows Cluster name (VC2Win) from the example above – this is NOT the vCenter cluster name and IP address which you used on the initial virtual machine during installation. Unlike with the SQL post I wrote, you can add all available cluster storage as both additional drives are used for the cluster (D – App, Q – Quorum). Now that the cluster has been created, you should have an AD object called VC2Win. Using option #2 from this MSDN blog post, create your vCenter cluster AD object. Failing to do this will cause the cluster to fail when you attempt to start it.

The rest of the steps for creating the vCenter cluster role are well documented with one caveat, so rather than copy paste them here finish reading the VMware vCenter Server Availability Guide. That caveat, because your vCenter services were set to manual, and thus not started after the reboots, when you create the initial vCenter role service it will come us as failed – which made me go ZOMG not again! This message is actually just the status of the clustered service, which is stopped, thus failed from a Windows Failover cluster perspective – it is okay to proceed with creating the remaining services and setting the dependencies.

At this point, you should be able to start the cluster and have all services come up.

vCenter services on Windows Failover Cluster

Once it is up, access the web client and set permissions as required. For example, as you can see in this screenshot, here is both vCenters in the web client after since my account was given the appropriate permissions to both.

vCenter on a Windows Failover Cluster

The last item I have to tackle is automating the backup, copy, and restore of the ADAM database. There are a lot of words in the doc which basically says – xcopy the backup to the correct location. The document talks about stopping/starting services before placing the file. But if the services aren’t running on VC2-2, I should just be able to drop it in. Now when the services start there is an up to date file which will get loaded.

Working on building out a lab that is going to be used to demonstrate setting up a vCenter environment. We were fortuneate enough to be given some time to set it up “right” – meaning setup a SQL cluster for vCenter, SSO in HA behind a load balancer with valid certificates. I drew the SQL straw, and it s the first time I have setup SQL clustering. I had to pull from a few different resources, none were completely what I was trying to do but thank you to Derek Seaman’s blog and the MSDN blogs for being able to answer questions when they came up. You can find more information on Windows Failover Clustering on vSphere 5.5 here (nope not on 6 yet). An over view of our setup:

Two Windows 2012 R2 virtual machines on separate hosts; SQL1 and SQL2

Each virtual machines with two NICs; one for production/client access the other for cluster communication.

Each virtual machine has 2 drives; 60GB “C” for OS and 20GB “D” for SQL installation

3 XtremIO drives presented via VPLEX

AD accounts for SQL and SQL Agent were created in AD

IP addresses for each of the SQL virtual machines, the Windows cluster, and the SQL cluster; for this setup that is 4 total.

Windows was installed, patched and joined to the domain. On each virtual machine I ensured that Windows Ethernet0 was first in the biding order and used for “production.” NIC1 would be used for cluster communication. Ensure RSS is not enabled on the NICs.

With networking setup, add the .NET 3.5 (needed for SQL), and Failover Clustering features. Next, shutdown SQL1, and SQL2. On SQL1 I added 3 new drives as physical RDMs; 1 each for the vCenter, and vRealize Automation database files and 1 for a quorum volume. The mapping file will need to be stored on a shared datastore that is accessible by both hosts. When adding these to SQL1, add 1 at a time, select the next available SCSI adapter – for example the first volume was set to 1:0, the 2nd 2:0, and the quorum volume 3:0. The OS (C), and SQL (D), drives were 0:0 and 0:1 respectively. A few considerations:

In this format you will use all 4 of the possible SCSI controllers

The SCSI controllers for the RDMs must have their SCSI Bus Sharing set to Physical (the virtual machine has to be off to do this, thus why I mentioned shutting down SQL1)

Drives need to be added in the same order on both SQL1 and SQL2

Repeat adding drives to SQL2, except this time select “Use an existing virtual disk” and navigate to the SQL1 folder on the datastore to select the mapping file. You can check SQL1 to ensure that the correct mapping file is on the correct SCSI controller. For example, here you can see that SQL1_2.vmdk is on SCSI 1:0, so I want to make sure that SQL1_2.vmdk is using 1:0 on SQL2.

Power on SQL1, and SQL2; configure each of the drives in Server Manager >> File and Storage Services on SQL1. DO NOT configure the drives on SQL2, when you build the cluster, and ultimately fail the cluster over from SQL1 to SQL2 it will mount those volumes. SQL2 should only have C (OS) and D (SQL) configured.

Now that the drives on SQL1 are configured, open Server Manager >> Tools >> Failover Cluster Manager and click Create Cluster… Follow the wizard to add both SQL1 and SQL2. You will likely receive warnings about drives 0 and 1 since they can’t be used for the cluster. When prompted, DO NOT select the option to add available storage, otherwise it will add D. We only want to add the RDMs to the cluster. During this process is you will also create an AD object for the Windows cluster and assign 1 of the 4 IP addresses mentioned earlier (2 others were already given to SQL1 and SQL2)

Once the cluster is created, expand Storage and click on Disks. Uncheck the 20GB D drive, in my case I added 2x 250GB volumes, and the 50GB volume. The 50GB volume will be the quorum drive. Once added they should all be online. Click on your SQL cluster, and in the Actions pane click More Actions >> Configure Cluster Quorum Settings…

When the Wizard opens, click Next and select Use default quorum configuration; in my case it correctly selected the 50GB volume for the quorum witness. Now if I go back to Storage >> Disks you can see Disk 3 is set as the Disk Witness in Quorum.

Next create an AD object for the SQL cluster, I opted to go with option #2 for from this MSDN post. This allows the AD Windows cluster object to manage the SQL AD object. You can finally start the SQL installer. When the SQL Installation menu opens, click Installation >> New SQL Server Failover Cluster Installation.

Select the options you wish for SQL, ensure that you install to the appropriate drive – in the case of this example D. Verify the installer correctly identified the RDMs as the data destination, and complete the installation. Once the installation finishes, enable FILESTREAM on the SQL Server Service. Start SQL Server Configuration Manager, click on SQL Server Services, right click on SQL Server and select properties. Click on the FILESTREAM tab and enable for both SQL and I/O access.

Start the SQL installation on SQL2, except this time select Add Node to a SQL Failover Cluster. Select the cluster and most of the configuration options will be pulled from SQL1 (including the SQL installation directory, in my case D). Once complete you should be able to connect to the SQL instance, I did a named instance and called it VC so I connect to sqlc1vc

Apologies for lack for explicit steps, it would have been quite a lengthy post but happy as always to answer any questions I can. Here is a quick demo where I connected to the instance, created a DB, ran a query to list all the DBs, powered off SQL1 (hard shutdown, no role draining) and re-ran the query. It took a few seconds for the failover to happen but was fairly quick.

I wanted to share some of the example Ansible playbooks used during last Wednesday’s US #vBrownBag. During the show I went over examples of how you can use Ansible to create, clone, and update virtual machines in vCenter without the need for other provisioning tools. Based on my testing (and I’m still learning as well), the items noted in the comments are the bare minimum needed to run the playbook, even though the official documentation may currently state otherwise. If you are already using Ansible for configuration management, this is a handy option to have as you can perform the provisioning tasks without leaving Ansible.

Scenario: You try to install the VMware vCenter Server Appliance (VCSA) or Platform Services Controller but receive an error during the installation. After correcting the problem during installation you attempt to re-install the appliance but receive the following error message:

Virtual Machine Already Exists

As of the release candidate of vSphere 6.0, the vCenter Server Appliance installation wizard does not clean up deployed virtual machines after failed deployments. Virtual Machines deployed are still present on the selected ESXi hosts inventory. Log into the ESXi host, power off, and delete the virtual machine from the failed deployment.

If you attempt to redploy the virtual machine with a different name (appliance and host name) using the same IP address you receive the following error message:

Encountered an internal error. see /var/log/firstboot/vmafd-firstboot.py_6399_stderr.log

Because the virtual machine was deployed and powered on, there is a duplicate IP address on the network.

Generally, installing virtual appliances has been pretty straight forward – import an OVA and enter the necessary details in the deployment wizard, or access the virtual appliances management interface (such as those typically on port 5480 from VMware). However, as of the Release Candidate for VMware vSphere 6.0, the vCenter Server Appliance (VCSA) installation takes a much different approach than what you’ve been used to.

A few vCenter Server Appliance prerequisites

First, it should be noted that you can only install the vCenter Server Appliance (VCSA) from Windows. I was first turned onto the VCSA because I was at an all OSX/Linux shop so it made sense to use something we were accustomed to using already. For now, you’ll need a Windows box to at least get the appliance deployed; then you can punt (please note also this is based on Release Candidate (RC) code and could change in the final release).

You CAN deploy the VCSA 6.0 to both ESXi 5.5 or 6.0 host. If you currently have a 5.5 environment you can deploy the VCSA without upgrading your hosts, but if you did not take the plunge into 5.5 you’ll have to bring at least one host online running 5.5. or 6.0.

Finally, before getting started, you MUST create DNS records before running the installer. I was struggling with the new installer because I’ve just been used to doing my DNS records after I deployed the VCSA, but before running the setup through the management interface. However with a little help from Emad Younis (@Emad_Younis) I was able to point me in the right direction. With 6.0 all of the configuration is done from the initial setup wizard. When it’s finished installing, vCenter is ready to run.

The installation wizard will NOT give you an error if this does not exist, instead it will fail during the installation!

As you can see here I have my forward and reverse DNS records ready to go on .9

Installing the vCenter Server Appliance

As with the older versions of the VCSA, it all starts with a download; however in this case you will be downloading an ISO image. Once the ISO image is downloaded either mount the ISO on your Windows box or extract the ISOs into a folder (as seen here).

Now that you have access to the files, drill down into the vcsa folder, there you will find the VMware-ClientIntegrationPlugin-6.0.0. Install this application on your Windows box (double click, Next, Accept/Next, Next, Install, Finish). Once the plugin finishes installing, back up one folder level and open the index file. As you can see here I am on Windows Server 2012, thus at least IE10 however opening the index in IE10 gives me a warning that I need to upgrade to at least IE10 or 11, so yea I’m going with Chrome. As with any plugin, you must enable it in Chrome. Click on the puzzle piece with the red x, then click Always allow and refresh the page and click the Allow button.

You should now see the vCenter icon along with a large Install button, click on it. You will get a UI very similar to what you would get deploying a virtual appliance.

1. After carefully reading the license agreement, printing it for your records, and having it signed by an attorney, click the I accept… check box and click Next.

2. Now you can chose to deploy to your target server. Specify your ESXi host (5.5 or above!), username and password – now click Next.

If you are using self signed/untrusted certificates click Yes when prompted.

3. The next step is to name your appliance. In my case, like I have created in DNS, my appliance name will be vxprt-vc02.vxprt.local. Click Next

4. On the deployment type you can chose to install an embedded Platform Services Controller (which includes Single Sign-On in vSphere 6.0), just the the PSC, or just vCenter. You can have multiple Platform Services Controllers, and they can be different types. For example you could do a stand-alone PSC and have an embedded one with the VCSA. When the installer says “embedded” it really just means the components will be installed on the same virtual appliance as vCenter. I’ll be doing embedded here. Click Next

5. Chose whether you have an existing SSO domain or you will be creating a new one. I will do this install as a greenfield type deployment, so select Configure Single Sign-On. Now enter the administrator password, and domain. To stay consistent with what I know about SSO, I’ll enter vsphere.local here. Click Next.

VMware vCenter Server Appliance (VCSA) step 5 – configure SSO

6. Select the appliance size that supports your environment, including the new “tiny” deployment for up to 20 hosts. Click Next

7. Select the datastore you will to install to, and whether to THIN PROVISION the vmdk (no VMware, I’m not calling it “Thin Disk Mode” – THIN PROVISION!). Click Next

8. If you’re an Oracle shop, you have a choice on step 8, otherwise just click Next.

9. Chose a network (this will be based on the host you deployed to), and how to assign IP information including the host name – This MUST match DNS. I’ll select static as that is what I would want to do for this type of server. Finally enter the NTP server and click next (I’ve also enabled SSH so I can connect directly to the virtual machine.

10. Review the settings you’ve enter, make sure your IP information and host name are all correct and click Finish. The installation of vCenter and the VCSA will start. You’ll even see it installing packages, that’s right this is a ground up build, not just a bunch of packages pre-installed on a virtual machine!

VMware vCenter Server Appliance (VCSA) installation process

Once the installation is complete, you can connect to https://fqdn/vsphere-client (no more 9443! One less question on the VCP6 I guess ).

So far on the release candidate I’ve had trouble deploying to a port group on a VDS (it gives errors almost immediately) even though it appears as a valid port group on the network settings page. It would be nice if VMware added more validation on the various steps to ensure there will be no errors during the installation. If you do run into an error, you need to re-run the installation wizard.