For our Citrix XenApp environment that runs on ESXi, we tend to keep a 1:1 ratio when it comes to physical cores and vCPUs. In other words, an ESXi host with 12 cores, will only be running 6 XenApp VMs with 2 cores or 3 XenApp VMs with 4 cores or any other combination, just as long as the vCPUs configured on the VM, never exceed the number of physical cores present in the host.

What about Hyperthreading? The extra HyperThreads that a physical CPU offers are not used in the core to vCPU ratio because specifically in a Citrix XenApp workload, they don’t offer a guaranteed performance boost. HyperThreading will be enabled in the BIOS though because it doesn’t hurt the performance.

This post by Frank Denneman explains how ESXi treats physical CPUs, cores and HyperThreads much better than I could. Be sure to read it.

When administering a vSphere environment with multiple admins, it sometimes happens and ESXi is put into maintenance mode, VMs are moved and when the ESXi is available again we see that our 1 vCPU to 1 core rule is not kept. Therefore we’re using this simple PowerShell script to quickly see if all VMs have their 1 to 1 ratio delivered.

I’m no PowerShell whizz and I bet Alan or Luc can do this in a one liner, but here it is:

Veeam is about to get its 100,000th customer and is launching an interactive contest for a chance to win a trip around the world and other prizes (Google Glass, iPad and Microsoft Surface).

To participate, visitors need to register and predict the location of Veeam’s 100,000th customer on the interactive map. The closer you are to the right spot, the better chance you have to win the trip around the world and other prizes.

We currently have a live pre-registration page. The main contest will start next week.

Join in on the fun! We hope you and your readers will participate and celebrate this great achievement with us.

BOSTON – January 28, 2014 – VMTurbo, provider of the only Software-Driven Control for virtualized environments, today announced a new version of its flagship product, VMTurbo Operations Manager, enhanced with control modules for storage and fabric to drive virtualized environments to their desired state and maintain control in that state across the data center and IT stack. These new solutions enable 30% improvement in utilization while providing greater control over all aspects of the environment the application workload touches – from compute and storage to fabric and cloud.

VMware has been clear about it, you should switch to the vSphere Web Client and stop using the good old Windows vSphere Client. Being best kid in the class, I of course tell my customers to start using the vSphere Web Client when possible and I work with it myself most of the time. During an upgrade to vSphere 5.5 the customer’s admin suddenly asked me where he could find the attributes in the vSphere Web Client. To my surprise, the vCenter Server attribute fields are not available in the vSphere Web Client. I did know that VMware now offers tags and categories for a lot of objects in vCenter and we should use these in favour of vCenter Attributes, but I wasn’t aware that the vSphere Web Client no longer shows the attribute fields. Vice versa, the tags and categories only work in the vSphere Web Client and not in the Windows vSphere Client.

What exactly are tags and categories?

According to the information in the vSphere Web Client: “Tags are a new feature of this vSphere release. Their purpose is to allow you to add metadata to objects. Tags allow you to bring information about your virtual infrastructure form outside vSphere and attach it to objects inside so that actions and decisions can be taken on the basis of that information. In order to avoid conflicts between the many possible uses of tags, tags are organized into categories.”. When looking at tags and categories you can best see them like this:

vCenter attribute “name” is now called a category and you can set a category to be tagged only once or multiple times per object. For example the “Datacenter”.

The value of an attribute is what is now called the name, for example “Open Line”.

How to use tags and categories

Tags can be assigned to the following objects:

Cluster

Datacenter

Datastore

Datastore Cluster

Distributed Port Group

Distributed Switch

Folder

Host

Network

Resource Pool

vApp

vCO Scheduled Workflow

vCO Workflow

Virtual Machine

To give you a few examples of how to use tags and categories I created a schema that I could use in the Open Line datacenter in which we host VMs for multiple customers. What does this vSphere environment look like? For most customers we host VMs in a shared environment but some have their own dedicated ESXi hosts (and cluster). We offer different types of backups for VMs: file level and vmdk level. We offer different types of management, some customers only buy a VM and they take care of their own guest, but for a big part we also manage the guest at the OS level and therefore we want to know what is running inside the VM. This by far is not a complete list but I think it is enough to help you understand how to work tags and categories.

Virtual Machine level

For each VM I would like to know the following details:

Datacenter. This can only be one datacenter at the same time.

Customer. This can only be one customer name at the same time.

Backup Method. Several values can be needed here, since a VM can be backed up through VMDK level and also have SQL Agent backup.

Guest Management. This can only be one, either Open Line manages the Guest OS or the customer manages the Guest OS.

DR. This can also only be one value per VM, either the customer has a DR contract or not.

VM Quality. This can only be one value per VM, either Gold, Silver, Bronze, etc.

Storage Quality. A VM could potentially have more types of storage quality since this can be per VDMK.

In the above, where I write about “can only be one”, what I mean is that a VM can for example not be in two datacenters at the same time. But there will be multiple tags for this category. In the datacenter example, there would be a tag “DC-1” and “DC-2” but only one tag can be attached to the VM.

Creating categories and tags

First we need to create the categories. Let’s start with the datacenter. Open the vSphere Web Client, go to HOME and click “Tags” on the left hand side, followed by clicking the “items” tab and “Categories”. Now click “New Category” and enter the following:

Category Name: Datacenter

Description: Datacenter for this object

Cardinality: One tag per object

Associable Object types: All objects.

Now switch to the “Tags” section and create a the following new tags:

Name: DC-1

Description: Datacenter Number One

Category: Datacenter

Name: DC-2

Description: Datacenter Number Two

Category: Datacenter

Now navigate to a VM, click the summary tab and see the section called “Tags” and click “Assign….” in the lower right corner of the “Tags” section. Choose the Datacenter “DC-1” tag and you’re done. The same can be done for hosts, datastores, networks, etc. if the Category can be used on those objects that is. It is also very easy to quickly add a tag to multiple objects. For example go to the cluster, select the tab “Related Objects”, click the section “Hosts” and select all hosts, right click and choose “Assign Tag…”.

Reporting on tags

Using tags should make it very easy to quickly find objects that match a given tag, but the search / reporting function isn’t that functional yet. It seems that the “old” search function is moved to the vSphere Web Client without adding functionality for the tags and categories. It is possible to search “for” a tag but not “on” a tag.

For example I want a list of Virtual Machines in DC-1. On the home screen go to the “New Search” section on the left hand side and create a search in the middle pane. Click “Advanced Search” to search for a “Virtual Machine” that satisfies “any” of the following criteria. Strange thing is that in the properties I cannot select a Tag.

The closest thing to getting a list of Virtual Machines from DC-1 is to search for: “Tag Name” contains “DC-1” and click Search. What happens now is that you get a list of all tags that have “DC-1” in the tag name. You can now click that tag “DC-1” and get the list of related objects. But you will get all objects with that tag and then in the bottom right, you can choose to export as CSV. In this CSV you then have to filter out all non-VM objects, which is not very convenient.

Work around

There is a work around that is actually too stupid, but what you can do is created an object tag for each object. If you assign the tag “Virtual Machine” to each virtual machine, you can use that in an advanced search. To do this, create a new category named “Search-Objects” (or anything you prefer) and make it “one tag per object”.

Now create a new tag named “VirtualMachine” and connect it to the “Search-Objects” categorie. Now go to “New Search” select “Advanced Search” and create the following search:

You have to enter something in the criteria line, so I just picked “Virtual Machine host name” but it could have been anything. Now click search and you will get a list of all Virtual Machines in this vCenter Server. Select them all, right click and assign them the tag “VirtualMachine”. You have to select VMs and Templates separately or you can’t assign a tag. Now create a new search:

“Everything” that satisfies “all” of the following criteria:

“Everything tag name” – “contains” – “dc-1”

“Everyhting tag name” – “contains” – “VirtualMachine”

And there you have it, a list of all VMs in datacenter one.

vCenter Attributes migration wizard

Luckily VMware has offered a migration wizard that will convert the vCenter attributes into tags and categories. Let’s walk through it using the following example. I have two vCenter custom attributes named “Customer” and “Datacenter”. Using the old Windows vSphere Client, you can see them and their values.

In the vSphere Web Client go to Home, then “Tags” on the left hand side and use the “Getting started” tab. In the middle pane you can now choose “Convert Custom Attributes”. A wizard pops up and in step 2 I get a list of all attributes I can convert. Here I select all the “customer” and “datacenter” entries.

Click next and now see the categories that the wizard suggests. By clicking on a category you can edit the category. After you’re satisfied with the entries, you can click next and see what the tags will be that will be created. Here you can also edit the description.

Click Next and Finish to complete the wizard. Now check if the VM indeed has these new tags. The wizard does not remove the vCenter attributes.

PowerShell commands

Since VMware PowerCLI version 5.5R1 and up there are a number of commands available to work with tags:

Get-Tag

New-TagAssignment

Remove-TagAssignment

And the –Tag property has been added to many existing commands, for example:

Get-VM -Tag “DC-2″ will list all VMs with a tag “DC-2”.

Conclusion

Working with tags does take some getting used to, but in the end they will give you more freedom and ease of use but I do hope reporting will improve a little in the next version.

During my migration from vSphere 4.1 to vSphere 5.5, I ran into an issue I had never experienced before. When your VMware HA network is partitioned, vCenter will not let you perform VMotions. At first I was surprised but after some searching I learned the reasons behind it and it makes complete sense now.

I had a cluster running ESX 4.0/4.1 hosts with a very basic network config. Service Console was in VLAN100, VMotion in VLAN110 each in different IP ranges. The new ESXi 5.5 hosts would get a completely different network config. First vmkernel port in VLAN200, second vmkernel port in VLAN210 both enabled for management traffic. And there were two VMotion vmkernel ports in VLAN220. When adding these new hosts to the same clusters as the ESX 4.0/4.1 hosts, I received a number of HA error messages:

The vSphere HA Agent on the host is alive and has management network connectivity, but the management network has been partitioned.

This state is reported by a vSphere HA Master Agent that is in a partition other than the one containing this host.

The vSphere HA protected VMs running on the host are monitored by one or more vSphere HA Master Agents, and the agents will attempt to restart the VMs after a failure.

This seemed logical since the old hosts had their Service Console interface in a different IP range than the new ESXi 5.5 hosts. Shouldn’t be a problem, since this situation would only last one day during working hours, while I was working on that cluster and my plan was to VMotion the VMs in the cluster from the old hosts to the new hosts. And that is where I was stopped. I was unable to free an 4.1 host because VMotion was not allowed to a new ESXi 5.5 host. Fortunately, KB 1033634 “vSphere HA and FT Error Messages” came to help and explained why I could not VMotion.

And as in many situations before, it is completely logical that VMware decided to treat a partition HA situation in this way. Taken from the KB Article: “The powered-on virtual machine you are attempting to migrate will not be protected by vSphere HA after the vMotion operation completes because the vSphere HA master agent is not currently responsible for it. vSphere HA will not restart the virtual machine if it subsequently fails. To restore vSphere HA protection, resolve any network partitions or disk accessibility issues.“.

Makes perfect sense and now you know too or even better, you already knew

At my current customer I was upgrading their vSphere 4.1 environment to vSphere 5.5. When installing and configuring the first ESXi 5.5 host, I noticed a difference in the Path Selection Policy (PSP) between the ESX 4.1 hosts and the ESXi 5.5 host. Each host is connected to two Hitachi storage arrays, each a different model. In the current 4.1 config a general setting is used to have both arrays use Round Robin:

Because of the above rule in the ESX 4.1 hosts, all storage that is not captured by the existing rule sets, will be set to Round Robin (VMW_PSP_RR). Although applying this rule to my ESXi 5.5 host, would solve the problem for me on short term, I could imagine running into an issue when the customer would attach a third storage that didn’t required Round Robin, but would then somehow still default to it, because of this “catch-all” rule.

To write a very specific rule for this storage array I would need to find out what vendor and model this storage array is according to ESXi. This was actually the hardest part, since I was using the wrong command at first, but with some help of the VMware Community and specifically Tom Verhaeg, I found the right command. Let me take you through the steps.

Changing SATP Claimrule

Changing the SATP Claimrules is easiest when done on the command line of the ESXi host using the esxcli module. First we need to find out for a specific LUN, what the current path policy is by running the command below. The naa number used is the LUN that has the incorrect Path Selection Policy. You see that for this LUN the current policy is VMW_PSP_FIXED, since the Storage Array Type (SATP) is VMW_SATP_DEFAULT_AA according to ESXi.

The next step is to find out what storage array Vendor and Model type this LUN is coming from, because we need this info to create a new SATP claiming rule. Running the following command, shows that we’re working with Vendor = Hitachi and Model = OPEN-V.

Before creating the new rule, I wanted to check the current SATP rule list, so I could also check after creating the rule, if there was a change. For easy of reading I only display the results here that were important to me:

COMMAND:

esxcli storage nmp satp rule list

RESULT:

Name

Vendor

Model

Rule Group

Claim Options

Default PSP

VMW_SATP_DEFAULT_AA

HITACHI

system

inq_data[128]={0×44 0×46 0×30 0×30}

VMW_PSP_RR

VMW_SATP_DEFAULT_AA

HITACHI

system

The first line tells ESXi that if you find a storage of Vendor Hitachi with specific claim options “inq_data[128] ={0×44 0×46 0×30 0×30}” (which I don’t fully understand), then the VMW_PSP_RR policy should be used. The second line says to apply the system default connected to VMW_SATP_DEFAULT_AA for all Hitachi arrays. Let’s check what the default for VMW_SATP_DEFAULT_AA is, although we already know by what we’ve seen before.

What the admin of this environment did in the past, was changing this “overall” default to VMW_PSP_RR, which is not the approach I want to take. The above table explains why the ESXi 5.5 host is now working with the VMW_PSP_FIXED policy. We’re now ready to change the SATP rule using “HITACHI” as storage Vendor, “OPEN-V” as model type. We’re telling the ESXi to use VMW_SATP_DEFAULT_AA with a PSP of “VMW_PSP_RR” when Venor and Model match our specification:

To check how this worked out, we check the satp rule list again:COMMAND:

esxcli storage nmp satp rule list

RESULT: (filtered results)

Name

Vendor

Model

Rule Group

Claim Options

Default PSP

VMW_SATP_DEFAULT_AA

HITACHI

system

inq_data[128]={0×44 0×46 0×30 0×30}

VMW_PSP_RR

VMW_SATP_DEFAULT_AA

HITACHI

OPEN-V

user

VMW_PSP_RR

VMW_SATP_DEFAULT_AA

HITACHI

system

To check if this changed the way the policy was applied to the LUNs, run the command below. In the documentation I found that the host should check the claimrules every 5 minutes, but I decided to reboot the host after adding and loading the new SATP rule. Maybe I was just being impatient

WARNING: Changing Path Selection Policies or SATP rules, should only be done when you are 100% sure of what you’re about to do. Always check the VMware HCL to see what the prefered policy is for your storage array in combination with firmware and ESXi version. Also check what your vendor’s documentation recommends.

You find yourself sitting at your desk on a Saturday morning waiting for the cloning of a VM to finish. It seems to take forever and I just wished I could run VSAN in my homelab to get some more speed out of my kit. I have tried to run VSAN on my home mode ESXi hosts, but my RAID controller is not compatible (yet) with VSAN, so I had to remove it again and now I’m waiting for the official release.

While waiting I browsed through some VMworld posts in my twitter feed and there was a link to William Lam’s post about his Barcelona session: “VMworld Barcelona #NotSupported Tips/Tricks for vSphere 5.5 Slides Posted”. Browsing through his slides, I got a really stupid idea. What if I would run Virtual ESXi with VSAN included in my home lab, would it be faster than my current NFS NAS that keeps me waiting forever? The virtual ESXi would only be used for offering the VSAN datastore to the physical ESXi hosts in the cluster there is no need to run VMs in the memory of these virtual ESXi hosts. Hmmm, sounds like a plan to test this config, let’s do it.

Disclaimer: Don’t use the data I got from my tests as a reference for anything. My kit is not good enough to produce serious numbers that you can relate to business environments. This is testing just for fun and for me personally to see if I can get some more speed out of my current environment even it means setting up this idiotic configuration.

To start with, this is what I have in my home lab:
- 2x ESXi host with 32GB each and Intel i5-3470 CPU Quad core
- Iomega IX4 200D NAS configured with NFS offering 2.7TB of storage.
- Disks on each host:

Test sequences

The test VM that is running IOmeter is fully running on the IX4-200D NFS volume.

The test VM that is running IOmeter is fully running on the physical SSD of the physical ESXi host.

The VSAN ESXi host has the virtual SSD disk on the local SSD disk of the physical ESXi host and has the virtual SATA disk also on the local SSD disk of the physical ESXi host.

The VSAN ESXi host has the virtual SSD disk on the local SSD disk of the physical ESXi host and has the virtual SATA disk on the IX4-200D NFS volume.

You see that in step 3 and 4 I’m moving the data disk that offers the real storage to the VSAN. With VSAN this is the disk that should be relieved from heavy reads and writes by using the VSAN technology, the SSD read and write cache.

In the above image you see the situation of Test04 and how the pESXi (Physical ESXi) host has a pSSD (Physical SSD) and an NFS datastore to offer to the VMs running in the pESXi host. Inside that pESXi host I’m running a virtual ESXi (vESXi). That vESXi host is offered a virtual SSD (vSSD) which is running on the pSSD and a virtual SATA disk (vSATA) running on the NFS datastore. These vSSD and vSATA disks are then used by VSAN in the vESXi host and offered as the vSAN Datastore to all the hosts in the cluster of which the vESXi host is a member. And finaly, in memory of the pESXi host, there is my test VM getting running its VMDK on the vSAN datastore. Inception to the max…..

The results

For my little home lab the most important thing is the comparison between the first test (all on the IX4) and the fourth test where I have the VSAN sitting between the IX4 and the VM. Looking at the results of the “Reallife 60% random and 65% read test”, I can see an improvement of 71 to 3865 IOPS and a throughput gain from 1 MBps to 30 MBps. That is a big improvement for a cheap home lab like this.

Even though the VSAN will cost me some overhead in RAM on the physical ESXi host since it has to run the virtual ESXi of 4GB, this configuration will bring me 2.7TB of pretty fast storage if I make the VSAN SATA disk as big as the storage available on the IX4. Of course there is the added risk of losing all VMs I put on the VSAN, if that one VSAN virtual SATA disk would fail, but hey we’ve got to be living on the edge a little don’t we.

Another option I have is to run the VSAN on the SATA disks of the physical host, which are the results shown in test number 3. That will give me a little better performance than when the SVAN SATA is on the IX4, but the difference is very small. The small difference can be explained since probably everything is written to the cache on the virtual SSD. In this configuration I don’t have the full 2.7TB available.

The fastest solution is of course to run everything on my physical SSDs on the physical host, but that will give me only 2x 256GB of capacity.

For me, putting the VSAN in front of the IX4, even by using virtual ESXi hosts that don’t have to run a VM load themselves, will greatly improve the performance of my IX4 and it will give me the opportunity to get more experience with the VSAN product even though I don’t have the proper RAID controller that is supported with VSAN.

The numbers

Remember: Don’t use them for any performance reference, I just use them to see the difference in performance in my setup.

When trying to open the console of a VM using the vSphere Webclient with the VMware vCenter Server Appliance 5.5, I received the following error: “can’t establish a connection to the server at vcsa.vanzanten.local: 7331″.

Recently I was installing an additional VMware vCenter Server in an environment in which I had replaced all default certificates with official CA certificates and ran into an issue where installation of the VMware vCenter Inventory Service kept hanging when communicating with the VMware SSO server. In the installation wizard of VMware vCenter Inventory Service you’ll be asked for the vCenter Single Sign On information. You enter the admin@system-domain account, password and the URL to the Lookup Service.

Normally, in the next step you’ll be presented with a default certificate and you continue with installation of the certificate.

However in my configuration, installation hung before this step and would never present the certificate. After some playing around I discovered that killing the OpenSSL process in taksmanager would continue the install. After installation I continued with installation of the vCenter Server on the same Windows Server, but didn’t ran into any issues. After vCenter Server installation had finished all seemed to work nicely.

I’m not sure why this happens, but my guess is that openssl is waiting for a response, which you can’t see because it is running in a different context.

now adding another vCenter Inventory Service and vCenter Server 5.1 update 1 on Windows 2012, which didn’t have the Enterprise signed certificate during installation yet. And on this server I got the above mentioned issue.

Will also report this with VMware and see if there are any consequences. Use at your own risk

This week I installed a fresh vSphere 5.1 Update 1 environment and I wanted to configure it will real world certificates to get rid of all those “Do you really really reeeeeally accept this insecure website” messages. Using the VMware SSL Certificate Automation Tool I generated all the new certificates and then started changing the certificate on the VMware SSO server. When doing this, you’ll be asked for the Master password. Since I learned a while ago in a very painful way that the Admin@System-domain password is not equal to the Master password, I had written down the Master password and was 100% sure I had the correct Master password. But updating the certificate failed with the error: Incorrect master password. Tried it a few times but it kept failing. Logged in with admin@system-domain in the vSphere Web Client and this was the correct password.

I switched to command line and tried to run some SSO Util commands to make sure my password worked and then everything became very clear. I have a bad character in the password. In the password I set during install, there is an “&” (ampersand) and in many console languages this has a special meaning. When running some rsautil commands using the master password VMware&77 I get messages like: “77″ is not recognised as a command.

In my homelab I installed a fresh new SSO just for this test. During installation I set the master password to: VMware@55. Then I tested my rsautil command: rsautil manage-secrets -m VMware@55 -a list. This worked, I got a list of … well things.

I then changed the master password using this command: rsautil manage-secrets -m VMware@55 -a change -N VMware&77. This should set the old password to the new password “VMware&77″. Check the output below and notice that the rsautil did perform the change, but also reports an error. Trying the list command with the ‘new’ password, didn’t work.

What had happened is that the master password was changed to “VMware” and everything behind the & was lost. Proof would be if the rsautil command would work with the “VMware” password and it did: rsautil manage-secrets -m VMware -a list.

I did a new test. I removed SSO and the SQL Express database and again installed SSO using the master password “VMware&123″ to see what would happen. Login through the Web Client works using the user admin@system-domain and password VMware&123 but the command line tools don’t work.

Conclusion

As long as you don’t have to use the command line tools to change anything in SSO or to recover a password, you’re fine and using the & ampersand in your password won’t hurt you. But if you ever need to change anything with the help of the command line tools, for example when you lost your admin@system-domain password, then you’re lost.

My advice is to use a master password that doesn’t have the & in it. I tested with @ and that works fine. Using the exlamtion point ! also has some issues sometimes, so I would stay away from that too. And the release notes already mention that a space in the password will also get you into trouble.

Update 2:

In some situations it can be fixed with: rsautil manage-secrets -a change I’ve also tested this in my home lab and somehow it didn’t work, later on I tried it in production because everything else had failed and it worked !!!

Recently I had a chat with Saradhi Sreegiriraju and Ed Lee from Tintri about their new version of their Tintri Storage appliance, Tintri VMstore. This new version is an update of their first model with some really nice new features. New features that (I think) make the Tintri VMstore a real Enterprise solution.

Let me first explain what the Tintri appliance is. It is a box with a fixed set of SSDs for performance combined with a set of SATA disks for capacity. The utilization of the SSDs has been designed in such a way that by using compression and dedupe, between 98% and 100% of all read and write actions can be handled from the SSDs directly. Customer experience (Tintri currently has over 180 customers) has shown that only during backups a request is sometimes NOT serviced from flash. The Tintri T540 model offers 13.5TB of usable data storage of which 2.4TB is SSD storage (before dedupe and compression).

The Tintri appliance is able to keep “hot” blocks in flash if needed and if needed, the admin can move a whole VMDK into the SSD cache. “Cold” blocks of data are periodically moved to the SATA disks, continually freeing up space in flash for active workloads. The beauty in management of the storage is that you no longer need to worry about LUNs and RAID configurations, since there is only one big datastore managed by Tintri. Where other storage mostly reports IOPS and latency based on LUNs or RAID groups, Tintri is very VM oriented and will work with you on a VM level. You can see per VM and VMDK latency and how many IOPS each VM is generating.

Tintri VM performance

The Tintri appliance is placed in your data center as one shelf and cannot be extended. If you need more storage you buy an additional appliance. Each appliance will present itself as NFS storage to your ESXi hosts. Each Tintri appliance can handle up to 1,000 VMs, depending of course on type and size of each one. Unlike other storage, Tintri enables both server and desktop VMs to coexist on the same datastore, allowing for both VDI and server apps to be on the same system.

What’s new?

On the hardware side, not much has changed. Where the very early model had just one storage adapter, this new release now has two redundant controllers and each controller has a 2x10Gbit connection. Nothing spectacular but something a true Enterprise ready storage should have.

At the software level we can find the most changes. Tintri now introduces the Tintri Clone feature. As stated in the first part of this article, Tintri is VM orientated storage and this new cloning feature is also fully VM orientated, which means you clone at the VM level, not at the LUN level.

This new cloning technique works mostly in the meta data and therefore has minimal performance overhead. When using the clone feature for VDI solutions, Tintri is able to deploy 1000 desktops including customization in little over an hour.

Another interesting observation and improvement I noted from this software release was the similarity of the software-defined storage features that Tintri delivers today to the future functionality that vVols will deliver in a couple of years. This will make it extremely easy for Tintri to support and their customers to adopt vVols when it does arrive on the market.

Tintri can now attach storage attributes and policies to a VM and set a QoS on a VM. It allows for VM and VMDK pinning to flash and you can isolate VDI IOPS from Server IOPS. To be clear, this is already present without VMware vVols.

Snapshots and Replication

The biggest changes in this new product from Tintri are the snapshot and the replication features. The snapshot feature allows you to create VM snapshots on a regular schedule and use them for fast recovery, shortening your RTO. A snapshot can be created hourly, daily or weekly or a combination of all. Per snapshot you can configure if it should be replicated and how long to keep the snapshot on the local storage and how long to keep it on the remote system. You can also choose for a crash-consistent or VM-consistent snapshot. The crash consistent is just a “dirty” snapshot of the VM, where the VM-consistent snapshot will quiesce the guest OS using the VMware Tools.

Snapshots

The new replication feature, called ReplicateVM, will replicate on a per VM basis to its partner using the snapshot schedule for each VM, which can be as often as every 15 minutes. Replication can be done in a many to one relation, meaning that you can have multiple Tintri systems replicate to one destination system.

Replication

To replicate as efficient as possible, Tintri has come up with this intelligent algorithm. First of all, you can pre-seed the target side with existing VMs or backups. Secondly there is a very intelligent dedupe and compression over WAN mechanism where the source will send a fingerprint of all the changed blocks. The target compares this fingerprint with data the target already has and then reports the missing parts of the fingerprint to the source. The source will now send the missing data compressed and the target will write this to SSD. After a few iterations the snapshot is available on the destination.

Tintri dedupe and compression

These snapshots on the target can also be used to create clone VMs on that same target that then can be offered to the ESXi hosts on the target side.

Monitoring

One of the big plusses of Tintri is the in-depth monitor that can tell you latency, IOPS, replication bandwith, etc. on a per VM basis. In most vSphere environments you need to combine vCenter Server performance graphs and your storage monitoring to get a clear view on how your VMs are performing storage wise. The storage will often give you a good view at what is happening at the LUN and storage level, but not at the VM level. Tintri offers a view on your VM performance that goes right from vCenter to the storage level. This is a very big plus when searching for possible performance issues.

Experiences

When writing this post, Tintri gave me access to their lab and I was impressed by the speed at which VMs could be deployed and how easy it is to setup snapshotting and replication. All functions available in the Tintri appliance are easily accessible and it is very simple to learn how to use them. In my opinion you will get a storage appliance that is very easy to use and maintain. No more worries on LUN sizing, pool sizing, RAID sets, etc, etc. This will save the storage admin a lot of time.

A customer of mine, who was already running a vSphere environment on Cisco UCS blades, asked to expand his environment with a number of ESXi hosts that could run a VMware View environment. To be able to run as much View desktops on each host as possible, we offered them Cisco UCS Blades equipped with a Fusion IO card, the UCS 785GB MLC Fusion-io ioDrive2 to be exact.

For this customer I had already deployed a vSphere environment a year ago, fully based on VMware Auto Deploy and it now was time to try and add a driver to this Auto Deploy environment, which I had never done before. It wasn’t an easy road, but I wrote down all the steps and hope it helps you should you ever have to do this too.

Download the drivers

The company Fusion IO has a very tight policy as it comes to driver downloads. You can only get access to the support site and download the drivers after registering the serial number of the Fusion IO card. This will grant you access to the support site, but only to the drivers for that specific card. Very inconvenient when you’re only the guy implementing the stuff and especially if the cards are already built-in the UCS Blades located in a datacenter far far away.

After some time on the phone with their support department, I was able to get a link to the Cisco Support site and download a big ISO file containing all drivers for the Cisco UCS Blade. After unpacking the ISO and browsing through the folders, the Fusion IO driver only contains a shortcut to…….. the VMware Website. And there it is, free for download, the Fusion IO driver: VMware website – Fusion IO driver

After you downloaded the driver zip, place it in a separate directory and unpack it. In the directory in which you unpacked the zip, you will now find the following:

iomemory-vsl-5X-3.2.2.869-offline_bundle-895853.zip <– this one we want !!!

Subdir: doc

Prepare the Auto Deploy image

For the deployment we need three images. One is of course the ESXi image that can be downloaded from the VMware website, second is the Fusion IO driver (iomemory-vsl-5X-3.2.2.869-offline_bundle-895853.zip ) and third very often is the vmware-fdm VIB for VMware HA.

I’m using vc-mgmt.vanzanten.local as the vCenter Server since that is the one running in my homelab, replace it by your own vCenter Server. Let’s get started:

After these three depots have been added, we can now combine packages from them into a rule, we just need to find out which ESXi image we would like to use. Let’s take a look:

Get-EsxImageProfile | Sort-Object -Property name | ft Name

You now get a list of all images available, for this case I pick the image: “ESXi-5.1.0-20130504001-standard” and use it to create a new ImageProfile called “ESXi-5.1.0-20130504001-std-FusionIO” and give it a Vendor ID.

Now we want to add the drivers to this new profile but for this we need to know the name of the software package that holds the Fusion IO driver. Run this to get a list of all software packages available:

Deploy rules

What we’ve done until now is create a new ESXi image that can be deployed to an ESXi host, but we didn’t create any rules for deployment yet. I have already explained this in earlier posts, so I’ll keep it short. Deploy this image to the ESXi host with IPv4 address 192.168.0.146 and assign the image “ESXi-5.1.0-20130504001-std-FusionIO”. Put the host in the cluster CL-MGMT and use the host profile named “ViewFusion-Hosts”.

A few days ago I blogged about my new idea in this post: “The Doctor is: IN” and I received a lot of positive comments and even my first request for assistance. So… here we go !!!

Tonight ( June 7th ) at 19:00 CEST the first session will go live. That is, if Google Hangouts works because last night during my test run there were some issues with Google Hangouts.

This first session has been requested by Ryan Conley. Ryan is originally from Nashville area, moved to Greenville for two years which is where he got his feet wet with ESX/ESXi 4.x and 5.0. He took a position in San Francisco over a year ago and is now the primary person for managing two vSphere 5.0 clusters for two separate companies. He passed VCP5 in April and as a result of the prep time he is now hooked on learning more and more. He has his IaaS exam scheduled for the 17th of June and plans to attend VMworld in SF and sit the VCAP-DCA exam since it will be 75% off. Work wise he is about to standup a DR cluster in SF and begin implementing SRM to tie the two together.

Ryan e-mailed me at Gabrie (at) GabesVirtualWorld (dot) com and asked me for help on a few subjects. In tonights Hangout On Air we will be discussing the following:

Some help, tips and tricks on VMware SRM and his Nimble Array.

Upgrading vSphere 5.0 to 5.1

Using the vCenter Server Appliance (vCSA)

Study tips for his VCAP-DCA exam

The session has ended. See the video below. Click here for the PowerPoint slides I used.

Many of us in IT read a lot of whitepapers, blogposts, how-to articles and view numerous Podcasts or training video’s to learn all the details about new products or features. Still, I don’t always get some of the details or can’t find the info I need. Meeting people at VMUGs or VMworld gives me the changes to ask for those last missing piece of information. But what if you don’t have that chance?

The Doctor is IN

When I blog about stuff I’m sometimes surprised about the comments I get and about how more people than I thought were struggling with the same questions. In responds to the comments I have been able to help quite a number of readers of my blog by e-mail and lately I did a few Google Plus Hangout sessions to help in an even better way. And that is when I came up with the following idea: Why not do a video-chat help session?

I’m still thinking about what would be the best way to do this and I may change everything from session to session but this is what I came up with until now:

Send me an e-mail ( Gabrie AT GabesVirtualWorld dot com) in which you write about a specific subject you would like to know more about or maybe an issue you have with a design or installation.

I’ll then plan a meeting with you for a Google Hangout session. This might be an one-to-one session or maybe we’ll have a session with more people and collect a number of questions.

The session will be live broadcasted and published on YouTube afterwards.

Customer had all his VMware databases for vCenter Server, Update Manager, Single Sign On (SSO), vCloud Director and vCenter Chargeback running on one big SQL Server where they shared resources with other databases. They asked me to move the databases to a new MS SQL Server because the load of the VMware databases was more than expected. I prepared the move by reading a few VMware KB’s that described how to move the databases, but still experienced some issues, that is why I wrote this blogpost to have a good manual for the next time I have to move these databases.

On the old database server make a backup of the SSO database (default name is RSA).

After a successful backup, set the SSO database to Offline

Copy the backup to the new database server

On the new database server, import the SSO database

On the security section of the SSO database, check if the user RSA_USER is present.

After this, the SSO database is available, but there is no login user connected. Usually when you create a SQL user, you also make a mapping to a database which then automatically creates the database user. In our case the database user is already present but is not yet mapped to a SQL user. That’s what we’ll do now. First let’s make sure the RSA_USER is not yet mapped:

Run the following sql query against the SSO database to show all unmapped users of the database: sp_change_users_login report

Now create a new SQL User (SQL Authentication) at the SQL Server level not at database level. Name this user RSA_USER and use the same password the database RSA_USER has. Set the default database to RSA (the SSO database).

Run the following sql query against the SSO database to map the user RSA_USER (server level) to the RSA_USER (database level): sp_change_users_login ‘update_one’, ‘RSA_USER’, ‘RSA_USER’

To check if things worked out, rerun the query against the SSO database. The RSA_USER should not show up: sp_change_users_login report

When running the queries, make sure you run them against the correct database. See image.

Next step is to create the RSA_DBA user which is only a SQL Server user and not a database user. But this user should become the owner of the SSO database.

At SQL Server level create the SQL user RSA_DBA and be sure to use the same password you previously used. (Well, you can always reset it later on).

After the RSA_DBA user has been created, open the properties of the user and now set this user as the owner of the SSO database

Your database is now ready for use. We just have to tell the SSO Server that it should look somewhere else from now on. To do this run the following on the SSO Server:

You will be prompted for a password, this is the master password you also used to login to SSO with the user: admin@system-domain or root@system-domain.

Check the jndi.properties file for the correct database reference. You can find it in C:\Program Files\VMware\Infrastructure\SSOServer\webapps\ims\web-inf\classes\

Edit the file C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties and modify all values that need to be updated.

To update the SSO DB RSA_USER password, run the command if, for example, the RSA_USER password has expired or the Database has been moved to another SQL instance: ssocli.cmd configure-riat -a configure-db –rsa-user-password new_db_password –rsa-user New_RSA_USER

Now cross your fingers and start the SSO Service. Check if you can logon to the SSO Server: https://<your SSO server>:9443/vsphere-client/ Login with: admin@system-domain

When working on a design in which there would be two physical datacenters, I started to rethink about why I would create multiple datacenters inside vCenter Server. I noticed that almost all example designs I look at that cover two physical datacenters, also have two datacenters in the vCenter Server design. But why?

When searching for an answer in the vSphere 5.1 documentation, the only reason mentioned to create a datacenter is “You can create multiple datacenters to organize sets of environments.“. But it also states that creating datacenters creates limits: “Inventory objects can interact within a datacenter, but interaction across datacenters is limited. For example, you can hot migrate virtual machines from one host to another host in the same datacenter, but not from a host in one datacenter to a host in a different datacenter.“

For organising objects in vCenter Server, I can also use folders. I can put a datacenter in a folder, I can put a cluster in a folder and I can set permissions at folder level. In other words, creating the logical design in vCenter Server to match the physical design, can also be done using folders, but without the limitations.

My basic statement is: “When I have two physical datacenters, I should not by default create two logical datacenters in vCenter Server”.

Yes, there are reasons to do create two datacenters, but those are very limited. When I asked the question on twitter I got quite a few responses and only one really demanded two logical datacenters, when creating a DMZ and you want to make sure VMs don’t travel between the two datacenters.

Most other cases in which people created two datacenters could also be matched by using folders.

I’m not saying that you shouldn’t be using datacenters, but I just want to see if my statement is correct and that you shouldn’t just blindly create multiple datacenters and thereby limiting functionality like Shared-Nothing Storage VMotion.

Hope you enjoyed the previous series on VMware vCloud Director networking. In this post I will show you how to make the vCloud Cells available through a Load Balancer using the vShield Edge. By publishing two or more vCloud Cells behind a load balancer, it is easier to separate management network from user network and you make the vCloud more redundant because you can shutdown a cell without bringing down the whole vCloud. Do keep in mind that a user session will get disconnected when you shutdown a cell, but the user can immediately reconnect and will be redirected.

Design

To better understand the configuration we need a design. Below you’ll see the design in which the ESXi hosts and the vCenter Server are in the management network (172.10.0.x range). The user network is in the 192.168.0.x range and all traffic goes over the user network to the internet. The real situation this design is based upon is a little different, since the management network is routed to the internet over another network, but that would only complicate the example.

Preparing the vCloud cells

Before we start configuring the load balancer, something have to be prepared on the vCloud cells.

The first step is to make sure the certificates are the SAME on both cells. If you generated a self-signed certificate on each cell during install, you will have to change this.

The second step is to make sure that the vCloud cells are properly syncing time.

I have been fighting an issue where the console proxy session (VMware Remote Console) would disconnect when two vCloud cells where behind the load balancer. If only one vCloud cells was running, then everything worked fine. Starting the second vCloud cell behind the load balancer would work well for the HTTP sessions, but the VMRC (VMware Remote Console) session would always disconnect. Didn’t matter which vCloud cell I would shut down, as long as just one was running the VMRC would work fine. This turned out to be an issue with the cells running a 2min time difference.

Installing same certificate on both vCloud cells

During installation of the vCloud cells you had to generate a certificate. In my case I put my certificates in a file named /etc/certificates.ks. Assuming the certificate on vCloud cell is ok, we’re now going to copy the certificate to the second vCloud cell. Let’s first check the certificates currently on both the vCloud cells. Open a browser and go to both the vCloud cells websites. You’ll see the security certificate icon telling you that this website uses a certificate. In my case I’m using firefox and when opening the certificate information, I see the following. Notice how the thumbprints are different.

Logon through SSH on the first vCloud cell, make sure you are root or if you want to do it really safe use sudo

Copy the certificates from this vCloud cell to a location where you can access it from the second vCloud cell. In my case I copy it to the data transfer directory. I have a special directory created there that holds the installation bin files of vCloud Director, the responses file and the certificates. Copy:

Make sure you have the responses file from the first installation in a location that you can access from the second vCloud cell, but actually I would have expected that you already have this since you already installed a second cell. Just to be sure:

To active the certificate, you need to rerun the configuration of vCloud Director. By using the responses file, you don’t have to answer all the database questions. In my case I need to run this to reconfigure:

After configuration is complete answer YES to have the configuration script start the vCloud service or start it yourself using:

service vmware-vcd start

Monitor the startup of the vCloud cell and wait for “Complete. Server is ready”. Monitor:

tail -f /opt/vmware/vcloud-director/logs/cell.log

Let’s check if the certificates on the second vCloud cell indeed has changed by browsing to the second vCloud cell. Open the information of the certificate in your browser and see how the thumbprints match. Fase1 completed.

Time sync of the vCloud cells

It seems that having a time drift between the two vCloud cells of more than 2 minutes will cause disconnects of the console proxy session. Since the ntpd is NOT running in a standard RedHat install, we need to make sure it is, it will at start up and that the time is synced. The ntpd depends on the /etc/ntp.conf file which is already configured correctly by default if your vCloud cell is allowed to connect to the internet. In the next steps we’re going to start ntp, make sure it will run at startup and force a time sync:

service ntpd start

chkconfig --level 345 ntpd on

ntpdate -u clock.redhat.com

watch date

Run the above on both cells and to see if time is in sync, it will run the date command and refresh every 2sec. Stop by hitting ctrl+c. Fase 2 completed. Now we can finally start with configuring the load balancer.

vShield Edge Load Balancer

To create the vShield Edge Load Balancer, we need to go to the vShield Manager. Logon to the vShield Manager and go on the lefthand side to the datacenters section and select your datacenter, not the cluster. On the righthand side click “Network virtualisation” and “Edges”.

Before we start configuring the vShield Edge Load Balancer, make sure you have all the IPs noted down that you need. The vShield Edge Load Balancer will get two IP addresses in the user network, one for HTTP, one for ConsoleProxy. On the management network only one IP is needed which can connect to the vCloud cells.

Now press the plus sign to create a new Edge device. The wizard will pop up and will ask for the name of the Edge device. For the name I entered: “vCloud-LoadBalance“. The hostname is “vcloud.domain.com“, this is the outside FQDN. You can leave the tenant field empty and I didn’t enable HA. Press next and you’ll be asked to enter the credentials for this edge device. If you want to use the same credentials you use for the vShield Manager, just leave it like it is. Click next. In this screen you can select the Appliance size, choose Compact and you can specify where the appliance should be place. Click on the little plus to configure placement. Select the Cluster/Resource pool and the datastore. I choose the “System vDC” resource pool because I feel that it belongs there. If you have better suggestions let me know. Click SAVE and then next.

In this next step we will be configuring the interfaces of the Edge device. Click the plus sign to add a new interface.

The first interface will be the external interface. On this interface we will connect two IP addresses, the HTTP (192.168.0.1) and ConsoleProxy address (192.168.0.2). Click the plus sign to add a new interface. Name it: “external“, select the PortGroup you want it to connect to. Make sure the connectivity status is “connected” and then click the plus sign to add IP addresses. Enter the IP addresses and the subnet mask. In our case: 192.168.0.1 (primary) and 192.168.0.2. Subnet mask 255.255.255.0. Save the IP settings and you will return to the first wizard. Leave Mac addresses empty, adjust MTU if needed, leave Enable Proxy ARP and Send ICMP Redirect empty. Click ADD. Now add the second interface which will be connected to the internal address 172.10.0.1 in the same way you did for the external interface.

Click Next in the wizard and you will get the “Default Gateway” screen. Enable “Configure default gateway”, select the external nic and enter the correct gateway address. In my case that would be 192.168.0.254. Change the MTU if needed. Again click Next.

Now the firewall & HA screen will ask to enable the firewall. Since this is a device that could be connected to the big bad world, enable the firewall policy and set the default to “Deny”. Click next to continue and finish to close the wizard. When you look at your vCenter Server you should be able to see the deployment of an OVF file.

When done you should see the following in the vShield Manager.

The next step is to configure the real load balancing. First we need a “Pool” of servers that offer the same functionality. In our case the Pool of servers will hold all vCloud cells. Second we then need a virtual server. The virtual server is what is published to the outside world.

Click the “vCloud-LoadBalance” we’ve just created and click “Actions” -> “Manage”. Here you’ll see an overview of the configured settings. Click “Load Balancer” and be sure that “Pools” is selected. Click the plus sign to start the “Add Pool” wizard. The name will be “vCloud_HTTP” (no spaces allowed). Next we configure the services. Enable HTTP and HTTPS and choose the balancing method “Least connections“. Leave the ports unchanged at 80 and 443.Click next. Now the Health Check screen appears. Only select HTTP (although I think HTTPS should work too) and for the URI for HTTP Service use: “/cloud/server_status“. Be sure to start with a forwardslash. Click Next.

Now we need to enter each member of the pool. Be aware that we are only defining the HTTP part of the vCloud cell, so we add only the HTTP addresses of the vCloud cells for this pool. In our case add two IP addresses: 172.10.0.11 and 172.10.0.21. Add them both with a weight of 1. Leave the defaults of HTTP and HTTPS.

Next we’ll add the Console Proxy servers to the pool. Run the same wizard again and name it “vCloud_Console“. In the Services screen you have to pay a little attention, you only need TCP over port 443, not HTTPS !!! This is because the console proxy traffic travels over port 443 but it is no HTTPS traffic. In other words, on the services screen select TCP and change the port to 443 and select the “Least Connections” balancing method. In the Health Check screen select TCP and monitor port 443, leave the URI for HTTP Service empty. In the next screen we again need to add some IP addresses. Enter the IP addresses of the console proxy of the vCloud Cells ( 172.10.0.12 / 172.10.0.22). You now return to the vShield Manager screen. A little above the list of pools you see the “ENABLE” button, click this to enable the pool and push the “Publish Changes” button.

We now have a pool of servers defined that can accept the traffic from the load balancer. Now we need to define the services that the load balancer will allow from the outside.

Click “Virtual Servers” and then the plug sign to start the “Add Virtual Server” wizard. You are now prompted to enter a name for this virtual server, use “vSrv_vCloud_HTTP“. Now enter the IP address of the outside HTTP interface 192.168.0.1. Select the pool “vCloud_HTTP” and enable HTTP and HTTPS. Leave the defaults of persistence method cookie for HTTP and SSL_SESSION_ID for HTTPS. Click Add.

Again click plus to add a new virtual server. Name this server “vSrv_vCloud_Console“. Now enter the IP address of the outside HTTP interface 192.168.0.2. Select the pool “vCloud_Console” and enable TCP and change the port to 443. Click Add. Click Publish Changes.

Today Mike Laverick drew my attention to his blogpost about BareMetalCloud. He writes about the possibilities to run your virtual ESXi host on their cloud offering. And best of all, by using a promocode, something like MikeLaverickIsAnAwesomeDudeAndThisWillGiveMeCloudForFree or maybe a shorter version, you can get free access.