2011-01-31

For the past 6 months we have been working on a joint project. It has actually been kept pretty quiet, and it is now time to make this public.

Previous VMware vSphere books have focused on how to master the technology, deep-diving into certain elements and giving tips & tricks that help you manage your virtual infrastructure.

But we felt there was something missing in all these books. How to design an infrastructure, accounting for all the elements that you need to consider. For example:

What kind of servers should I use?

Which storage - NFS/iSCSI/FC?

How to scale an vCenter server appropriately?

and many more similar questions.

This is how VMware vSphere design was born...

The three of us collaborated on the book, to not only explain how to configure each element of your infrastructure, but to make you think about all the options available, and how each choice can impact the overall design. It should help you find the right solution for your environment – because no “one size fits all”.

It is the only book focused on designing VMware vSphere implementations. It is written for engineers and architects who plan, install, maintain and optimize vSphere solutions.

The book details the overall design process, server hardware selection, network layout, security considerations, storage infrastructure, virtual machine design, and more. We debate the merits of scaling up servers versus scaling out, ESX versus ESXi hypervisors, vSwitches versus dvSwitches, and FC, FCoE, iSCSI or NFS storage. We show you which tools can be used to monitor, to plan, to manage, to deploy and to secure your vSphere landscape. We run through the design decisions that a typical company might face, and question the choices you come to. The book is packed with real-world proven strategies. VMware vSphere Design examines how the virtualization architecture for your company should ideally look – be it a newly deployed environment or an optimization of the existing infrastructure.

The VMware vSphere design is now available for pre-order on Amazon and will be in the stores around the middle of March 2011.

We would like to thank Jason Boche for acting as the technical editor for the book.

We hope you enjoy reading this book, as much as we enjoyed writing it.

2011-01-19

This post was triggered by a very informative conversation I had with William Lam this week. Thanks William!

<Begin Quote>

vMA - vSphere Management Assistant, is a virtual machine that includes prepackaged software such as a Linux distribution, the vSphere command‐line interface, and the vSphere SDK for Perl. Basically it is the missing service console for ESXi. But it's more than that too.

This allows administrators to run scripts or agents that interact with ESX/ESXi and vCenter Server systems without having to explicitly authenticate each time. vMA can also collect ESX/ESXi and vCenter Server logs and store the information for analysis.

</End Quote>

Ever since VMware announced that ESXi will be be the de-facto platform going forward for the hypervisor starting from 4.1 - a large amount of speculation was raised about how to continue forward, and how we will continue to perform what we were doing up until now on the service console - with the console that is gone.

VMware is pushing all of us to move to the vMA to do all this administration.

Now let me share a trail of thought with you. (I would like to stress, these are my thoughts - and are based on my speculation)

Does the vMA have a future? Will there be a need for it down the road? Up until now one of the two limiting factors that were not available in was esxcli and esxtop. Since PowerCLI 4.1.1 this is no longer a limitation.

So my question to you is, what do we need to keep the vMA for? If someone would tell me because of the built-in syslog server which is available - that does not sell me.

The amount of API objects that are currently available for use in PowerCLI - that are not available or exposed through the Perl SDK, has continuously been rising with each new version released.

What is the reason to keep the vMA around in the future? What (if anything) is the necessity for you to have a vMA? Is there anything that you cannot do today with PowerCLI that can only be done with the vMA?

And if that is case - why go down that route and add another VM that has to be managed, has to be patched and will use up resources on your vSphere infrastructure?

2011-01-17

I would close up what we have learned from part 1 and part 2. During my work with the VMware Support on this case, one of the questions I asked, was is it possible to to inject a parameter into the customization process, but the only option was to import a full sysprep file. This would in essence break a good deal of the automation process, such as the rename of the OS to match the deployed VM name. So this was out of the question.

Unfortunately there was not much I could do here. So I rebuilt a new VM. But I had to take some certain precautions. Once your KMS is registered in the DNS and new OS connects to the network and gets an IP, it automatically looks up for a DNS record for the KMS server and will activate - which is exactly what I did not want.

The machine was started disconnected from the external network. I applied a MAK Key to the machine and then connected it to the network. Once a machine is activated with a MAK key, it will not try to contact a KMS server, so I was set.

So just to re-cap

Build your VM

Customize the OS - including your default profile settings

Do not connect it to the network

Install a MAK Key

Connect to the network

Activate the MAK key

The last part of the puzzle was how to change the MAK key to KMS for all the VM's that will be deployed from this template? This was actually the easiest part. In the Customization Spec you can enter a License Key, and yep you guessed it here you insert the MAK key.

Machine is deployed, the Customization Spec injects the xml (with the MAK key) into the VM, the vm is sysprep'ed (without a problem because you have 3 ReArms left) and when it comes back up automatically is activated with the KMS server.

A few things still puzzle me though. All of this worked - flawlessly on vCenter 4.0 - I did not have to go through this whole process with MAK and KMS and USD and BS! So did something change? Was there a change in the way the machines are customized from the previous version?

The deployment and templates were a bit icky after the upgrade. I could not edit of my previous templates in vCenter - they all had to be re-registered VMware KB………

I also had another issue of not being able to deploy any templates at all (Windows / Linux) with customization until the vCenter Server got a kick in the butt and was restarted.

Small little things that do not add up. I am happy to say that I did manage to educate VMware support a bit on how this works, and in turn I learned a decent amount in the process of how the VMware deployment process works as well. All in all it was an educating experience all around. I was told that the information learned from this case will possibly be used in a VMware KB for all of our benefit.

Over the past 8 months this template has been updated with Microsoft security patches (released once a month) and during that time the template OS was activated - while connected to the network.

After the upgrade to 4.1, I noticed that I could no longer deploy a Windows 2008 R2 template with customization. (There were several issues here. In the beginning I could not deploy a template at all - but after a restart of the vCenter server and an SR submitted to VMware on the issue that was solved). But when customizing the the template - either with a manual Customization Spec or with a previous one we had been using for almost a year, the customization would not work. None of it.

Now this can become extremely annoying. Because once you get used to working with deploying a template with a customization spec, to do it manually takes time and can be cumbersome and prone to inconsistencies.The VM name would not be changed, network settings would not work, VM's were not being deployed with their vNIC's connected. There of course is a workaround to the whole problem, to do it manually but as I said before, this is not ideal.

Last week I finally found the reason this was happening, and partly for my own benefit (documentation) and to share the experience, let me explain what was happening. When a Windows (7 or 2008) OS is activated with a MAK License Key, the OS is automatically set with a ReArm count of 3. This you you can see with a running the following command.

Once your OS has been activated through a KMS server - your ReArm count will be set to 1. If you try and change it back to a MAK key thereafter - the Rearm count will still be kept as 1. So what happened? Somewhere along the way, my template was was activated with a KMS Key - which meant I only had 1 ReArm left.

During the deployment process of a VM with a Customization Spec, a number of files files are injected into the VM by vCenter.

Expanding C:\Windows\TEMP\vmw979D.tmp\guestcustutil.exe Expanding C:\Windows\TEMP\vmw979D.tmp\imgcust-reboot.exe Expanding C:\Windows\TEMP\vmw979D.tmp\sysprep\guestcustutil.exe Expanding C:\Windows\TEMP\vmw979D.tmp\sysprep\sysprep.xml Expanding C:\Windows\TEMP\vmw979D.tmp\sysprepDecrypter.exeThese files are then moved later on to c:\Sysprep Moving directory 'sysprep' to 'C:'vCenter takes the information passed from the the Customization Spec, decrypts the info and creates a new sysprep.xml file which is then in turn called from the customization process Executing command C:\windows\system32\sysprep\sysprep.exe /quiet /generalize /oobe /reboot /unattend:C:\sysprep\sysprep.xml

Now how do I know all of this? A bit of reverse engineering. The deployment logs are located in C:\Windows\Temp\vmware-imc. The Sysprep logs are located here: C:\Windows\System32\sysprep\Panther

Two log files are created during the Sysprep process - setupacct.log and setuperr.log The first is the activity of the Sysprep and the second reports the errors that occurred. When deploying a VM with an activated KMS license - I was getting these errors from the setuperr.log

2011-01-16

I have been dealing with an issue that has been bugging me for quite a while. It has to do with the deployment of Windows 7 or Windows 2008 R2 VM's and License Activation.

First let's describe the environment and situation. The whole infrastructure is at 4.1, vCenter and all ESX hosts as well. It was upgraded recently from 4.0. 7 months ago I created my templates - with all my customizations. Some of these customizations included different OS settings, menu sizes, toolbars etc. In order to copy all of these settings to the default profile, once upon a time all you had to do was to copy that current user profile to the Default User Profile, and subsequently every user than would logon thereafter would have all those settings defined. Starting with Windows 7 and 2008 (perhaps also Vista - I am not sure, we gave this version a skip) this was not the recommended way to this. There is a detailed Microsoft KB that explains the method - which is to Sysprep the machine and provide a setting in the unattend.xml file which will copy the profile. This can be done manually but that is not recommended. You should use the WAIK. This tool is available from Microsoft.

After installing the software you attach a Operating System Image and create an answer file. In that answer file you can search for the Copy Profile option and set:

<CopyProfile>true</CopyProfile>

My first problem I ran into then was that I was not aware of the fact you can only Sysprep a machine 3 times, thereafter you will not be able to do it any more. This of course led me to a problem of after making changes to the VM and Sysprep'ing again and again and again - I could no longer continue with this Template (Thank you VMware for snapshots!!).

There is a solution to this issue which can avoided by adding an additional flag to the unattend.xml file <SkipReArm>1</SkipReArm>.

The Activation grace period is typically 30 days. It begins after Windows Setup finishes and the computer boots for the first time. While there is no limit to the number of times that the Sysprep command can run on a computer, there is a limit to the number of times Windows can be rearmed. Typically, a system can be rearmed only three times. Using this setting enables you to run the Sysprep command multiple times without resetting the activation clock.

And now I finally had my template customized, and ready for deployment.

Now over the past 8 months the we have implemented a Microsoft KMS server. Before that let us go into what has changed since the days of XP/2003 in terms of licensing/activation. With Windows 2003/XP organizations were provided with a VL (Volume License) Key. That meant I could put the serial number in the image / template / Sysprep file for each and every machine. Once installed there was no further action needed. Starting with 2008/Windows (remember we skipped Vista) we were provided with 2 different License Keys, a MAK key and a KMS Key.

Frequently Asked Questions About Volume License Keys. A MAK key is one that you add to the OS and that has to be activated with Microsoft. This can be done over the internet or with the VAMT (I will not go into how to provide a proper licensing mechanism for your OS's in your organization). This is not an automatic process and for a small amount of OS's it is quite suitable. Christian Mohn has created a good explanation of how to use this tool. But when you are talking about hundreds and thousands of operating systems, this does not scale well. Also not always do you want to open your firewall to allow each and every computer to activate with a Microsoft Server somewhere "out there". Therefore Microsoft introduced the KMS (Key Management Server). It is a role that you can install on Windows Server 2008 which will act as your activation server for all OS's in the organization. There is a record in the DNS that is created which each new OS will look for by default and if found will activate the OS - automatically. Much better for a bigger environment.

2011-01-12

I would be interested in hearing your opinions on what you use for monitoring your VMware environment. I am conducting a study on what tools you find useful for monitoring and Capacity Management/Planning

If you would be so kind as to answer the poll below and if you could leave a comment below on why you made your choice - it would be highly appreciated.