Virtualization Pro

VMware Infrastructure 3 Update 1, made available on April 10 2008, introduces some core updates to ESX Server 3.5, VirtualCenter 2.5, and the VMware Infrastructure Management Installer. The biggest reason to upgrade, however, is the inclusion of Storage VMotion.

Among the core features now available with ESX 3.5 Update 1 are the addition of the Intel 82598 10 GB Ethernet controller, support of Jumbo Frames and NetQueue, additional Microsoft Clustering Services (MSCS) support, additional backup product and management agent support, additional guest operating systems, and additional server models.

I’ve been working with ESX 3.5 Update 1 for a few weeks, and the installation and behavior are indistinguishable from both ESX 3.5’s base release and ESX 3.02, with the exception of context sensitive tasks or options.

When I test upgrades, I make a point to test the upgrade in an environment with dissimilar ESX host server releases. For example, most of my hosts are ESX 3.02. When I upgrade the first one to ESX 3.5, I want to make sure that nothing goes wrong. I want to know that I’ll be able to sustain a mixed environment with all functionality. When I migrate running virtual machines through host-based VMotion to the ESX 3.5 host, and the reverse, I want to make sure to the best of my ability that nothing will fail. I also want to ensure that all of the VMware DRS and VMware High Availability rules are still enforceable with the mixed-host inventory.

Outlining a functionality matrix and the verification of the behavior is key to having no surprises during a live upgrade. Testing the update to VirtualCenter is a little more difficult but I am setting up a test environment soon to ensure that everything functions as expected in my environment. Overall, the fixes and new features make ESX 3.5 Update 1 an attractive upgrade for systems that are not there already.

VMware, Inc. released a statement today pushing the point that consumers can significantly reduce CO2 emissions and data center power consumption using virtualization. The company reports that consolidating 10 or more physical machines onto a single server can reduce power consumption and costs by 80-90%. VMware customers that have moved from a 1:1 application to server ratio to 60:1 or higher have achieved millions of dollars in capital and operational savings.

Of course, 60:1 is a very high ratio, and most users report a 30:1 or 40:1 application to server ratio. That said, it can be done; a VMware case study on the Ecclesiastical Insurance Group shows the UK based company was able to consolidate 120 servers on two host machines, with two machines for disaster recovery purposes. VMware estimates that for every server virtualized, customers can save about 7,000 kilowatt hours (kWh), or four tons of CO2 emissions, every year. The virtualization giant has virtualized more than 6 million server workloads since 1998, resulting in an estimated energy savings of nearly 39 Billion kWh, or roughly $4.4 billion. This is roughly equivalent to the total energy consumption of Denmark for one year.

VMware reduces power and related costs by increasing server utilization rates and with power management capabilities that can power down servers when not in use. By powering down idle servers and desktops during inactive times, consumers can reduce power consumption by about 25%, according to VMware.

UK-based Sheffield Hallam University, for instance, implemented VMware Infrastructure 3 to reduce power and cooling requirements in the datacenter. Using VMware, they created 170 virtual machines, virtualizing over half of their servers. The virtual infrastructure at Sheffield Hallam runs 170 virtual machines at only 60,500 kilowatt-hours (kWh); this compares to the power required to run 170 physical machines, estimated at 686,000 kWh per year. In all, the company cut 269 tons of CO2 and saved £43,000 ($85,006.62 USD) on their annual power bills.

More information on how other companies have gone green and reduced costs using VMware virtualization is available on VMware’s website.

Virtualization expert Barb Goldworm has been banging the desktop virtualization and application virtualization drum for a while now. She contends that there is still a lot of innovation to be had in this space and that businesses are poised to reap the benefits, despite pushback from execs. It’s a symbiotic relationship, however, between IT departments and virtualization vendor. In other words, an opportunity for a business to virtualize some aspect of their infrastructure translates to an opportunity for a vendor to sell that virtualization product.

But how do you choose which vendor will fulfill that need? Chances are if you’re on this site, you’re either deeply invested in VMware, or about to become deeply invested, and will probably choose VMware Virtual Desktop Infrastructure (VDI) for your desktop virtualization needs. For the time being, that’s probably a smart decision for you. It’s also a good thing for VMware, which is positioned to recognize Goldworm’s assessment and to take advantage of the opportunity in the virtual desktop space.

In a recent interview with SearchVMware.com associate editor Hannah Drake, VMware expert and author David Rule agreed. “Virtual desktops is where VMware is going to see a huge surge of revenue. Take a company that has 1,000 servers, that’s 3,000 – 4,000 users. Ten percent of number is a huge amount of net new [virtual desktop] licenses,” he said. Furthermore, Rule believes that investment to be sound because they’re already ahead of the game. In his view, VMware will remain the virtualization technology market leader. “VMware is the innovator in the marketplace. Citrix and Microsoft haven’t brought anything new to the market that VMware hasn’t.”

He’s right, at least in comparison with Microsoft desktop offerings. Virtual PC is more of an emulator than an enterprise virtual infrastructure for distributed computing, and it might take a while to see how Microsoft’s acquisition of Kidaro pans out. In the meantime, VMware is going to have to look out for Citrix’s offering, XenDesktop, according to virtualization guru Chris Wolf. As the major player in the thin-client space, Citrix has the shortest to go in order to compete with VMware VDI and ACE. In fact, Wolf noted during a recent interview with Drake that at last week’s Microsoft Management Summit, when customers asked Microsoft about desktop virtualization, Microsoft reps said that they [don’t] have any current offerings but “strongly recommended Citrix XenDesktop” to their customers.

But Microsoft recommends XenDesktop for good reason. “Citrix has put a lot of work into Xen virtual desktop, and they don’t try to compete with Microsoft. They’ve been working towards interoperability from day one. I can run a VM on Hyper-V, power it down and run the VM on XenServer. I might have to change some device drivers, but I know it’ll run fine for me,” he said, adding that VMware has a lot to learn in terms of backend architecture. “From a scalability perspective, the XenDesktop architecture is a better architecture on the backend. If you look at the VMware line moving forward there will be substantial enhancements for virtual desktops, especially with storage and sharing virtual images.”

But what about price point? It turns out that the lackluster economy may turn out to be the wild card forcing change in the desktop and overall virtualization market. According to Wolf, virtualization price will play a bigger role in determining market leadership than some experts have predicted:

“For your average organization, it’s not just about feature for feature comparison, which VMware is always going to win. It’s about what is good enough, and what can I do with my IT budget, especially in [what is] arguably a recession right now. Price becomes a major consideration in purchasing decision. If I can get by with a product, then that’s going to be a compelling reason to look at that product. For VMware to maintain their market dominance, they have to lower their prices. There’s just no other way around it right now.”

Will VMware drop its prices this year? Only time will tell. We’ll continue to follow the virtualization price war; stay tuned.

For those of you who have come to use VMware Converter as part of your virtualization migration you probably had at least one conversion that failed for reasons you may not be able to explain. Since the release of VMware Converter 3.02 Update 1 in December of 2007 and in conjunction with the release of VMware ESX 3.5, there have been some practice issues that can address some of the failures that can occur. One issue in particular that plagued me frequently was using Converter 3.01 to perform conversions with an ESX 3.2 as a destination system. The conversions would regularly fail, with the only relevant message in the vmware-converter-x.log file is an entry stating that the image failed with a cryptic “vmodl.fault.InvalidRequest” entry.

Most of these issues are resolved by exclusively using the 3.02 Update 1 version of Converter, however there can be some other issues that prohibit the simple upgrade from correcting the failures. For example, if you had attempted a conversion on a Windows system on a version prior, you may receive the following message about the status of the agent:

In this scenario, you may need to manually remove the agent files. This is one of the two options when installing the agent during the initial steps of the conversion. The other option is to have Converter automatically remove the agent when the import succeeds. If the import fails because of the versioning issue between Converter and ESX, the agent will remain on the Windows client. To remove the Converter agent manually, go to the Windows Control Panel and select Add/Remove Programs and select the VMware Converter agent. For some systems using prior versions of Converter, additional steps may be required to remove files if a prior conversion’s remove operation did not complete successfully. This is mentioned in the known issues section of the release notes for Converter version 3.02 Update 1, and similar entries are in the release notes of prior versions as well. The manual removal process of the Converter agent involves removing files and performing a registry modification for the installed service.

VMware Converter is a strong part of the virtualization project aresenal, and keeping sharp on the issues for failed conversions and maintaining product version currency can help address many issues.

The main interview covered security, management tools and best practices for implementation, but there was some other strong material about VMware 3i, Microsoft Hyper-V’s potential affect on VMware adoption and how Citrix plans to gain a better foothold in the marketplace with XenSource, so we decided to include them in this supplementary blog post.

SearchVMware.co: Do you think VMware 3i is VMware’s answer to the problems with the service console (patches, security vulnerabilities, etc.)?

Rule: Yes, and I think 3i is a great product for a variety of reasons. It’s a simpler, lighter footprint on hardware, which in turn gives better reliability and performance.

Do you think Hyper-V will detract from VMware’s customer base?

Rule: I don’t think there’s a compelling enough reason to move from one platform to another. From our internal lab testing, as well as customer feedback, I can say there’s more interest than there used to be in Microsoft, but the main issue is supportability. For at least the next 12-18 months, VMware will remain the market leader. 18 months out we may see more competition. But even looking at the long term, VMware is in a pretty good position because of their current stronghold in the market.

On the other hand, I have seen a few customers here within the last one to two quarters that don’t have any virtualization in place. Microsoft does come up more in conversations with new adopters than it used to.

Given the low licensing cost of Hyper-V, do you think VMware will have to alter its pricing?

Rule: At least with our customers, I don’t see a lot of push back on VMware licensing just because of the amount of consolidation you can get. At a 10:1 consolidation, you’re saving the customer so much it becomes a non-issue. VMware may have to focus on the business approach, and focus on, say, in a 1,000 server data center, here’s how much we can help you save. It’s really going to be about VMware and VMware-partner marketing.

Do you think Citrix XenServer will become a bigger contender?

Rule: A lot of it hinges on Citrix bringing their solutions together. Xen had good momentum behind it in the marketplace. People that didn’t like the VMware licensing, such as smaller size SMBs or those who used virtualization for test and development, liked XenSource. So, Citrix has brand recognition. If Citrix can meld their application and desktop virtualization products together, that’s going to be their strength. And, actually, that’s the direction they want to see things go.

Citrix has two focuses: people that aren’t doing virtualization at all yet, and their core customer base from Presentation Server. They’re taking on more of a grassroots campaign. It would be a tough sell with accounts that don’t currently have a Citrix install base.

How do you think VMware will remain competitive?

Rule: They’re going for continual add-on business, net-new accounts and the licenses they’ll bring in. You’ll see movement with virtual desktops and Site Recovery Manager. But virtual desktops are where VMware is going to see a huge surge of revenue coming in. If you have 1,000 servers, that’s 3,000 – 4,000 users. Just 10% of that is a huge amount of net-new licenses.

VMware resource website l o s t c r e a t i o n s has just announced the immediate availability of the VI Toolkit for .NET. From the projects webpage:

The VI Toolkit for .NET is a .NET library that is patterned after the VI Perl Toolkit. It makes connecting to VI, finding a VM, and getting its properties a snap! The toolkit is written in C# 2.0, but can be used by any language capable of loading .NET 2.0 assemblies. One of the reasons that the VI Toolkit for .NET is so powerful is because it is the first .NET application to make use of the recent .NET port of the Apache Commons CLI library for parsing command line options and their arguments. We know it is the first, because we ported CLI! Administrators and developers who like the ability to quickly create scripts with the VI Perl Toolkit but appreciate the depth of .NET will love the VI Toolkit for .NET.

The built-in roles for access to VirtualCenter and the managed objects are okay for many common scenarios, but some situations require additional configuration. In some cases, creating custom roles has been a viable solution. Here is how I created a role that would allow a user only to view a virtual machine’s console.
Create the role and deployment model first

Before permissions are assigned, some thought should be given to user rights and how they would be administered. In most VirtualCenter environments, the permissions would be retrieved from a Windows Active Directory domain. To make that process easier, all permissions should be assigned to VirtualCenter through Windows groups.

To create a role or modify an existing role, select the Administration button within the VMware Infrastructure Client (VI Client). From here, you can create or modify a role for your desired access. If you wanted to allow a user to view a virtual machine’s console, for example, you would create a custom role such as the specified permission below:

Applying the custom roles

The ability to view the console of specified systems can come in handy for certain situations, particularly when traditional network connectivity to the guest operating system is not possible for normal methods such as remote desktop or VNC. Creating a console view-only role would be done in the VIC on a per-object basis. A per-object basis is one in which you can assign the permission to view just the console to an ESX host, a resource pool, a data center or even a virtual machine individually. Roles to objects in VirtualCenter are always applied via the Permissions tab for the object.

To make configuration consistent, create a Windows security group with the same name in the Active Directory domain. For my custom role, the Windows group (MSS\VMSpecified-Roles-ConsoleOnly) and the VirtualCenter role (VMSpecified-Roles-ConsoleOnly) are assigned to the object below:

From this point, the clients can log into a locally install the VIC and connect per the specified permissions. Be careful, however, as a username that may have multiple roles would have the permissions of the combined roles. You can work with some propogation, but singular assignment would be a better practice. VMware provides a document fully outlining the roles architecture available for download from their website.
Audit trail of connections into the VirtualCenter

With this functionality, an auditing requirement is fully justified. Within the VirtualCenter database, you can monitor the authentication log-on and log-off events. A January SearchVMware.com ITKE post has this outlined well.

Santa Ana, Calif.-based First American Corp., a Fortune 500 company that provides business information and analytics, is standardizing its data center with virtualization software from VMware, Inc. to improve power efficiency and better server utilization rates.

First American’s Enterprise Architect, Jake Seitz, said the increasing cost of power was the most significant driver for virtualization.

“In Q407 we started looking at our utility costs and our data center footprint, and realized that embracing virtualization would help us maximize what we have and shrink our utility footprint,” Seitz said. “My hope is that we can get to 80% virtualzed environment – we are at about 30% right now.”

Low hardware utilization rates also prompted the company to invest in VMware virtualization. “We wanted to better utilize our existing hardware. We had some hardware utilization as low as 4%, and rarely would be see anything above 15% utilization,” Seitz said.

When Seitz began looking at virualization about two years ago, he also deployed small Microsoft Virtual Server and Citrix XenServer environments – both less expensive than VMware – but decided to standardize on VMware going forward because he liked the management tools and support.

When the project started, the data center at First American was equipped with 2,800 physical servers running that many OSes. After virtualizing 700 of those servers, First American runs 3,500 OSes, Seitz said.

The OSes are mostly Microsoft Windows, though there are some Unix systems on mainframes and in Xen containers, he said. The physical servers supporting the virtual machines are x86 servers rom Hewlett Packard and Dell, and many are equipped with Intel Xeon quad-core processors, Seitz said.

First American hasn’t gotten rid of many physical servers because they lease them, but it will save power in the long run because with better server utilization, they won’t have to add more physical servers in the future, Seitz said.

Also, because their hardware is leased, it is refreshed every three years. When this happens, the IT staff has to port all of the applications and OSes from the old hardware to the new hardware. Normally, this takes months to accomplish, but it should be a lot easier using virtualization, Seitz said.

“Now, instead of doing a one-to-one refresh, we will virtualize our environment and it will shave off a few months in man hours,” Seitz said.

The company is starting to virtualize their mission critical applications now, and have even adopted a “virtual machine first” policy.

“You need a really good reason to get a physical server,” Sietz said. “Everything we do now is virtual.”

– The VMware guide to plug-ins is official. Mine is not. Although the VMware guide is experimental, it is more likely to be supported by VMware than any plug-in you write using my guide. That said, I have heard that the team responsible for plug-ins plans on developing a shim that will continue to support my plug-in methods in the next version of VMware Infrastructure (VI). That is hearsay, however, and it could change.

– Official plug-ins function very similarly to mine (they should, they are built using the same principals), but they could be considered inferior in one very important manner. Although the user interface to activate a plug-in is the same (context menus, tabs, menu items), the interface for official plug-ins can only be a web page. For instance, a user right-clicks on a virtual machine and clicks on the context-menu item labeled Migrate storage which launches my Storage VMotion plug-in rewritten as an official plug-in. Instead of having a Windows form appear that maintains a consistent user interface, instead a web browser appears and runs a script or web application that has authentication information and object information passed to it from the VI client (much like my Invoke plug-in).

– Official plug-ins must reside on a web server to which clients have access. This can be considered a good thing and a bad thing. On one hand, it centralizes plug-in updates. On the other hand, if the web server is offline ESX admins could lose plug-in functionality that they have come to rely on. Plug-ins written using my guide reside on the local client; always online and accessible.

– Given how hard it is for developers to build consistent web applications between browsers (even the VI online web interface fails to work properly on Safari), it is a tall order to expect plug-in developers to create plug-ins that look the same between the four major web browsers available to Windows (IE, Firefox, Safari and Opera). That said, you could just force users to use IE since you know it will be there.

– One big win with the official method is that the plug-ins could be written to be standalone web applications. This means that they could be accessed outside the context of the client.

The official plugin guide is a great achievement on the part of VMware; it shows their commitment to giving users what they want. Ultimately, though, the plug-ins created with it are forced to be web applications or scripts hosted on a web server. The biggest problem with this is that it forces users to leave the consistent look and feel of the VI client, ripping them out of their experience. The official plug-in guide is probably best thought of as an online alternative to the Invoke plug-in, but not as a replacement to the plug-in architecture that I have already exposed.

Hopefully with VI4 we will see a more fully fleshed out, official plug-in architecture.

Update 4/22/08:

Carter Shanklin, Product Manager of End User Enablement at VMware, sent me an email with the following note:

One point you make is the inconsistency of browsers in rendering, etc. Inside VI Client, if you create a custom tab it will be rendered using an Explorer control, regardless of what your preferred browser might be. This may be problematic if you want to use the same interface from outside of VI Client, but from within the client it should look fairly consistent.

Carter is absolutely correct. These controls should be rendered within the VI client, however you still need to make sure your web application looks and feels like a part of the VI client. Just because your web application is displayed from within the VI client, it does not mean that the VI client is rendering your list box or text box with its own style sheets.

Creating virtual machines (VMs) is quite easy, but how you remove them can make a big difference in storage space.

When removing a virtual machine, one popular method is to move the virtual machine to a resource pool with no assigned memory or processor resources. There are some concerns and strategies for doing this, however there are no virtual appliances or fully automated solutions within the VMware Infrastructure Client to help you with this process.

While moving the VM to a different resource pool is a convenient way to roll the system back online if necessary, the VM would still have its storage requirements. From an ongoing management perspective, storage is the biggest concern. I have found that using the local storage for the tentative decommission stage is acceptable in lieu of using the shared storage configurations across the other ESX hosts.

Once the virtual machines are in the resource pool with no assigned resources, you can then remove them on an interval that suits the ‘just in case’ situations. This can be a daily, weekly or monthly timeframe — whatever suits your environment and business needs. Of course, if the storage situation inhibits keeping these systems for long amounts of time, then the interval may be dynamic.

What are you doing from a matter of practice standpoint for the deletion of your virtual machines? Share your comments and ideas below.

About This Blog

The Virtualization Pro blog offers VMware-focused commentary and administrative tips from several respected experts with VCP certification and vExpert recognition. Also check out SearchVMware.com for in-depth how-to articles, screencasts and more.