This issue isn’t specific to Jetstress, Exchange, Microsoft, or a specific fabric type, storage protocol or storage vendor. Exceeding the virtual disk capacities listed above, per host, results in the symptoms discussed earlier and memory allocation errors. In fact, if you take a look at the KB article, there’s quite a laundry list of possible symptoms depending on what task is being attempted:

An ESXi/ESX 3.5/4.0 host has more that 4 terabytes (TB) of virtual disks (.vmdk files) open.

After virtual machines are migrated by vSphere HA from one host to another due to a host failover, the virtual machines fail to power on with the error:vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.

Adding a VMDK to a virtual machine running on an ESXi/ESX host where heap VMFS-3 is maxed out fails.

When you try to manually power on a migrated virtual machine, you may see the error:The VM failed to resume on the destination during early power on.
Reason: 0 (Cannot allocate memory).
Cannot open the disk ‘<<Location of the .vmdk>>’ or one of the snapshot disks it depends on.

The virtual machine fails to power on and you see an error in the vSphere client:An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)

A similar error may appear if you try to migrate or Storage vMotion a virtual machine to a destination ESXi/ESX host on which heap VMFS-3 is maxed out.

Cloning a virtual machine using the vmkfstools -icommand fails and you see the error:Clone: 43% done. Failed to clone disk: Cannot allocate memory (786441)

While VMware continues to raise the scale and performance bar for it’s vCloud Suite, this virtual disk and heap size limitation becomes a limiting constraint for monster VMs or vApps. Fortunately, there’s a fairly painless resolution (at least up until a certain point): Increase the Heap Size beyond its default value on each host in the cluster and reboot each host. The advanced host setting to configure is VMFS3.MaxHeapSizeMB.

Let’s take another look at the default heap size and with the addition of its maximum allowable heap size value:

After increasing the heap size and performing a reboot, the ESX(i) kernel will consume additional memory overhead equal to the amount of heap size increase in MB. For example, on vSphere 5, the increase of heap size from 80MB to 256MB will consume an extra 176MB of base memory which cannot be shared with virtual machines or other processes running on the host.

Readers may have also noticed an overall decrease in the amount of open virtual disk capacity per host supported in newer generations of vSphere. While I’m not overly concerned at the moment, I’d bet someone out there has a corner case requiring greater than 25TB or even 32TB of powered on virtual disk per host. With two of VMware’s core value propositions being innovation and scalability, I would tip-toe lightly around the phrase “corner case” – it shouldn’t be used as an excuse for its gaps while VMware pushes for 100% data virtualization and vCloud adoption. Short term, the answer may be RDMs. Longer term: vVOLS.

Updated 9/14/12: There are some questions in the comments section about what types of stoarge the heap size constraint applies to. VMware has confirmed that heap size and max virtual disk capacity per host applies to VMFS only. The heap size constraint does not apply to RDMs nor does it apply to NFS datastores.

Updated 4/30/13: VMware has released vSphere 5.1 Update 1 and as Cormac has pointed out here, heap issue resolution has been baked into this release as follows:

VMFS heap can grow up to a maximum of 640MB compared to 256MB in earlier release. This is identical to the way that VMFS heap size can grow up to 640MB in a recent patch release (patch 5) for vSphere 5.0. See this earlier post.

Maximum heap size for VMFS in vSphere 5.1U1 is set to 640MB by default for new installations. For upgrades, it may retain the values set before upgrade. In such cases, please set the values manually.

There is also a new heap configuration “VMFS3.MinHeapSizeMB” which allows administrators to reserve the memory required for the VMFS heap during boot time. Note that “VMFS3.MinHeapSizeMB” cannot be set more than 255MB, but if additional heap is required it can grow up to 640MB. It alleviates the heap consumption issue seen in previous versions, allowing the ~ 60TB of open storage on VMFS-5 volumes per host to be accessed.

When reached for comment, Monster VM was quoted as saying “I’m happy about these changes and look forward to a larger population of Monster VMs like myself.”

Shortly before VMware Partner Exchange, Microsoft Certificate Services in my long-in-the-tooth lab domain died and unfortunately took a handful of integrated services with it. One of those dependent services was Microsoft Exchange 2010. After attempts to repair and later rebuild and restore the Microsoft CA failed, I decided the time had finally come to ditch the boche.mcse Windows Active Directory domain which I originally built when Windows 2000 and Active Directory first launched. There were a number of other lingering issues with the domain which I haven’t been able to repair and the time investment and frustration really wasn’t worth it. I had been waiting for a catalyst in addition to a block of weekend time and today was the day.

I had been hosting Microsoft Exchange Server since version 5.5 and although I had accumulated quite a bit of experience over the years in building, migrating, and maintaining a small Exchange environment, the truth is Exchange has gotten way too complex and bloated for me to keep up with. Being a VMware guy, I’ve had the itch to give Zimbra a try since the acquisition. It seemed like a win-win solution: Ditch the complexities of Exchange and replace with a VMware messaging solution. So after building the new domain this afternoon Exchange 2010 was retired and Zimbra 6.0.13.406 was born.

Zimbra is incredibly easy to set up, especially when deploying the appliance version in .OVF format hosted on vSphere 5. I was up and running with incoming and outgoing email in very little time. One area I was concerned with when replacing Exchange with Zimbra was the integration and end user experience. Zimbra ships with an HTML Outlook Web Access like interface which can be used for processing mail on a regular basis. Would I lose the ability to use Outlook as a mail client or mobile device compatibility? The answer was no on both accounts. In addition to the web interface, I can continue to use Outlook 2010 connected to the Zimbra server via IMAP. This is just about as easy to set up as attaching Outlook to a native Exchange environment. Outlook detects the mail services provided by Zimbra and negotiates an IMAP configuration. Done.

Configuring Zimbra Mobile wasn’t as straight forward but I was able to piece together the steps. Zimbra’s documentation is decent but in some areas it feels explicitly vague, lacking step by step configuration detail. With Outlook and the web client functioning well, the last step was to integrate Zimbra with my mobile devices: the iPhone and iPad. These are the steps I followed:

Log into the Zimbra Collaboration Suite Appliance web console (https://<f.q.d.n.>:5480/). On the Zimbra Administration tab, change the pull down box from All addresses or Accounts to Profiles.

In a default out-of-box deployment, there should be just one profile named ‘default’. Modify that profile by enabling Mobile Sync on the Zimbra Mobile tab. Save the changes to the profile.

Now that we have Mobile Sync enabled in a profile, we can apply that profile to each Zimbra user who needs messaging access from their mobile handheld. Going back to the Dashboard tab, change the pull down box from Profiles to Accounts. Modify the account in which you’d like to grant Zimbra Mobile access. On the General Information tab for the user account, uncheck the box labeled ‘auto‘ and then begin typing the first few letters of the profile name of ‘default‘ in the Profile box. After the first few letters of the profile are typed, the default profile name will appear and that can be selected with the mouse. Now save the changes.

Now it’s time to configure the iPad and iPhone to connect to Zimbra Mobile. The first step is to create a new mail account. The type will be Exchange.

I found configuring the account with Zimbra Mobile easier than with Exchange 2010. Provide the following values:

2. Email – this should match the email address tied to the account in Zimbra.

3. Server – this will be the f.q.d.n of the Zimbra server. By default an SSL connection over port 443 will be attempted which Zimbra supports. You can optionally specify an alternative port. For example, I could come in externally over port 8080 and then have that translated by NAT to port 443 and sent to the Zimbra server. Without a port defined, 443 is implied.

Domain – not required with internal Zimbra authentication. If authenticating with LDAP integration, this field would likely need to be populated.

4. Username – the name tied to the account in Zimbra.

5. Password – 12345

Description – not used in Zimbra Mobile configuration. It’s merely a differentiator for the account on a mobile device which may have multiple accounts set up.

If things are working correctly, you’ll be able to tap the blue Next button and you’ll be prompted to verify the synchronization of all three types of items: Mail, Contacts, and Calendars. Once the account configuration is saved you’ll also see an additional slider showing that SSL is ON (the default).

Next up is to configure my wife’s Kindle Fire. I don’t expect any issues there.

Last weekend I successfully upgraded, ahem, migrated the lab infrastructure from Microsoft Exchange 2003 to Exchange 2010. This upgrade has been on my agenda for quite some time but I had been delaying it mainly due to lack of time and thorough knowledge of the steps. I had a purchased the Microsoft Exchange Server 2010 Administrator’s Pocket Consultant (ISBN: 978-0-7356-2712-3) in January and marked up a few pages with a highlighter. However, the deeper I got in the book, the more daunting the task seemed to have become, even for a simple one-server environment like mine. In my mind, Exchange has always been somewhat of a beast, with increasing levels of difficulty as new editions emerged. The pocket consultant series of books are wonderfully technical, but they haven’t been able to fit in my pocket for about a decade. They contain so much content that it has become difficult to rely on them as a CliffsNotes guide for platform upgrades, especially when it comes to Exchange.

Then two things happened miraculously at the same time. First, I was invited to a private beta test of a virtualization related iPad application. As part of this test, I needed to be able to send email from my iPad. I had been unsuccessful thus far in getting Microsoft Exchange ActiveSync to work with the iPad (even after following Stephen Foskett’s steps) and could only assume that it was due to several years of wear and tear on my Exchange 2003 Server. I needed to get that upgrade to Exchange 2010 done quickly. Second, the May 2010 issue of Windows IT Pro magazine showed up in my mailbox. To my delight, it was chock full of Exchange 2010 goodness, including a cover story of “Exchange 2003 to Exchange 2010 Step-by-Step Exchange Migration”. I’m pretty sure this was divine intervention with the message being “Get it done this weekend, you can do this.”

The upgrade article by Michael B. Smith started on page 26 and was 100% in scope. The focus was a single server Exchange environment upgrade from 2003 to 2010. I read the seven page artile in its entirety, marking up key “to-do” steps with a highlighter. Following are some things I learned along the way:

Naturally the Exchange server is virtualized on VMware vSphere.

My Exchange environment is built upon a foundation that dates back as far as Exchange 5.5 (pre-Active Directory). There would be no in place upgrades. Exchange hasn’t provided an upgrade since Exchange 2003. That suited me just fine as the Exchange 2003 server has been through so much neglect, although it had gotten pretty slow, it’s a miracle it was still functional. The Exchange migration will consist of bringing up a fresh OS with a new installation of Exchange, and then migrating the mailboxes and services, and then retiring the old Exchange Server. Microsoft calls this a migration rather than an upgrade.

Exchange must be running in Native mode. Not a problem, I was already there.

Exchange 2010 can be installed on Windows Server 2008 or Windows Server 2008 R2. I naturally opted for R2. No sense in deploying a two-year old OS when a more current one exists and is supported. Plus, I personally need more exposure to 2008 and R2… 2003 is getting long in the tooth.

Copy the Exchange DVD to a data/utility drive on the server. Reason being, you can drop the most recent rollup available into the \Updates\ folder and basically perform a slipstream installation of Exchange with the most recent rollup applied out of the gate. As of this writing, the most current is Rollup 3.

Here’s a big time saver, install the server roles and features Exchange 2010 requires using the provided script on the DVD:\scripts\ServerManagerCmd -ip Exchange-Typical.xml -restartOther sample pre-requisite installer scripts can be found here.

The 2007 Office System Converter: Microsoft Filter Pack (x64) is required to be installed. This is downloadable from Microsoft’s website. A little strange, but I’ll play along. It’s required for the Exchange full-text search engine to search Office format documents.

Run the following commands for good measure. It may or may not be required depending on what’s been done to the server so far:sc config NetTcpPortSharing start= auto
net start NetTcpPortSharing

Setup logs for Exchange are found in C:\ExchangeSetupLogs\ The main one is ExchangeSetup.log. Hopefully you won’t have to rely on these logs and you are blessed with a trouble-free installation.

There are the usual Active Directory preparatory steps to expand the Schema which seem to have increased in quantity but I could be hallucinating:

setup.com /PrepareLegacyExchangePermissions

setup.com /PrepareSchema

setup.com /PrepareAD

setup.com /PrepareAllDomains

Installation can be invoked by CLI with setup.com /mode:install /roles:ca,ht,mb however, I chose a GUI installation which was more intuitive for me.

The article stated the installation would take at least 20 minutes on fast hardware. My installation took less than 15 minutes on a VM hosted by four year old servers attached to fibre channel EMC Celerra storage – bitchin.

A Send connector is required before Exchange 2010 will route mailto the internet.

Exchange 2010 ships with two Receive connectors but they must be configured before they will accept anonymous email from the internet.

Exchange 2010 is managed by the Exchange Management Console which is called the EMC for short. That will be easy to remember.

Exchange 2010 is also managed by PowerShell scripts (also called an Exchange Management Shell, or EMS for short). There are some configuration tasks which can only be made via PowerShell script and not via the EMC.

Lend your end users and Helpdesk staff a hand by creating a meta-refresh document in C:\inetpub\wwwroot\ which points to https://<mail_server_fqdn>/owa effectively teleporting them into Outlook Web App (did you catch the name change? no more Outlook Web Access)

Mailboxes are no longer moved online due to their potential size and problems which may occur if a mailbox is accessed during migration. Mailbox migrations are now handled via EMC by way of a Move Request (either local [same org] or remote[different org]). When a move request is submitted, the process begins immediately but may take some time to complete obviously based on the size of the mailbox as well as the quantity of mailboxes multiple selected for the move request. Tony Redmond wrote a decent article on how this is done. Scheduled move requests can be instantiated via PowerShell script.

One of the final steps of a successful migration is properly decommissioning the old Exchange 2003 environment. This is where things got a little hairy, and I half wasn’t surprised. Upon attempting to uninstall Exchange 2003 to properly remove its tentacles from Active Directory and the Exchange organization, I was greeted by two errors in the following message:
In the legacy Exchange 2003 System Manager, there are two Recipient Update policies which exist. Going from memory, one was for the domain which I was able to remove easily, and one was an Enterprise policy which cannot be removed via the System Manager. Follow the instructions near the end of this article for the procedure to modify Active Directory with adsiedit.
The second error message deals with removal of the legacy Routing Group Connector. There were actually two which needed to be removed. The only way to remove the Routing Group Connector is via PowerShell and it is also described towards the end of this article.

After addressing the issues above, the uninstaller ran briefly and then failed for an unknown reason. Upon attempting to re-run the uninstall, I noticed the ability uninstall Exchange 2003 via Add/Remove Programs in the Control Panel had disappeared, as if it was successfully uninstalled. Clearly it was not as the Exchange services still existed, were running, and I could launch System Manager and manage the organization.

ActiveSync doesn’t work out of the box on privileged administrator level accounts due to security reasons. If you accept the risk, this behavior can be changed by enabling the inheritance checkbox on the user account security property sheet.

I’m pretty happy with the results. The process took took quite a few steps but I am nonetheless pleased. Careful work following a very nicely outlined procedure by Michael B. Smith has yielded both a snappy-fast Exchange 2010 server on Windows Server 2008 R2 as well as ActiveSync integration with my iPad. Exchange 2010 is a beast. I can’t imagine tackling an Exchange project for anything larger than the smallest of environments. I’m not sure how I can have so many years experience managing my own small Exchange environment yet still lack the confidence in the technology. I guess it mostly runs itself and as I said earlier, it’s quite resilient meaning it doesn’t require much care and feeding from me. And thank God for that.

I often hear questions or concerns about virtualizing Exchange. E-Oasis found a new VMware white paper and provides a nice lead in explaining how VMware corporate took their physical servers and migrated to virtual, reducing aggregrate hardware usage.

One might ask why VMware’s Exchange servers were not virtualized before this, particularly when VMware was a smaller company with less mailboxes? Perhaps they decided earlier versions of Exchange were not virtualization candidates? Maybe limitations in earlier version of ESX made it less than attractive? I don’t know why but it would have been cooler to see VMware put their money where their mouth is earlier on. Perhaps someone from VMware can chime in on a comment here.

At any rate, it’s an absolutely beautiful white paper and I’m actually surprised at the level of detail some of the diagrams get into providing network host names and IP addresses for the infrastructure. I suppose they could be ficticious, but the names look rather authentic and not made up to me. Kudos.