Friday, July 14, 2017

You attempt to expand a hard disk of a vSphere Replication replicated virtual machine but immediately receive the following error:

Reconfigure virtual machine

Invalid or unsupported virtual machine configuration.

See the error stack for details on the cause of this problem.

vSphere Replication does not support changing the length of a replicated disk.

Solution

The reason why the error is thrown is because vSphere Replication prevents sizing changes to the protected copy of the virtual machine (source) when it is replicated to a recovery copy of the virtual machine (target). This makes sense because the replication engine likely tracks the changes in a certain way where changing the disk size would cause issues. VMware has released the following two KBs to explaining the steps required to expand a replicated virtual machine’s hard disk:

Begin by documenting replicated virtual machine's vSphere Replication configuration because you will need this information when specifying the Target Location in one of the later steps (screenshots usually suffice assuming the path fits in the text field):

Ensure that you can read the full path of the Target Location:

The same applies to every Hard diskTarget Location field:

Document the Quiescing method configuration:

Document the Recovery Settings and I usually select Cancel to avoid unintentionally making any changes:

Step #2 - Rename the Replicated Virtual Machine's Datastore Folder(s)

Proceed to browsing the Target Location datastore of where the replicated virtual machine is stored. Note that this is the replicated copy and NOT the live copy:

One of the reasons why it is important to document the Target Location of the replicated copy is because the VMDK files are not always stored in the same directory as the VMX files as shown in this example:

Proceed to rename the replicated copy's folder as such. Remember that this is the replicated copy and NOT the live copy:

3. Stop replication for the virtual machine at the protected site using the vSphere Replication User Interface (UI).

So with this in mind, proceed to right-clicking on the replicate virtual machine in the vSphere Replication console and select Stop:

Note that it is important to you have renamed the Target Location virtual machine folders. From what I've seen, if the replicated VM was seeded then the target files would not be deleted but if the replicated VM was not seeded then the files would get deleted.

You should now see tasks executed under the Recent Tasks pane indicating replication is being disabled for the virtual machine:

The virtual machine should no longer be displayed once the operation completes.

Step #4 - Expand Source/Live Virtual Machine's VMDK

With replication stopped, you should now be able to expand the source/live virtual machine's VMDKs so proceed to expanding them to the size required.

Step #5 - Expand Target/Replicated Virtual Machine's VMDK

Since the target/replicated virtual machine is not inventoried on a host, expanding the drives will need to be done with the vmkfstools command. Proceed by either accessing the console or SSH to a host that has access to the datastore and navigate to the directory of the renamed folders containing the replicated virtual machine files.

**The ls -lah command can be used to display the files in the command line.

Once in the directory containing the files, proceed to increase the hard drive VMDK file with the command:

vmkfstools -x XXXGB <filenameOfVMDK>

A similar output below will be displayed upon successfully increasing the VMDK:

Refreshing the datastore browser will show the new size of the VMDK:

Continue by renaming the folder back to the original name:

Proceed to reconfigure replication for the virtual machine:

The default Target location that is used will most likely be different than the folder with the replicated VMDKs so use the previously documented configuration to select the same Target location as the original location:

Once the previous Target location has been configured, the amount of hard disks of the replicated virtual machine will now be displayed (there are 4 in this case):

Configure all of the hard disks to use the same folder as the previous location:

Selecting the folder with the existing replicated VMDK will display the following message:

Replication Seed Confirmation

Duplicate file found. Do you want to use this file as a seed?

Select Yes when receiving this prompt:

Continue and repeat the same procedure for the rest of the disks and the same Replication Seed Confirmation prompt should be displayed:

Configure the Recovery settings as previously documented and complete the configuration by clicking Finish::

The replicated virtual machine should now be displayed again with an Initial Full SyncStatus:

Clicking on the i icon in the GUI would provide information:

To obtain more information on the status of the synchronization, log onto the esxi host with the protected VM is inventoried and execute:

vim-cmd vmsvc/getallvms

... to list all the VMs along with their Vmid:

Note the Vmid and execute the command:

vim-cmd hbrsvc/vmreplica.getState <Vmid>

This will display an output similar to the following:

I usually periodically execute the vim-cmd hbrsvc/vmreplica.getState command to check the progress as it provides more information:

The time it takes for the process to complete will vary depending on the size of the virtual machine but note that only changes are replicated over and not the full virtual machine. The following are some screenshots taken during the synchronization:

After the required time, the virtual machine should get back to an OK Status as such:

Hope this would help anyone looking for what the full process of expanding a replicated virtual machine’s hard disk would look like.

Monday, July 3, 2017

You’re in the process of migrating mailboxes from Exchange 2010 to Exchange 2016 with both a live as well as an archive mailbox and while some mailboxes successfully move to the new databases, you noticed that others remains in Syncing status indefinitely:

Expanding the More Details… option show the synchronization has stalled for quite some time:

Attempting to log onto the Exchange 2016 server and adjusting the parameters MaxActiveJobsPerSourceMailbox and MaxActiveJobsPerTargetMailbox in the configuration file MSExchangeMailboxReplication.exe.config located in the directory E:\Program Files\Microsoft\Exchange Server\V15\Bin does not correct the issue:

Solution

Attempting to search for the error messages:

StalledDueToTarget_MailboxCapacityExceeded:

… and:

StalledDueToTarget_Processor

… did not return any helpful posts and what ended up being the problem was the archive mailbox server we were using to move the archive mailboxes to. The server’s CPU utilization wasn’t particularly high (2%), memory usage was average (90%) but the server uptime was 82 days and there were pending Windows patches asking for a reboot. Previous mailboxes that were stalled would successfully completed after the server restart:

Hope this helps anyone who may encounter this issue and unable to find any useful information on the internet.