VDDK 5.0 was tested with (and supports) the following operating systems to perform proxy backup:

Windows Server 2008 and Windows Server 2008 R2

Windows 7

Windows Server 2003 SP2

Windows XP SP3

CentOS 5.5

Red Hat Enterprise Linux (RHEL) 5.3, 5.4, and 5.5

SUSE Linux Enterprise Server (SLES) 10 and 11

Ubuntu 10.04

In this release, VDDK features for Linux have achieved virtual parity with Windows,
except for the vixMntapi library,
which still does not support advanced transports for Linux virtual machines.

VMware Consolidated Backup (VCB) has a writeup
(knowledge base article)
showing the support matrix for storage multipathing.
VMware does not provide a similar support matrix for VDDK.
Customers should seek this information from their backup software vendors.

These release notes provide information about issues identified or resolved since VDDK 1.2.1.
For information about VDDK including general information about the development kit,
how to obtain and install the software, redistributing it,
and issues that were identified or resolved in previous releases,
see the VDDK 1.2 Release Notes.

Compatibility Notices for Partners

ESX/ESXi 3.5 NBD/SSL.
When connecting through vCenter Server 5.0 to an ESX/ESXi 3.5 host,
VDDK 5.0 programs using nbd or nbdssl transport
cannot open VMDK files in read/write mode (access error).
The only known workaround is to connect directly to the ESX/ESXi 3.5 host.

GPT unsupported. Despite its appearance in the Virtual Disk API Programming Guide
as a volume type, GPT (GUID partition table) volumes are not supported by VixMntapi.

Licensing. In vSphere 5.0, the SCSI HotAdd feature
is enabled only for vSphere editions Enterprise and higher,
which have Hot Add licensing enabled.
No separate Hot Add license is available for purchase as an add-on.
In vSphere 4.1, Hot Add capability was also allowed in Advanced edition.
Therefore, customers with vSphere Essentials or Standard edition
who use backup products (including VMware Data Recovery)
are not able to perform proxy-based backup, which relies on SCSI HotAdd.
Those customers must use alternate transport modes.

Storage vMotion. To prevent orphaned virtual disk
caused by Storage vMotion during backup, VDDK 5.0 adds the
VixDiskLib_PrepareForAccess and
VixDiskLib_EndAccess functions,
which disable and re-enable the RelocateVM_Task method.
Storage vMotion can wait for the backup to finish, or relocate a different VM.
See the Virtual Disk API Programming Guide for details,
or the HTML reference pages below.

VSS pre-freeze and post-thaw scripts.
In ESX/ESXi versions before 3.5 U2, VMware Tools looked in the C:\Windows folder
for the pre-freeze-script.bat and post-thaw-scripts.bat
VSS quiescing scripts. In ESX/ESXi 3.5 U2 and later, VMware Tools first looks in
C:\ProgramFiles\VMware\VMware Tools\backupScripts.d
for scripts in alphabetic order, calling them all with freeze argument,
and afterwards in reverse alphabetic order, calling them with thaw argument
(or freezeFail if quiescing failed).
In ESXi 5.0, both locations are searched.
A good way to support both old and new ESXi versions is to place these lines
at the beginning and end of the C:\Windows pre-freeze and post-thaw scripts:

rem At beginning of script
if not "%1" == "" goto :exit
...
rem At end of script
:exit

VDDK 5.0 requires C++ library support,
which many older Linux versions do not provide in the base install.
When using an old Linux distribution as a backup proxy,
your customers might need to install C++ library packages from the distribution ISO.

The libvixDiskLib.so.5.0.0 shared library for Linux
is smaller than it was in earlier builds.

While all versions of vSphere allow the at-sign (@) in datastore names,
the VixDiskLib API is unable to open VMDK files inside such datastores.
You might want to modify your backup code to throw an error in such a case.
The customer workaround is to avoid datastore names with the at-sign (@)
when using VDDK.

If you want to perform silent install or uninstall of VDDK with InstallShield on Windows,
run one of the following commands. The /s option means silent, the /x option means uninstall,
and the /v option passes /qb (basic UI) or /qn (no UI) to MsiExec.

Recently Resolved Issues

The VDDK 5.0 release resolves the following issues:

HotAdd code refactored into a service thread.

In the process of refactoring the source code,
several problems in HotAdd were addressed.
The code now cleans up HotAdd files left behind after a crash,
such as virtual machines with a previous snapshot,
or files belonging to a snapshot that no longer exists.

File mount and full backups with HotAdd sometimes failed.

The HotAdd code now deals correctly with a connection sequence issue.
This issue occurred infrequently, but was the result of multiple backups in parallel.

Cleanup function deletes leftover directories after a crash.

VixDiskLib_Cleanup code now deletes directories left behind after a crash.

Logging added to indicate transport plug-in failure.

VixDiskLib logging was added to help diagnose initialization failure
of the transport plug-in.

New functions to coordinate with Storage vMotion.

Online HTML reference pages were added for the new functions,
VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess.
These two functions are recommended to prevent Storage vMotion
while vixDiskLib is accessing the disks associated with a virtual machine.

VDDK version number updated.

The VDDK version number was changed to 5.0 to be in sync with the VADP version number.

Robustness of vCenter Server connections improved.

The vixDiskLibPlugin, which connects to vCenter Server, was fixed
to deal with an initialization sequence problem that would occasionally cause a crash,
resulting in a “panic ASSERT” message.

SAN transport and the new VMFS version.

The SAN code was updated to recognize the new VMFS file system version, VMFS-5.

SAN transport uses direct access mode for Linux.

The SAN code for Linux now uses the O_DIRECT access mode, which prevents
some buffer-induced problems with data that got written but never committed.

Solution for write-protected SAN on Windows Server 2008 proxy.

When using SAN transport or HotAdd mode on a Windows Server 2008 proxy,
make sure to set the Windows SAN policy to onlineAll, otherwise
writes produce the “media is write protected” error message.

Write integrity improved for writes to thin provisioned disk.

The code for writing to thin disks has been fixed to correctly handle data
that overlaps the edge of a disk allocation unit area. In previous releases,
data past the allocation unit boundary would get dropped, and yet the write
would be declared OK (return success).

VDDK 5.0 works with VirtualCenter 2.5 and ESX/ESXi 3.5 now.

The communications code was updated to connect to ESXi 5.0,
while maintaining compatibility with older versions.
VDDK 5.0 Beta did not support VirtualCenter 2.5 and ESX/ESXi 3.5 hosts.

All transport modes support multithreading.

Tests were done to validate that all transport modes work with multiple threads.

Although vSphere 5 provides IP version 6 (IPv6) support,
VADP and VDDK have not been tested with IPv6 networking.

Can now simultaneously mount cloned disks on Windows.

When VixMntapi_MountVolume tried to simultaneously mount two disks with same
disk signature, the mount failed with error 12, VIX_E_FILE_ALREADY_EXISTS,
“cannot create a file when that file already exists.”
This occurred only on Windows when mounting disks from two VMs
if one was cloned from the other, or both were cloned from an original VM.
The issue was fixed by checking similarity of UUID and getting another if needed.

Password no longer error-logged in clear text.

When an VDDK error occurred, such as failure to connect or open a disk,
the error was logged along with the user name and password in cleartext.
This issue was fixed by sanitizing the password to read “xxx” instead.

Advanced transport modes available again for ESX 3.5 systems.

VDDK 1.2.1 and 5.0 again support advanced transports,
such as SAN mode, with VirtualCenter 2.5 and ESX/ESXi 3.5.
VDDK 1.2 had a regression where it did not support
advanced transport modes in VC 2.5 and ESX 3.5.

No more time-outs for NBD operations in ESXi.

VDDK 1.2 introduced a 3-minute timeout when opening disks
in nbd or nbdssl mode.
If network communication was interrupted so the proxy could not communicate with the host,
but the host could not detect a network communication failure,
disks were left open on the host,
and could not be closed without restarting management services.
This change caused issues with applications that opened the disks but did not
read or write them for more than three minutes.
The timeout has been eliminated. You can still set a custom timeout
by editing the VixDiskLib_InitEx() configuration file.

Most of the messages now pass through the Log function.

In previous VDDK releases, some error messages were sent only to the console
(stderr) instead of going through the Log function.
So the log file did not record all messages, missing those sent to stderr only.
The default Log function sends messages to the console and writes them to the log file.
Now if programmers define a custom Log function to replace the default one,
most messages will pass through their function.

Assert panic from AIOmanager during backup.

When a full-VM backup was being taken from a VDDK Linux proxy,
the assert message below could appear when AIOMgr_ExtractSystemErr
expected an AIO_SYSTEM error but got something else, such as end of file.
The code for handling disk data transactions was modified to correctly handle
an end-of-file (EOF) condition, instead of reporting an error.
The issue was fixed in the RC release.Panic: ASSERT /build/release/.../lib/public/aioMgr.h

Known Issues and Workarounds

The following known issues pertain to this VDDK release.
Lower-down issues are holdovers from the VDDK 1.2.1 release.

SAN restore of lazy-zeroed thick disk with mismatched block size.

When restoring lazy-zeroed thick disk using SAN transport,
if the restore buffer size is larger than the block size of the datastore,
writes may fail because the lazy zero flag is not cleared
for the second and subsequent blocks.
After restore, the virtual disk could be corrupted.
Consequently the restored virtual machine could be corrupted.
See KB 2055682.
A fix was identified for VDDK 5.1, and was included in the 5.0 U3 release.

Metadata write is not supported for HotAdd and SAN transport.

When a program calls VixDiskLib_WriteMetadata with HotAdd transport,
the function returns success (no error)
however the supplied metadata is not written to virtual disk.
With SAN transport, the “operation is not supported” error appears.
The workaround is to close the disk from HotAdd or SAN mode,
reopen the disk using NBD or NBDSSL mode, then call VixDiskLib_WriteMetadata.

Calling PrepareForAccess more than once returns success.

In vSphere 5.0 and 5.1, VixDiskLib_PrepareForAccess() returns successfully
even if programs call it multiple times before calling VixDiskLib_EndAccess().
Multiple calls indicate a problem in program logic.
In the VDDK 5.5 release, the VIX_E_FAIL error will be returned
when programs make a redundant VixDiskLib_PrepareForAccess() call.

Possible segmentation-violation if running multiple backup processes.

When many simultaneous backup processes are running,
some of them might crash with a SIGSEGV error.
This is due to a possible race condition in VixDiskLib,
which can be reduced by calling VixDiskLib_Exit() at end of your program.
A fix has been identified, and will be available in the VDDK 5.1 first update release.

SUSE Linux with < 2.6.27.21 kernel cannot use vixMntapi.

The vixMntapi library, and the vmware-mount command,
cannot mount disks on SUSE Linux Enterprise Server (SLES)
with kernel versions earlier than 2.6.27.21.
This is because earlier versions lack FUSE support.
So you cannot mount disks on SLES 11 (kernel 2.6.27.19)
without installing kernel 2.6.27.21 or a non-SUSE kernel.

Linux libgmodule shared object not included.

The VDDK 5.0 package does not include libgmodule-2.0.so.0
as it does in later releases.
If a Linux virtual machine lacks this shared object glib dependency,
you will have to install it.

Connect call crashes if non-existent snapshot is specified.

If a program calls VixDiskLib_ConnectEx() passing as third parameter
the managed object reference (moRef) of a deleted or non-existent snapshot,
the program crashes when dereferencing the snapshot moRef.
A fix has been identified, and will be available in a future release.
The workaround is to create a snapshot for each operation,
and never operate on virtual machines with different backup programs at the same time.

Problem in GetMetdataKeys with a large number of snapshots.

VDDK programs can crash in VixDiskLib_GetMetadataKeys() with an access violation
during backup of a virtual machine with many snapshots (about 30).
Although the VixDiskLibSample program does not do it,
the workaround is to open, get metadata, and close each snapshot one at a time.

Changed Block Tracking could return full thin-provisioned disk.

On very large thin-provisioned virtual disks (> 200GB)
the first QueryChangedDiskAreas call could return the entire disk,
rather than only the in-use areas of thin-provisioned disk.
This results in excessively large backup data.
It is especially noticeable on Linux ext2 and ext3 file systems.
A fix has been identified, and will be available in the VDDK 5.1 first update release.
The workaround is to use the numeric changeId returned from the first snapshot,
rather than the "*" (star) change ID, in the QueryChangedDiskAreas call.

HotAdd open fails when volume spans multiple disks.

When a Windows HotAdd proxy is configured with a volume that spans multiple disks,
VixDiskLib_Open() fails to open the HotAdded virtual disk
of the target virtual machine.
A fix has been identified and will be available in post 5.1 releases.

SLES 11 GUI interferes with HotAdd proxy backup.

When using SUSE Linux Enterprise Server (SLES) 11 SP2 as a backup proxy,
the Gnome automount utility gvfs-mount tries to mount HotAdded virtual disks,
causing I/O errors and forcible unmount after backup tries to access the disk device.
The workarounds are to avoid running the GUI, or disable the gvfs-fuse daemon.
You do not want Linux to mount the HotAdd virtual disks.

Slower than expected read speed when using NBDSSL transport.

Apparently due to use of the OpenSSL library version 0.9.8q (also designated 0.9.8.17),
throughput of NBDSSL transport is only about a third of the speed experienced
in VDDK 1.2.1, or about one sixth the speed of unencrypted NBD transport.
In the next VDDK release, this will be fixed
by installing bin\ssleay32.dll version 0.9.8.20 on Windows,
and libssl.so.0.9.8 version 0.9.8t library files
under /usr/lib/vmware-vix-disklib on Linux.

NBD transport limit is misdocumented as 1TB.

The Virtual Disk API Programming Guide says on page 23 that
there is a 1TB limit for LAN transport. This was true of VCB, but as of vSphere 4.0,
NBD transport size is limited only by the underlying file systems.

HotAdd restore by Windows 2008 proxy when opening multiple disks.

When a Windows Server 2008 virtual machine is deployed as a proxy,
HotAdd failures can occur when writing to the second and subsequent disks
of a virtual machine that is being restored. Due to SAN Policy changes,
W2k8 R2 and W2k8 R2 SP1 are more prone to this than earlier editions.
The cause: if COM function IVdsServiceLoader::LoadService is called soon after
its interface was released by a previous iteration, the call might fail.
There is no workaround. IVdsServiceLoader::LoadService can fail
the second time when called rapidly in succession, and VMware code
enumerates and mounts all disks the first time any disk is opened.
A fix has been identified for availability in a future release.

Get Volume Handles returns zero for dynamic disk.

The VixMntapi_GetVolumeHandles() function returns
zero volume handles for a Windows dynamic disk,
when newly created dynamic volumes are not assigned to a drive letter.
The workaround is to assign a drive letter.

To delay Storage vMotion, PrepareForAccess must run on vCenter not ESXi.

The Virtual Disk API Programming Guide on page 37 gives the impression that
programs can run VixDiskLib_PrepareForAccess() and VixDiskLib_EndAccess()
on ESXi hosts. Doing so actually throws an error saying
“VDDK: HostAgent is not a VirtualCenter, cannot disable sVmotion.”
These two functions must be run when connected to vCenter Server,
not when connected to an ESXi host.

EndAccess sometimes does not reenable relocation after PrepareForAccess.

With vCenter Server managing Storage vMotion on ESX/ESXi hosts,
the VDDK 5.0 library routine VixDiskLib_PrepareForAccess()
followed by VixDiskLib_EndAccess() occasionally does not
change virtual-machine state to reenable relocation after backup.
This seems more likely to occur when VixDiskLib_PrepareForAccess() is called
twice in a row (the second call gets “[relocation] already disabled” response).
Additionally the message “Insufficient permission in the host operating system”
(error 3014, VIX_E_HOST_USER_PERMISSIONS) could appear after these calls.
The two problems might or might not be related. Neither is easily reproducible.
Processes running as Administrator or root should be able to change migration state.

VDDK fails to return error for SAN writes to offline or read-only LUN.

When attempting SAN transport writes to virtual disk on an offline LUN,
write operations appear to succeed despite the LUN being offline.
Specifically, the VixDiskLib_Open() function returns success
because the LUN is exposed to the program,
but VixDiskLib_Write() fails to return an error
even though it cannot write to an offline LUN.
For backup and restore software running through a proxy,
this issue causes restores to return success,
when nothing is actually written to virtual disk.
VDDK 1.2.1 did return an error when attempting writes to an offline LUN,
so this is a regression. When a LUN is offline,
read and write operations should return an error.
A fix has been identified and will be provided in a future release.
Meanwhile, one workaround is to check if a LUN is offline before attempting a restore.
An alternate workaround is to keep LUNs always in online mode.

Cannot load, unload, and reload VixDiskLib.dll in one process.

A VDDK 5.0 program cannot call VixDiskLib_Init() or VixDiskLib_InitEx(),
then VixDiskLib_Exit(),
followed by VixDiskLib_Init() or VixDiskLib_InitEx() again.
Because the vixDiskLibPlugin must unload vmacore.dll on exit,
the program crashes when Init or InitEx requests VixDiskLib.dll reload.
Although the crash did not occur with VDDK 1.2.1 and before,
programs should be revised to eliminate multiple Init or InitEx calls in a process.

Problem using UTF-16 characters in pathnames.

VDDK supports only UTF-8 locales.
When VixDiskLib_Open() is given a pathname containing UTF-16 characters,
the virtual disk library fails to find the file.
On Windows 2008 for example, the pathname Temp\vmware-système\*vm*
contains è as a UTF-16 character, whereas VixDiskLib expects UTF-8.
One workaround is to override the Temp pathname in the configuration file
by setting the tmpDirectory key, using a non-UTF-16 pathname.
For details, see documentation for VixDiskLib_InitEx().
KB 1037379 discusses a similar issue.

Managed object not found for newly added virtual disk.

As of ESX/ESXi 4.1, if a virtual disk is added to a VM, the disk layout cached object
_cachedLayoutEx does not get updated with newly added disk information,
so operations on the new disk fail with a vmodl.fault.ManagedObjectNotFound (MONF) error.
There are three possible solutions:
(1) call vim.RefreshStorageInfo after adding virtual disk to a VM,
(2) create the disks when creating the VM so that
the _cachedLayoutEx object is updated at inception, or
(3) create a snapshot that will eventually call SendPropUpdateForSnapshots
to update the _cachedLayoutEx object.

HotAdd code should ignore independent disks, which are unsuitable for backup.

The HotAdd transport should not be used to back up virtual machines with a VMDK
configured as “independent“ (not capable of snapshots).
Any attempt to use HotAdd in this circumstance causes an error that results
in none of the VMDKs being HotAdded, even ones capable of snapshots.

Backup fails if a VM is relocated when a backup is in progress.

A backup in progress fails if a VM is relocated (for example with Storage vMotion)
before the backup completes. The workaround in this release is for programs to call
VixDiskLib_PrepareForAccess() to prevent VM relocation during backup,
and VixDiskLib_EndAccess() when the backup is complete.

If using HotAdd backup, add SCSI controllers to Linux VMs in numeric order.

Linux systems lack an interface to report which SCSI controller is assigned to which bus ID,
so HotAdd assumes that the unique ID for a SCSI controller corresponds to its bus ID.
This assumption could be false.
For instance, if the first SCSI controller on a Linux VM is assigned to bus ID 0,
but you add a SCSI controller and assign it to bus ID 3,
HotAdd advanced transport mode may fail because it expects unique ID 1.
To avoid problems, when adding SCSI controllers to a VM,
the bus assignment for the controller must be the next available bus number in sequence.
Also note that VMware implicitly adds a SCSI controller to a VM if a bus:disk
assignment for a newly created virtual disk refers to a controller that does not yet exist.
For instance, if disks 0:0 and 0:1 are already in place, adding disk 1:0 is acceptable,
but adding disk 3:0 breaks the bus ID sequence,
implicitly creating out-of-sequence SCSI controller 3.
To avoid HotAdd problems, you should also add virtual disks in numeric sequence.

Sometimes you must restore a VM directly to an ESXi host,
for example in disaster recovery when ESXi hosts the vCenter Server as a VM.
A new vSphere 5 feature tries to prevent this if the ESXi 5.0 host is managed by vCenter.
To circumvent this and restore the VM, you must first disassociate the host from vCenter.
In earlier releases,
vCenter management had less state but was revocable only from vCenter.

Using the vSphere Client, connect directly to the ESXi 5.0 host.

In the Inventory left-hand panel, select the host.

In the right-hand panel, click Summary.

There in the box titled Host Management, click Disassociate host from vCenter Server.
It is not necessary to put the host in Maintenance Mode.

Once the vCenter Server has been restored and is back in service, use it to reacquire the host.

SAN transport mode does not support writing snapshots, only base disk.

After opening remote disk in SAN transport mode,
programs can write to the base disk, but not to a snapshot (redo log).
Opening and writing to a snapshot is supported for hosted but not managed disk.
The snapshot is created to quiesce disk for backup, and should be deleted afterwards,
so writing a snapshot should seldom be required.

Advanced accelerated transport modes not available without XML.

Attempts to use advanced accelerated transport modes might fail if the
XML library expat is not installed in the guest operating system.
To resolve this issue, install version 1.95.8 of the XML library.
If binary compatibility is present, you can install a higher version.

Multiple sequential SAN mode connections cause WrongThreadException.

Attempts to use SAN transport mode to establish multiple sequential connections
might fail, resulting in a WrongThreadException.
For example, a first thread is created that connects to a disk,
reads from the disk, closes the disks, and then disconnects.
If a subsequent thread attempts to open the same disk, a failure occurs.
To avoid this issue, use one thread to establish sequential connections,
and worker threads to perform I/O on each connection,
as recommended in the Virtual Disk API Programming Guide.

Windows proxy HotAdd can cause differing disk signatures.

When backup is done on a Windows proxy with HotAdd transport,
Windows may rewrite the disk signature if it finds two identical signatures,
resulting in the first sector of the backup not matching the original.
To avoid this issue for backups, read the first sector of the disk using NBD transport,
then read the remaining sectors using HotAdd transport.
To avoid this issue for restores, write the entire disk using HotAdd transport,
close the disk and its connection handle, open a new disk connection handle,
and rewrite the first disk sector using NBD transport.

Cleanup of HotAdd disks can affect changed block tracking.

During backup or restore, disks may be HotAdded,
and subsequently removed after task completion.
This cleanup might remove the change tracking (ctk) file,
so that changed block tracking (CBT) could fail.
This issue should resolve itself automatically.
If it does not, the workaround is to power cycle the virtual machine.

Backing up thick disk requires the maximum disk size to be available.

To back up thick disk,
the proxy virtual machine datastore must have at least as much free space
as the maximum configured disk size for the virtual machine to be backed up.

VixMntapi does not support advanced transports on Linux.

The vixMntapi library for Windows supports advanced transport for SAN and HotAdd,
but the library for Linux supports only local and LAN transport (file, nbd).

Should reboot Windows after uninstalling and reinstalling VDDK.

After you uninstall and then reinstall VDDK on a Windows system,
you must reboot to ensure that the vStor driver gets replaced properly,
otherwise you may see an “Internal Error 29144” message.
This does not happen with a clean install.

VixDiskLib should check SAN accessibility on disk open.

Programs that open a local VMDK in SAN mode might be able to read
(if the disk is empty) but writing to disk always fails.
Checking accessibility on open is a feature enhancement request.
When a program calls VixDiskLib_ConnectEx() with NULL parameter
to accept the default transport, SAN is selected as the preferred mode
if SAN storage is available from the ESX/ESXi host,
So after the program opens a local VMDK, upcoming writes will fail.
In this case, the program should explicitly request nbd or
nbdssl as the preferred transport mode.
This advice also applies to thin-provisioned disk,
which is supported on local VMDK.

VixDiskLib_PrepareForAccess

This function is used to notify the host of the virtual machine that the disks
of the virtual machine will be opened. The host disables operations on the
virtual machine that may be adversely affected if they are performed while the
disks are open by a third party application.

This function must be called before creating a snapshot on the virtual machine
or opening any disks of the virtual machine