About the 5.1 Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 5.1 is an update to support vSphere 5.1
and to resolve issues discovered in previous releases.
VDDK 5.1 adds support for ESXi 5.1 and vCenter Server 5.1,
and was tested with the following VMware platform products:

ESXi 5.1, ESXi 5.0, and ESX/ESXi 4.1

vCenter Server 5.1 and 5.0 managing ESX/ESXi 4.1 and later

VDDK 5.1 was tested with (and supports) the following operating systems to perform proxy backup:

Windows Server 2012 (64-bit only)

Windows 8

Windows Server 2008 R2 (64-bit only)

Windows Server 2008 (64-bit only)

Windows Server 2003 R2

Windows XP

SUSE Linux Enterprise Server SLES 11 SP2.

Red Hat Enterprise Linux RHEL 6.1.

For Linux, only the default out-of-box file system is tested and supported.

The handling of writable snapshots changes in the ESXi 5.1 release,
however this seldom affects backup applications, which can use read-only snapshots.

VMware Consolidated Backup (VCB) has a writeup
(knowledge base article)
showing the support matrix for storage multipathing.
VMware does not provide a similar support matrix for VDDK.
Customers should seek this information from their backup software vendors.

These release notes provide information about issues identified or resolved since VDDK 5.0.
For information about the 5.0 release,
see the VDDK 5.0 Release Notes.
For general information about the development kit,
how to obtain and install the software, redistributing it, and issues in previous releases,
see the VDDK documentation landing page.

Compatibility Notices for Partners

Linux proxy SAN backup can fail with powered on VM.
If any virtual machine I/O operations are occurring for a VMDK on a LUN
when the Linux proxy starts a SAN transport backup, proxy systems
before SLES 11.2 and RHEL 6.2 receive an EIO instead of EBUSY error,
causing backup to fail. This is more likely to occur with a powered on VM.
The cause is a Linux kernel bug when handling SCSI reservation conflicts,
fixed in newer kernels, and encountered on the following releases:

SLES 10.4 and SLES 11.1

RHEL 5.8, RHEL 5.9, RHEL 6.0, RHEL 6.1

Only one plug-in.
After implementation of SSL certificate checking,
it was necessary to limit the VDDK framework to one plug-in,
diskLibPlugin.dll or libdiskLibPlugin.so,
which handle the advanced transport modes: SAN, HotAdd, and NBDSSL.
As a result, VDDK 5.1 can no longer load a custom partner plug-in.
This is a known issue, and it will be corrected in a future release.

Registry on 64-bit Windows.
To support registry redirection, registry entries needed by VDDK
on 64-bit Windows must be placed under registry hierarchy Wow6432Node.
This is the correct location for both 32-bit and 64-bit binaries on 64-bit Windows.
On 32-bit Windows, VDDK registry entries are in the old location.

Snapshot changes.
In previous VMware releases, reverting to a writable snapshot
sometimes left orphaned virtual disks that the system never removed.
In this release, writable snapshots are correctly accounted for as sibling snapshots.
This avoids orphaned disks, and streamlines snapshot management
because the disk chain always matches the snapshot hierarchy.
VMware snapshots are read-only by default. In a CreateSnapshot call,
the quiesce flag synchronizes data and creates a read/write snapshot
for use with Microsoft VSS, which requires writable snapshots.
VMware recommends testing snapshot behavior for that use case.

Licensing. In vSphere 5.1, the SCSI HotAdd feature is always available.
In vSphere 5.0, SCSI HotAdd was enabled only for vSphere editions Enterprise and higher,
which have Hot Add licensing enabled.
No separate Hot Add license was available for purchase as an add-on.
Therefore, customers who used backup products (including VMware Data Recovery)
with vSphere Essentials or Standard edition were not able to perform proxy-based backup.

SSL certificate check.
VDDK 5.1 can be set to check the SSL certificate on Windows,
or to verify the SSL thumbprint on Linux.
To enable and disable this feature, you can edit the Windows registry,
or add linuxSSL.verifyCertificates=1 to the configFile
passed as a parameter to VixDiskLib_InitEx().

Relocated files.
For reasons related to SSL security, in VDDK 5.1 the following directories found in
previous releases were eliminated, and files therein moved to the following locations:

Windows

VDDK\install\directory\plugins

VDDK\install\directory\bin

Linux 32-bit

VDDK/install/directory/plugins32

VDDK/install/directory/lib32

Linux 64-bit

VDDK/install/directory/plugins64

VDDK/install/directory/lib64

Older VDDK 5.0 items follow below.

DNS.
To avoid SSL certificate errors, DNS services must be configured in the backup proxy,
consistent with vCenter Server and ESXi hosts in the datacenter.

Storage vMotion. To prevent orphaned virtual disk
caused by Storage vMotion during backup, VDDK 5.0 added the
VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions,
which disable and re-enable the RelocateVM_Task method.
Storage vMotion can wait for the backup to finish, or relocate a different VM.
See the Virtual Disk API Programming Guide for details.

Silent VDDK installation and the EULA.
If you want to accept all defaults and install VDDK components silently on a Linux machine,
providing you agree to terms in the end user license agreement (EULA),
you can use the combined options --defaultEULA_AGREED=yes.

Windows silent install.
If you want to perform silent install or uninstall of VDDK with InstallShield on Windows,
run one of the following commands. The /s option means silent, the /x option means uninstall,
and the /v option passes /qb (basic UI) or /qn (no UI) to MsiExec.

Recently Resolved Issues

The VDDK 5.1 release resolves the following issues:

SAN restore of lazy-zeroed thick disk with mismatched block size.

When restoring lazy-zeroed thick disk using SAN transport,
if the restore buffer size was larger than the block size of the datastore,
writes could fail because the lazy zero flag was not cleared
for the second and subsequent blocks.
After restore, the virtual disk could be corrupted.
See KB 2055682.
This issue has been fixed in this release.

Improved support for very large files.

In previous VDDK releases, VixDiskLib_Create() enforced a 2GB limit on
virtual disks, except with NTFS, FAT32, or FAT file systems, which were limited to 4GB.
In this release, the NTFS limit is 16TB - 64KB (hex FFFFFFF0000),
the FAT32 or FAT limit is still 4GB - 1 (hex FFFFFFFF),
and the exFAT or ReFS limit is more than an exabyte (2^64 - 1).
POSIX based file systems including NFS no longer have a 2GB file size limit.
Although various checks are done to avoid creating impossibly large files,
it becomes the customer's responsibility to cope with 2GB limits on NFSv2
or Linux kernel 2.4 and earlier.

Slower than expected read speed when using NBDSSL transport.

Due to VDDK 5.0's use of OpenSSL library version 0.9.8q (also designated 0.9.8.17),
throughput of NBDSSL transport was only about a third as fast as VDDK 1.2.1,
or about one sixth the speed of unencrypted NBD transport.
In VDDK 5.1, this was fixed by installing newer versions of
bin\ssleay32.dll on Windows and libssl.so.* on Linux.

VixDiskLib transmits meaningful error code for thin-disk full.

In earlier releases, VixDiskLib returned an “Unknown error”
from diskLib when thin-provisioned disk ran out of space.
The actual diskLib error is now returned with VIX_E_DISK_FULL,
in the most-significant 16 bits of the error code.

QuickExit() error no longer appears.

In VDDK 5.0, the warning “This application is not using QuickExit()”
came from the vmacore library as a diagnostic for VMware applications.
Though it was harmless, this message no longer appears.

VDDK fails to return error for SAN writes to offline or read-only LUN.

When attempting SAN transport reads or writes to virtual disk on an offline LUN,
VDDK 5.0 I/O operations appeared to succeed despite the LUN being offline.
The VixDiskLib_Open() function returned success but
VixDiskLib_Read() and VixDiskLib_Write()
failed to return an error even when they could not read or write the offline LUN.
For backup and restore software running through a proxy,
this issue caused backups and restores to return success,
when data was never actually read from or written to virtual disk.
This is fixed in the VDDK 5.1 and VDDK 5.0 U2 releases.

EndAccess sometimes does not reenable relocation after PrepareForAccess.

With vCenter Server managing vMotion on ESX/ESXii hosts,
the VDDK 5.0 library routine VixDiskLib_PrepareForAccess()
followed by VixDiskLib_EndAccess() occasionally did not
change virtual-machine state to reenable relocation after backup.
This seemed more likely to occur when VixDiskLib_PrepareForAccess() was called
twice in a row (the second call got “[relocation] already disabled” response).
Additionally the message “Insufficient permission in the host operating system”
(error 3014, VIX_E_HOST_USER_PERMISSIONS) could appear after these calls.
Now, processes running as Administrator or root should be able to change migration state.

Snapshot consolidation when cleaning up after HotAdd backup.

When VMware Data Recovery (or VADP backup software) tried to clean up after HotAdd backup,
snapshot delete failed as the appliance was reconfigured, and an exception was reported.
The problem was caused by premature exit of the vpxa process,
so vpxa did not inform backup code that HotAdd disks should be removed.
Stale HotAdd disks remained, and were not recognized at subsequent backup time,
causing failure of snapshot delete and buildup of redo logs.
The solution is to wait longer for vpxa to report,
and apply heuristics to clean up stale HotAdd disks if they appear.
The VDDK 5.1 fix also works with earlier ESX/ESXi versions
and for stale redo logs left by earlier VDDK versions.

HotAdd restore by Windows 2008 proxy when opening multiple disks.

When a Windows Server 2008 virtual machine was deployed as a proxy,
HotAdd failures occurred when writing to the second and subsequent disks
of a virtual machine that was being restored. Due to SAN Policy changes,
W2k8 R2 and W2k8 R2 SP1 were more prone to this than earlier editions.
This advanced transport issue was fixed for the VDDK 5.1 release.

Known Issues and Workarounds

The following VDDK issues were reported after the October 2012 vSphere refresh.

Threads still running after VixDiskLib_Exit call.

When the VDDK plugin is loaded, several threads are started
that cannot be shut down until the calling application exits.
For instance on Windows Server 2008,
VDDK libraries may leave threads running after the VixDiskLib_Exit call.
Still-running threads are from the vCenter vmacore library
and cannot be unloaded until the application finishes.
The leftover threads should be harmless,
except if the backup proxy runs as a VM on ESXi 5.1,
the calling application can terminate unexpectedly.

Error code may not be set correctly before VDDK exit.

An application calling VDDK libraries should return its own exit code,
but QuickExit (part of the vmacore library)
may terminate the VDDK application before it can set the proper exit code.
This problem began when VDDK 5.1 was modified to call QuickExit.
A fix has been identified, but cannot be applied to a VDDK 5.1 update release
due to the extensive nature of required code changes.

Metadata write is not supported for HotAdd and SAN transport.

When a program calls VixDiskLib_WriteMetadata with HotAdd transport,
the function returns success (no error)
however the supplied metadata is not written to virtual disk.
With SAN transport, the “operation is not supported” error appears.
The workaround is to close the disk from HotAdd or SAN mode,
reopen the disk using NBD or NBDSSL mode, then call VixDiskLib_WriteMetadata.

When calling QueryChangedDiskAreas to get changes between application level
quiesced snapshots, the areas returned can represent a much larger amount of changes
than the actual amount of changes. This issue occurs when taking quiesced snapshots
to back up powered-on VMs running Windows Server 2003, 2008, 2008 R2, and 2012.
The workaround was to set disk.enableUUID=false for these target VMs,
but this resulted in file-level instead of application level quiesced backups.
The fix is to install ESXi 5.1 Patch 02, released 25 JUL 2013.
See KB 2052143 for details.

Calling PrepareForAccess more than once returns success.

In vSphere 5.0 and 5.1, VixDiskLib_PrepareForAccess() returns successfully
even if programs call it multiple times before calling VixDiskLib_EndAccess().
Multiple calls indicate a problem in program logic.
In the VDDK 5.5 release, the VIX_E_FAIL error will be returned
when programs make a redundant VixDiskLib_PrepareForAccess() call.

Prepare for Access identity string length limitation.

The VixDiskLib_PrepareForAccess() parameter *identity
is a string limited to 50 characters.

Possible segmentation-violation if running multiple backup processes.

When many simultaneous backup processes are running,
some of them might crash with a SIGSEGV error.
This is due to a possible race condition in VixDiskLib,
which can be reduced by calling VixDiskLib_Exit() at end of your program.
A fix has been identified, and will be available in the VDDK 5.1 first update release.

Connect call crashes if non-existent snapshot is specified.

If a program calls VixDiskLib_ConnectEx() passing as third parameter
the managed object reference (moRef) of a deleted or non-existent snapshot,
the program crashes when dereferencing the snapshot moRef.
A fix has been identified, and will be available in a future release.
The workaround is to create a snapshot for each operation,
and never operate on the same virtual machine at the same time
from two or more programs or threads.

SLES 10 SP2 requires newer OpenSSL library.

VDDK 5.1 libraries are non-functional on SUSE Linux Enterprise Server 10.2,
and perhaps other Linux distributions, until you install the libssl.so.0.9.8x
shared object to replace the libssl.so.0.9.8t version.
The newer OpenSSL library can deal with older versions of glibc on SLES 10.
A fix has been identified, and will be available in the VDDK 5.1 first update release.

Occasional segmentation-violation crashes during library disconnect.

When many Linux processes run simultaneously, some of them may get a SIGSEGV
when calling VixDiskLib_Disconnect() before exit. This may also occur on Windows.
One workaround is to have nothing important in your code after disconnect.
A fix has been identified, and will be available in the VDDK 5.1 first update release.

Hang in connect or cleanup due to intermittent race condition.

Windows processes can sometimes hang when calling VixDiskLib_ConnectEx() or
VixDiskLib_Cleanup() due to a race condition while loading or unloading libraries.
There is no known workaround.
A fix has been identified, and will be available in the VDDK 5.1 first update release.

The following VDDK issues were discovered during the vSphere 5.1 release.

Disk Open fails without an SSL thumbprint from vCenter.

Whenever vCenter Server does not provide an SSL thumbprint for the target host,
the VDDK library might core dump in the function VixDiskLib_Open().
This could happen (but does not always happen, depending on memory content)
when vCenter Server 5.1 manages virtual disk on ESXi 5.0 and older hosts.
A fix has been identified, and will be available in the VDDK 5.1 first update release.

Specification of PluginsDir is misdocumented.

Because of the Relocated files change
described under Compatibility Notices for Partners above,
in the Virtual Disk Programming Guide section Usage Message (pages 44-45),
when using 32-bit Linux substitute /lib32 for /plugins32,
and when using Windows substitute \bin for \plugins.

Problem in GetMetdataKeys with a large number of snapshots.

VDDK programs can crash in VixDiskLib_GetMetadataKeys() with
an access violation during backup of a virtual machine with many snapshots
(about 30). Although the VixDiskLibSample program does not do it,
the workaround is to open, get metadata, and close each snapshot one at a time.

HotAdd open fails when volume spans multiple disks.

When a Windows HotAdd proxy is configured with a volume that spans multiple disks,
VixDiskLib_Open() fails to open the HotAdded virtual disk
of the target virtual machine.
A fix has been identified and will be available in a future release.

The vmware-mount command does not work correctly in this release.

When you run the vmware-mount command in the VDDK 5.1 bin directory,
it displays the following error:
“Unable to mount the virtual disk. The disk may be in use...
verify that the file is a valid virtual disk file.”
This is the result of incompatible libraries.
At this time VMware knows of no workaround.

Get Volume Handles might return zero for dynamic disk.

When newly created dynamic volumes are not assigned to a drive letter,
the VixMntapi_GetVolumeHandles() function returns
zero volume handles for a Windows dynamic disk.
A possible fix was identified and may be available in a future release.
The workaround is to assign a drive letter.

With verbose logging, long-running VDDK processes might hang.

The dynamic library diskLibPlugin.dll could hang
after running for an extended period of time. If you encounter this issue,
one workaround is to set the logging level to 4 or less (non-verbose) by
setting transport.LogLevel in the configFile passed to VixDiskLib_InitEx().
Another workaround is to specify NBD explicitly
in the TransportModes parameter of VixDiskLib_InitEx().
A fix has been identified and will be provided in the first update release.

vixMntapi on Windows can access only the first 2TB of a volume.

The vixMntapi functions can be used to mount a Windows volume larger than 2TB.
However, attempts to access files on the resulting mount may result in this error
message: “The file system structure on the disk is corrupt and unusable.
Please run the chkdsk utility on the volume.”
Customers should not run chkdsk in this case.
Even if the message does not appear, there is a hazard of corrupting
the mounted volume if you attempt to change it, and a risk of getting
incorrect data if the accessed file exists beyond the 2TB limit.
There is no good workaround.
Splitting the volume is not always possible or convenient.

SLES 11 GUI interferes with HotAdd proxy backup.

When using SUSE Linux Enterprise Server (SLES) 11 SP2 as a backup proxy,
the Gnome automount utility gvfs-mount tries to mount HotAdded virtual disks,
causing I/O errors and forcible unmount after backup tries to access the disk device.
The workarounds are to avoid running the GUI, or disable the gvfs-fuse daemon.
You do not want Linux to mount the HotAdd virtual disks.

Changed Block Tracking could return full thin-provisioned disk.

On VMFS file systems holding very large thin-provisioned disk (> 200GB)
the first QueryChangedDiskAreas call could return the entire disk,
rather than only the in-use areas of thin-provisioned disk.
This results in excessively large backup data.
It is especially noticeable on Linux ext2 and ext3 file systems.
A fix has been identified, and will be available in the VDDK 5.1 first update release.
The workaround is to use the numeric changeId returned from the first snapshot,
rather than the "*" (star) change ID, in the QueryChangedDiskAreas call.

Dependency on libexpat.so.0 XML parsing library.

The VDDK libraries on a Linux proxy have a dependency on libexpat.so.0,
but most current Linux distributions are shipping libexpat.so.1 instead.
A workaround is to create a symbolic link to libexpat.so.1
with the name libexpat.so.0 in the /lib or /usr/lib directory.

The following known issues pertain to the VDDK 5.1 release
but were reported previously for the VDDK 5.0 release.

Cannot load, unload, and reload VixDiskLib.dll in one process.

A VDDK 5.x program cannot call VixDiskLib_Init() or VixDiskLib_InitEx(),
then VixDiskLib_Exit(),
followed by VixDiskLib_Init() or VixDiskLib_InitEx() again.
Because the vixDiskLibPlugin must unload vmacore.dll on exit,
the program crashes when Init or InitEx requests VixDiskLib.dll reload.
Although the crash did not occur with VDDK 1.2.1 and before,
programs should be revised to eliminate multiple Init or InitEx calls in a process.

Problem using UTF-16 characters in a pathname.

VDDK supports only UTF-8 locales.
When VixDiskLib_Open() is given a pathname containing UTF-16 characters,
the virtual disk library fails to find the file.
On Windows 2008 for example, the pathname Temp\vmware-système\*vm*
contains è as a UTF-16 character, whereas VixDiskLib expects UTF-8.
One workaround is to override the Temp pathname in the configuration file
by setting the tmpDirectory key, using a non-UTF-16 pathname.
For details, see documentation for VixDiskLib_InitEx().
Alternatively, software can be modified to use library routines
such as iconv() for transcoding UTF-16 to UTF-8 and vice versa.

VDDK sample code may require additional header files.

On some Linux distributions, you must include <stdio.h> and <string.h>
near the top of vixDiskLibSample.cpp for the program to compile correctly.

At-sign (@) in datastore names.

While all versions of vSphere allow the at-sign (@) in datastore names,
the VixDiskLib API is unable to open VMDK files inside such datastores.
You might want to modify your backup code to throw an error in such a case.
The customer workaround is to avoid datastore names with the at-sign (@)
when using VDDK.

Linux C++ libraries required.

VDDK 5.0 and later require C++ library support,
which many older Linux versions do not provide in the base install.
When using an old Linux distribution as a backup proxy,
customers might need to install C++ library packages from the distribution ISO.

VIX API and VDDK libraries are not compatible.

If a process calls both VixDiskLib_Open() and VixHost_Connect()
it results in a crash and possible core dump.
Both VIX and VDDK libraries depend on the gvmomi library
and call its init() function, which exits due to assertion failure.
See KB 2016054 for a workaround.
A fix has been provided in the VIX 1.13 release.

VixDiskLib should check SAN accessibility on disk open.

Programs that open a local VMDK in SAN mode might be able to read
(if the disk is empty) but writing to disk always fails.
Checking accessibility on open is a feature enhancement request.
When a program calls VixDiskLib_ConnectEx() with NULL parameter
to accept the default transport, SAN is selected as the preferred mode
if SAN storage is available from the ESXi host,
so after the program opens a local VMDK, upcoming writes will fail.
In this case, the program should explicitly request nbd or
nbdssl as the preferred transport mode.
This advice also applies to thin-provisioned disk,
which is supported by local VMDK.

Windows proxy HotAdd can cause differing disk signatures.

When backup is done on a Windows proxy with HotAdd transport,
Windows may rewrite the disk signature if it finds two identical signatures,
resulting in the first sector of the backup not matching the original.
To avoid this issue for backups, read the first sector of the disk using NBD transport,
then read the remaining sectors using HotAdd transport.
To avoid this issue for restores, write the entire disk using HotAdd transport,
close the disk and its connection handle, open a new disk connection handle,
and rewrite the first disk sector using NBD transport.