How to upgrade to VMFS 5 on VMware and VMFS 5 facts

I wrote this last year but never published, working on clearing out some old posts. Along with the long list of other features added to vSphere 5, VMware has included a new version of VMFS. The upgrade in VMFS brings us to version 5 of the file system.

The main focus of VMware while creating VMFS seems to be making it easier to manage storage in virtual environments. In VMFS-5 the number of storage related objects that need to be managed by an VMware administrator are far less. For example you can now use larger and fewer datastores, because the scaling limits of VMFS-5 have been increased.

VMFS-5 New Features

Unified 1MB File Block Size. Past versions of VMFS used 1,2,4 or 8MB file blocks. The larger block sizes would allow you to create files larger than 256GB. There is now just one block size in VMFS-5 allowing you to create up to 2TB VMDK files using the 1MB file blocks.

Larger Datastores. In previous versions of VMFS, the largest datastore size without extents was 2TB-512 bytes. With VMFS-5 this limit has been increased to 64TB.

Smaller Sub-Block. VMFS-5 introduces a smaller sub-block. The new size is now 8KB rather than the old 64KB size from previous versions. Now a small file less than 8KB but larger than 1KB in size will only consume 8KB rather than 64KB. This will reduce the disk space being consumed by these small files.

Small File Support. Support for files less than or equal to 1KB, now use the file descriptor location in the metadata for storage rather than file blocks. If they grow above 1KB, these files will then start to use the new 8KB sub blocks. The net result is a reduction in space consumed by small files.

Increased File Count. VMFS-5 now allows support for more than 100,000 files. In VMFS-3 the limit was 30,000 files.

ATS Enhancement. ATS is now used all through VMFS-5 for file locking. ATS (Atomic Test & Set) is a Hardware Acceleration primitive, and is part of the VAAI (vSphere Storage APIs for Array Integration). This improves the file locking performance over previous versions of VMFS.

Upgrade considerations and effects

I can say with all confidence that there is no reason not to upgrade to VMFS 5. The question would be how will you go about the upgrade process to VMFS 5? You can do a non-disruptive in place upgrade without affecting any running virtual machines on the datastore. The second option is if you have the space create a new datastore and migrate your virtual machines over to it once created. I have explained the differences between the two methods below and what drawbacks an upgrade might have below.

One of the biggest drawbacks that I came across was the performance hit that you could take on your Storage vMotions. Since upgraded datastores retain their original block size it could be larger than the unified 1MB size in VMFS-5. You will not get the VAAI benefits of faster Storage vMotions when migrating a virtual machine between datastores with different block sizes. This means that Storage vMotions will revert back to an application level Storage vMotion and the process will take 2 to 5x longer than a hardware assisted copy.

This negative affect on svMotions would affect manual migrations but would have a big effect on Storage DRS where VMs are move based on storage performance decisions.

Facts about VMFS-3 to VMFS-5 Upgrades

You can upgrade from VMFS-3 to VMFS-5 is a non-disruptive operation. Your VMs can continue to run on the datastore.

Upgraded VMFS-5 can use the new 1KB small-files feature.

Upgraded VMFS-5 can be grown to 64TB, same as a newly created VMFS-5.

Upgraded VMFS-5 has all the VAAI ATS improvements that a newly created VMFS-5 has.

What’s different newly created vs. upgraded VMFS-5 datastores

Upgraded datastores continue to use the previous file block size which may be larger than the unified 1MB file block size.

Upgraded datastores continue to use 64KB sub-blocks and not new 8K sub-blocks.

Upgraded datastores continue to have a file limit of 30720 rather than new file limit of greater than 100,000 for newly created VMFS-5 datastores.

Upgraded datastores continue to use MBR (Master Boot Record) partition type; when the VMFS-5 volume is grown above 2TB, it automatically & seamlessly switches from MBR to GPT (GUID Partition Table) with no impact to the running VMs.

Upgraded datastores continue to have its partition starting on sector 128; newly created VMFS5 partitions will have their partition starting at sector 2048.

RDM – Raw Device Mappings

There is now support for passthru RDMs to be ~ 60TB in size.

Non-passthru RDMs are still limited to 2TB – 512 bytes.

Both upgraded VMFS-5 & newly created VMFS-5 support the larger passthru RDM.

Other Facts.

The maximum VMDK size is still 2TB -512 bytes on VMFS-5.

The maximum size of a non-passthru (virtual) RDM on VMFS-5 is still 2TB -512 bytes.

The maximum number of LUNs supported is still 256 on an ESXi 5.0 host.

I would recommend based on all the information listed in this chapter that the best method to migrate to VMFS-5 is to create new VMFS-5 datastores. To do this you will of course need to have the storage space to do this and you can then evacuate your existing stores one by one with Storage vMotions. It’s a bit of a shell game moving VMs around as you create new datastores but you will end up with a storage environment that is not affected by any limitations carried over from VMFS-3.

How to upgrade a datastore to VMFS 5

Now that you have either installed or upgraded some hosts to ESXi 5 you probably have datastores that are still at the VMFS 3 level. With vSphere 5 you will be able to continue running your VMs on a datastore that is at VMFS 3 but will want to upgrade them when possible. The upgrade is non-destructive but I would recommend that you make sure you have verified backups of any VMs running on the datastore being upgraded or Storage vMotion them off before upgrading.

During the writing of this book we did numerous tests in a lab environment and each test VMFS upgrade of a datastore with powered off VMs succeeded without issues.

Another thing to note is the option to select your block size on a datastore is gone. When upgrading a VMFS 3 datastore to VMFS 5 it will retain its original block size. All freshly created VMFS 5 datastores will have a block size of 1MB. They will both act the same, allowing you to create a datastore greater than the previous 2TB limit. You are still bound with a 2TB limit for individual VMDK files.

Upgrade datastore to VMFS 5 from vSphere Client

Of the different methods using the vSphere Client is the easiest method to upgrade a datastore to VMFS 5. If you refer to Figure 1 below, you will be looking at a view of the datastore. From this view you can see the selected datastore type is still VMFS 3. Just below you will have the option to click on “Upgrade to VMFS 5” which will kick off the upgrade.

Figure 1: Datastore view

Prior to the upgrade process starting you will see a warning that looks like the one in Figure 2 below. The warning is letting you that any hosts not capable of communicating with VMFS 5 datastores will lose their ability to read or write to them.

Figure 2: VMFS upgrading warning

Upgrade datastore to VMFS 5 with esxcli command

This method will use the esxcli command and can be executed from the ESXi shell or a remote vCLI prompt. This gives you flexibility on where you can run it from and include in scripting.

The command syntax will look like the one of the options listed below, depending on whether you choose to use the Volume name or the UUID for the datastore. This method does not provide any warnings or ask for any confirmations, refer to Figure 3 below.

#esxcli storage vmfs upgrade –l volume_name

#esxcli storage vmfs upgrade – volume_uuid

Figure 3: esxcli command example

Figure 4 below shows the command syntax for the CLI command.

Figure 4: esxcli help

Upgrade datastore to VMFS 5 with vmkfstools command

This method has been around for a while and most of you might has used this to upgrade to VMFS 3. One thing of note is you will not be able to use vmkfstools to upgrade to VMFS 5 from the remote vCLI. This command can only be run from the ESXi shell on the host.

To perform the upgrade you will use the command structured like the one listed below and in Figure 5 below.

#vmkfstools –T /vmfs/volumes/datastore_name

You will be warned about having no active connections to the datastore while upgrading. Also a warning to backup any data on the datastore and to make sure that no ESX(i) 3.x/4.x are still trying to access the datastore.

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Hi Brian,
We are just now looking at the upgrade path from 4.1 to 5.x.
I’m missing a nuance in the different statements your article makes:
“You can do a non-disruptive in place upgrade without affecting any running virtual machines on the
datastore.” “You can upgrade from VMFS-3 to VMFS-5 is a non-disruptive operation. Your VMs can continue to run on the datastore.”
BUT “During the writing of this book we did numerous tests in a lab environment and each test VMFS upgrade of a datastore with **powered off VMs** succeeded without issues,” And the CLI interface warns “any hosts not capable of communicating with VMFS 5 datastores will lose their ability to read or write to them”, and “You will be warned about having no active connections to the datastore while upgrading.”
Those appear to be contradictory – your guests were offline during the upgrade tests, and the CLI and GUI warnings refer to lost connectivity. Does that mean the guests appear offline though internally they will still be running?
Also, Is the duration of the vmfs upgrade affected by the size of the volumes you are using, in that it has to touch each data block or sector, or is it just a disk table index update?
Regards,
Corrie

That could have been written a clearer. It can upgrade a datastore without interruption of running VMs. The warning would be if you have hosts that are not vSphere 5.0 using that datastore, they would loose connection to it after the upgrade.