Also remember that vSAN traffic will use the default gateway of the Management VMkernel interface to attempt to reach the Witness, unless additional networking is put in place, such as advanced switch configuration or possibly static routing on ESXi hosts and/or the Witness Appliance

New in 6.5VMware vSAN 6.5 supports the ability to directly connect two vSAN data nodes using one or more crossover cables.

This is accomplished by tagging an alternate VMkernel port with a traffic type of "Witness." The data and metadata communication paths can now be separated. Metadata traffic destined for the Witness vSAN VMkernel interface, can be done through an alternate VMkernel port.

I like to call this "Witness Isolated Traffic" (or WIT), I think we calling are it "Witness Traffic Separation" (or WTS).

With the ability to directly connect the vSAN data network across hosts, and send witness traffic down an alternate route, there is no requirement for a high speed switch for the data network in this design.

This lowers the total cost of infrastructure to deploy 2 Node vSAN. This can be a significant cost savings when deploying vSAN 2 Node at scale.

The HowTo use ports for vSAN today, VMkernel ports must be tagged to have "vsan" traffic. This is easily done in the vSphere Web Client. To tag a VMkernel interface for "Witness" traffic, today it has to be done at the command line.
To add a new interface with Witness traffic is the type, the command is:

esxcli vsan network ipv4 add -i vmkX -T=witness

We can configure a new interface for vSAN data traffic using this command (rather than using the Web Client)

esxcli vsan network ipv4 add -i vmkX -T=vsan

When looking to see what our vSAN network looks like on a host, we use:

esxcli vsan network list

It should look something like the image to the right:

Notice that vmk0, the management VMkernel interface in this example, has Witness traffic assigned.

In the example shown, vmk0, on each data node, requires connectivity to the VMkernel port vmk1 on the Witness Appliance. The vmk2 interface on each data node could be direct connected.

Also keep in mind that it is cleaner & easier to have the direct connected nics on their own vSphere Standard Switch, or possibly a vSphere Distributed Switch. Remember that vSAN brings the vSphere Distributed Switch feature, regardless of what version of vSphere you are entitled to.

If we have dual nics for redundancy, we could possibly assign vMotion traffic down the other 10Gbps nic. Couple that with NIOC, and we could ensure that both vSAN and vMotion could have enough resources in the event of one of the two nics failing.

Comments

22 Comments been added so far

Brandon

June 14th, 2017

Sweet. Thank you! This helped me very much. Thank you for the video. It helped show me how to setup the networking. The part that had me stumped for awhile, was how to configure only 1 nic per port group. I finally found the setting editing the distributed port group settings. Then clicking on Teaming and failover I configured 1 uplink active and the other to standby.

Jase McCarty

July 13th, 2017

Actually 2 Node vSAN has been supported since 6.1.
2 Node vSAN with Direct Connect was introduced with vSAN 6.5, but is supported on vSAN 6.0 Patch 3 or higher (vSAN 6.2).

Thank you,
Jase

Pradeep

July 20th, 2017

Hi Jase,

Does it require vcenter also to be version above 6 update 3 for a 2 node direct connect setup?

Hi,
Does it require vcenter also to be version above 6 update 3 for a 2 node direct connect setup?

Thanks.

Jase McCarty

August 16th, 2017

vCenter Server will need to be at a version that is compatible with vSphere 6.0 Update 3.
I haven’t personally tested this with vSphere 6.0 Update 3 and an older vCenter 6.0.http://vmwa.re/vsanvcinterop

What about the Idea to use the management VMkernel for the Witness traffic also in a 4-Node stretched Cluster?
I’m just thinking of it – because, then there would be no need to route the VSAN Network (Layer 3) to the Witness Host?

In our case, the 10Gbit VSAN Network could still act as an “isolated LAN” and would need no uplink to the core-network…

Thanks,
Patrick

Valter

September 29th, 2017

Since direct connect is used and the witness traffic is carried over management vmk, what to do with the WitnessPg vmk in witness esxi host?

Hi guys, about the direct connect, where do I find some HCL information about 10GB SFP+ direct connect without switches and over long distance +-3km. I think to interlink 2 VSAN nodes with Transceiver, SFP+, 10GbE, LR, 1310nm Wavelength over fibre.

Hi Mr. McCarty,
Thank you for your very usefull explanation regarding this capability of VSAN.
So as I understand, the witness appliance is required anyway, even in this case of two nodes attached directly by the 10 GigaB NIC.
Is that right??
Thank you in advance for your gently reply.
Best regards
Dante Carlini

Hi Jase
I have configured this but it seems incorrect with messages saying 0 witness detected etc

Only thing I did was ran the CLI commands after everything was setup
Do the hosts require rebooting ? Or is there a NW rest required of some sort ?
Tks

Jase McCarty

March 14th, 2018

Pradeep,

Sorry for the late response, just seeing this. Your vCenter should be at least vCenter 6.0 Update 3 for this to work with vSphere 6.0 Update 3.

Jase McCarty

March 14th, 2018

Cezar,

As long as the connection medium supports it, and there is <5ms latency between hosts, you should be fine.

Jase McCarty

March 14th, 2018

Dante,

You do have to have a vSAN Witness Host (could be our free appliance) for 2 Node vSAN.

Jase McCarty

March 14th, 2018

Valter,

You can use the data node vmk0 for Witness Traffic, and that would need to communicate with the vSAN Witness’s vmk1 (WitnessPg). If you would like to use the Witness’s vmk0, that’s fine, just untag “vSAN Traffic” on the Witness vmk1 and tag it on the Witness vmk0. (Fully supported)

Jase McCarty

March 14th, 2018

Hey Patrick,

We don’t support that today, but stay tuned!

Jase McCarty

March 14th, 2018

Glad to help!

Jase McCarty

March 14th, 2018

Apologies for the late replies folks, it would appear there is a routing issue for comment moderation.

Jase McCarty

March 14th, 2018

FQ,

You may want to reboot your VCSA before doing any additional troubleshooting.

Run the health check again, and everything should clear up.

Let me know how it goes.

Mike Hoffmann

March 26th, 2018

Thanks for the great article. I am new to vSAN, and having some issues here and there. One thing I noticed, is that when I deploy my witness appliance, it sets up a Portgroup called “witnessPg” under “witnessSwitch.” The VMkernel adapter under this is vmk1. When I run esxcli vsan network list I see that this adapter is set for traffic type VSAN. Is this correct? Or do I also need to set this adapter to witness type traffic?

Jase McCarty

March 26th, 2018

Hey Mike,

In a 2 Node Direct Connect configuration, the vSAN Witness Appliance is still going to have vmk1 (WitnessPg on witnessSwitch) set to “vSAN Traffic”.
The only VMkernel interfaces that will have “Witness Traffic” set, are the alternate VMkernel interfaces (connected to a switch and routing to the vSAN Witness vmk1).
The backend vSAN Node interfaces are still set to “vSAN Traffic”.

I hope this helps.

Philip Irwin

May 25th, 2018

Could the commands in the article be fixed?
This is what you have
To add a new interface with Witness traffic is the type, the command is:

esxcli vsan network ipv4 add -i vmkX -T=witness

We can configure a new interface for vSAN data traffic using this command (rather than using the Web Client)

esxcli vsan network ipv4 add -i vmkX =T=vsan

Should be

esxcli vsan network ipv4 add -i vmkX -T=vsan

Mario

September 12th, 2018

Jase,

Can I use L2 networking for the witness traffic? I know that is not recommenced in a general terms to avoid the potential vSAN traffic on management/witness up-link in case of failure….but for small internal cluster should not matter I think.

This is the official documentation:

*Local Witness with 2 Node Configurations: A typical Option 3 configuration would include the
Witness being remote, but running the vSAN Witness Appliance on alternate infrastructure in the
same site is supported if so desired. In this configuration a Layer 2 configuration would likely be used,
which may not require any static routing. This vSAN Witness Appliance may NOT run on the 2 Node
cluster, but may run on alternate ESXi 5.5 or higher infrastructure.