Using Secondary Management Network for vSphere Replication

I have written a lot about vSphere Replication best practices. This time, I want to share an experience of implementing SRM and vSphere Replication in a brownfield virtual infrastructure.

As we all know, vSphere Replication uses the management interface for the ESXi servers to send the replication traffic to the DR site vSphere Replication Appliance. It is important that we understand the network flow clearly before diving deeply into configuration of the networks. The diagram below illustrates how the data flows.

Understanding Network FlowLet’s see how the traffic flows in generic sense before we add IP addressing to it:

Changed blocks are captured by the VR Filter on the ESXi server in the primary site.

This data is sent to DR Site VR Appliance using the primary management interface of the ESXi server.

The VR Appliance in the DR Site passes the data to the ESXi servers in the DR Site using the NFC Service.

This data is then written on to the designated DR Site data store.

Note: Just reverse this sequence to do a reverse replication while doing re-protect in SRM.

A Real-Life ScenarioLet’s look at a real life setup and see how this replication will flow. Here’s a quick view of my setup, along with the IP addresses:

Let’s look at each component one by one:

This is the IP address of the vCenter Server. Notice that the IP sub-nets are different in the Primary Site and DR Site.

This is the IP address of the SRM Server. Notice that the IP sub-nets are different in the Primary Site and DR Site, similar to vCenter Server.

The IP address of the VRA server is not in the same range, because you do not want to use same IP segment as the management network. In this case I have a point-to-point connectivity between the site and the IP configured is on the 10.12.12.x sub-net. This is configured on both sites, as the VRA server will receive the traffic from the ESXi servers on this interface. Remember, this appliance will connect on a virtual machine port group.The default gateway for this sub-net is 10.12.12.1 at the Primary Site and 10.11.12.1 at the DR Site.

VMK0 is the primary management network interface. This is used to manage the ESXi servers in the primary site. Notice that ESXi and vCenter are on the same sub-net.

VMK1 is configured for vMotion on a non-routable VLAN, which is why I have a completely different IP segment here.

VMK2 is the third VMKernel interface I have configured. This is to use the point-to-point connectivity for vSphere Replication. I want the vSphere Replication traffic to go out of this VMK interface and reach the vSphere Replication appliance on the DR Site.

One of the most important things to note that in case of ESXi, the Default Gateway will always be defined with VMK0. Hence, all the VMKernel port-groups will have the same default gateway.

The last point is the problem for me. I do not want the vSphere Replication traffic to hit the gateway (172.16.3.1) in the DR site when the traffic is sent to the vSphere Replication appliance in that site. I want it to hit the gateway configured for 10.11.12.x sub-net. To be precise, the default gateway is 10.11.12.1.

Define a Static RouteThis is not possible until you define a static route to force the vSphere Replication Traffic to go through the vSphere Replication Interface (VMK2) and then hit the vSphere Replication appliance on the DR Site with the default gateway. Remember, you will have to just reverse this action and add a static route on the ESXi servers in the DR site for (10.12.12.1) default gateway in the primary site.

Hope this makes things easy for you and allows you to setup vSphere Replication on your preferred network interface.

Sunny Dua is an Enterprise Solution Architect for VMware’s Professional Services Organization, focused on India and SAARC countries. He holds several certifications including VCP, VCAP-DCA, and VCAP-DCD and the VMware vExpert award. Sunny also speaks regularly at VMware User Groups conferences. Most of his ramblings on virtualization and cloud computing can be found on his personal blog: vxpresss.blogspot.com. He tweets as @sunny_dua.

In the add route command, I think the network part should be an IP or a subnet represented with /## bit subnet mask.
Are you sure that you can use /50 as a range? What if it was a range of 24? Would you write it as /24? This would be iterpreted as a subnet mask.

The static route is only for the single IP address of the VR Server. This ensure all replication traffic (to the VR server) is routed out the interface that is on the same network as the gateway that is defined.

In order to implement VMKernel replication adapters, your entire environment has to be on 6.x. You can’t use it with vCenter 6.x and ESXi 5.5 even though vSphere Replication is supported in that configuration.

This architecture should NOT work. It does NOT mention how VR Appliance communicate with VC. Are 172.x and 10.x routable network? Does VR Appliance or VC has dual vNic? VR Appliance 5.x does NOT support multi vNic.