VMware vCloud 5.1 Networking for dummies

In April of this year I finally was able to take the “VMware vCloud: Deploy and Manage the VMware Cloud” course lead by Mr. Eric Sloof himself. The training was based on vCloud Director 1.5 and I learned quite a lot and really thought I had this vCloud networking thing in my fingers. The exercises in class where easy to complete and made sense to me. Ok, Ok, I do admit I was sometimes struggling with the fact that I knew what I wanted to configure but didn’t always know where to configure it. Like creating a NAT rule to a static IP or opening some firewall ports. But in the end, it all worked. At home I didn’t get the time to play with vCloud and when I finally did get time, vCloud Director 5.1 was just released.

I completely rebuilt my home lab, add a new physical switch and two new hosts with 32GB RAM each and started playing with vCloud Director. Boy, was I confused on the networking part. Installing vCenter 5.1, ESXi 5.1 and vCloud 5.1 was a breeze, but getting any vCloud deployed VM to talk to the internet was pretty hard. It isn’t once you know how to, but when trying to figure out how you’ll notice that the VMware vCloud Admin Guide and Installation Guide don’t explain the networking concept very well. Therefore I decided to write this blogpost, mostly for myself but hopefully also for you, to better understand basic networking in vCloud and how to configure it.

Warning: Keep in mind that I’m also still learning and I might sometimes explain things a bit too simplistic, but I try to be as accurate as possible. Use this post to get a quick-start on vCloud networking.

The Plan

My vSphere environment consists of 5 ESXi 5.1 hosts. Three small hosts all have 8GB RAM and run all the VMs that make the vCloud, my SQL Server, my Domain Controller, etc. This is my management cluster. The two ESXi hosts with 32GB each only run VMs and vApps from vCloud. My home network is the 192.168.0.0/24 range and the default gateway (192.168.0.254) is my firewall to the big bad internet. The management cluster is not using any VLAN tagging and for the vCloud Environment I now have reserved a set of VLANs (200-205) named dvPG-vCloud-200 through dvPG-vCloud-205. Also there is a dvPG-ExternalCloud without VLAN ID, which will be used to connect the external network.

You can’t start a good network design if you don’t have a plan. I created a network design in which a number of different configurations are present and it will be my challenge to make them work and understand how they work. Walk with me on my little journey. When you look at the network design I made, you’ll see that the first task is going to be simple:

Create a vApp of two VMs

Connect them directly to an organizational network which will have the range: 192.168.10.0/24

Use NAT translation to give them internet access

The VMs will get their IP address through DHCP

The green arrows show that vm01 should be able to ping vm02, the gateway, the DNS server on the home network and get internet access.

Create an external network

The next screenshot shows the vCloud ClusterI have running and as you can see, there are no VMs setup yet and only the basic System Resource pools has been created by vCloud Director.

I have already created a Provider VDC named “Provider-01″. From the vCloud “Home” page I now select “Create External Networks”. With this step I create an external network which is my connection between my vCloud and the world outside. Click “Create External Networks” and the wizard will start. Select the vCenter that hosts my vCloud and select the vSphere Network (port group) that will be used to connect to the External Network. In my case this is dvPG-ExternalCloud with VLAN 0. Click next to move to the “Configure External Network” page. Click Add to enter the details of your external network. Remember, the external network is the network that is connecting my “home network” so the DNS and Gateway servers are the same I would use for all systems in my home network (not part of the vCloud). Also you need to enter a Static IP pool. This IP pool is a range of IP addresses that will be used by vCloud to directly connect to the external network. In my case I reserved the range 192.168.0.161 to 192.168.0.199. Click OK after you have completed the screen. Give the external network a name, in my case “vCloud-External” and click finish.

Create a network pool

When vCloud deploys vApps it uses VLANs to keep all the different vApps separated from each other at the networking level. For this first a set of VLANs has to be assigned (a pool) which vCloud can use. There are three types of network pools which I’m not going to explain any further. For my home lab I used the very simple “vSphere Port Group backed” network pool which will just take a set of pre-defined port groups from your vCenter environment and use them when deploying a vApp or OrgNetwork. In vCenter I created the following port groups: dvPG-vCloud-200 through dvPG-vCloud-205 holding the VLAN IDs 200 – 205.

On the “Home” screen, select “Create a network pool”. The wizard will start and present three types of Network Pools, select the “vSphere Port group backed”, next select vCenter and in the next screen select the Port Groups that need to be added. In my case dvPG-vCloud-200 through dvPG-vCloud-205. The network pool name will be “dvPG 200-205″.

Create a new organization

To be able to deploy VMs we need an organization. From the “Home” screen I select “Create a new organization” and the wizard will start. Enter the organization name ( “HomeLab” ) enter the Organization full name: “Home Lab”. Next select “Do not use LDAP” and then add a new local user and give him admin rights. I created the user vCloudUser. Walk through the rest of the wizard and select the options you prefer. When you’re done you have a new organization “HomeLab”.

Allocate Resources to an organization

With allocating resources to an organization you’re actually creating an Organization VDC. Again there is a wizard to walk through. The first steps are not that interesting for this post and I’ll start with the step “Select Network Pool & Services”. This is where you tell vCloud which networks (VLANs) it is allowed to use. Select the network pool created earlier (dvPG 200-205) and set the quota to 6. Click next to “Configure Edge Gateway”. Select “Create a new edge gateway” and name it “Edge-HomeLab”. Choose configuration option “Compact”, leave “Enable High Availability” unchecked. Select “Configure IP settings” and select “Sub-Allocate IP Pools”.

Why Select the “Sub-Allocate IP Pools”? That wasn’t clear to me at first either, but it seems that for NAT connections, the Edge Gateway needs an EXTRA (outside) IP address, it will not use the IP that was assigned to the outside nic of the Edge Gateway. This I think has been one of the things I struggled with the most, because I expected the Edge Gateway to just use its own outside IP and when configuring NAT, there is no warning that you also need to have a “Sub-Allocate IP pool” to make it work.

Back to the wizard. “Sub-Allocate IP pools” has been selected and now press next to “Configure External Networks”. There is only one thing to do here, select the External network this Organization VDC has access to, in my case “vCloud-External”. Also select “Default Gateway for DNS relay”. Press next to go to “Configure IP Settings”. Here you’ll see the external network, what subnet it connects to and the IP address assignment which defaults to “Auto”. Just to see what is assigned, click “Change IP assignment”. Here you’ll see the “Allocated IP address range” we defined earlier ( 192.168.0.161 through 199). Cancel the screen and continue the wizard to “Sub-Allocate IP Pools”.

As explained above, the Sub-Allocate IP pool is used for NAT translation on the Edge Gateway. In this screen you select a sub set of IP addresses from the IP Pool that is already assigned to the external network. In my case I used only 1 IP address: 192.168.0.162. Click Next for the last step in this wizard to create an Organizational VDC network. As seen in the Network plan, I want to use the IP range 192.168.10.0/24 for this organizational network. Select “Create a network for this virtual datacenter connected to this new edge gateway” and enter the network name. In my case this is: “OrgNet-HomeLab”. Now enter the Gateway address: 192.168.10.254 with network mask: 255.255.255.0. Select “Use Gateway DNS” and add a Static IP Pool of 192.168.10.50 through 192.168.10.100. Click Next for the last screen and name the Organization VDC: “OrgVDC-HomeLab”. After clicking finish watch your vCenter and see all the actions that are conducted by vCloud.

Deploying a vApp

To be able to deploy a vApp, an organizational catalog is needed first. From the “Home” screen click “Add a catalog to an organization” and walk through the wizard to create a catalog for the “HomeLab” organization and name it “CatHomeLab”. Choose “Publish to all organizations” just for ease of use. After the wizard finishes go to “Manage Organizations”. In the left pane you’ll see the “Manage & Monitor” section with the “Organizations” selected. On the right hand side select the “HomeLab” organization and open it. You’ll see and extra tab for the organization and the available catalogs. Open the “CatHomeLab” that was just created.

After opening the CatHomeLab, you’ll see the vApp Templates tab where you can import a new VM. I have a small Ubuntu VM that I use for testing which has the VMware Tools installed and uses DHCP to get an IP address. I import this VM from my vSphere environment. After the VM has been imported, go to the My Cloud tab and click “vApps” on the left hand side. Click “New vApp” and walk through the wizard. Name the vApp “vApp-01″, give it a runtime and storage lease, select the Ubuntu VM and add it twice! Click next to move to “Configure Resources”. Name the VMs “Ubuntu 01″ and “Ubuntu 02″, select the default storage profile. Next go to configure “Virtual Machines” and set the Computer Names to “Ubuntu 01″ and “Ubuntu 02″. In the column “Network” select the “OrgNet-HomeLab” network and set the IP assignment to “DHCP”. Since we’ve not yet created a DHCP pool we’ll do that after the wizard finishes. Click Next for the “Configure Networking” page, leave “Fence vApp” unselected. Click Finish. You’ll now see how the vApp-01 is being created.

Let’s not forget to create a DHCP pool. Since the vApp will be connected to the OrgNet-HomeLab network, the DHCP range should be enabled for that network. On the “HomeLab” tab, go to the “Administration” section, click “Virtual Datacenters”, open the “OrgVDC-HomeLab” on the right hand side and then choose the “Org VDC Networks” tab. You’ll now see the “OrgNet-HomeLab” network. Click right, select “Configure Services” and on the “DHCP” tab enable DHCP and add the range: 192.168.10.150-192.168.10.200.

Since one of the tests I wanted to perform is to show that the VMs can get onto the internet through NAT, a NAT rule has to be created. Go to the second tab “NAT” and create a SNAT (Source NAT) rule. For “Applied On” select the “vCloud-External” network. For the Original Internal Source IP / Range select “192.168.10.0/24″, the whole subnet will be NAT-ed when going to the internet. For the “Translated (External) Source IP range” enter the sub allocated IP address 192.168.0.162. After setting the SNAT rule, switch to the firewall tab and disable the firewall. We don’t want to make it too hard the first time. Click OK to close the “Configure Services” window.

Now it is time to start the vApp and check if everything is working. Go to the “HomeLab” tab, select “vApps”, select “vApp-01″ and press “Start”. In the vCenter interface you’ll now see how the VMs are powered on. If everything goes well you can see the IP address of the VMs through the VMware Tools in vCenter (if you have a VM with VMware Tools installed). This is also visible in the vCloud web interface by opening the “vApp-01″ and click on the “virtual machines” tab. The IP addresses 192.168.10.150 and 192.168.10.151 should be shown. But the ultimate test is of course to log into the VMs using the vSphere or vCloud Console and perform some simple tests:

Logon to the first VM and check what IP address was assigned: ifconfig

Ping the other VM: ping 192.168.10.151

Ping your default gateway: ping 192.168.10.254

Ping any other system in your home network: ping 192.168.0.11

Test DNS nameresolution: nslookup www.GabesVirtualWorld.com (Remember name resolution is done by the Gateway)

Download a webpage: wget http://www.GabesVirtualWorld.com

And we’re done!!! Going back to the network design, we can now complete the design with specific details from this vCloud. In the next post I will add some vApp networks.

Post navigation

This is a great overview. I would also like to see an example of a vApp connected via a routed network instead of the direct connection in your example. Specifically how do I add the snats to get to the internet from the vApp.

http://www.GabesVirtualWorld.com Gabrie van Zanten

Working on that for myself as well Hope to publish in a few days.

Sjaak Trekhaak

why did you use a VLAN backed pool instead of VCD-NI or VXLAN? Those two are more frequently used

http://www.GabesVirtualWorld.com Gabrie van Zanten

I started using those but had some problems that I couldn’t locate. Maybe it was in my physical switches, maybe in how I made the vApp network config. To rule them all out, I went for the most basic option.

My problem is that I can only make use of the first subnet out of the external network from the pVDC when connecting my vApp to the Org VDC network. If I enter an IP of another subnet, I get an error “The following IP addresses are not in the subnet range”.

Should I not be able to make use of the other subnets specified at the external (PVDC) level?

http://www.GabesVirtualWorld.com Gabrie van Zanten

I don’t know how to solve this, since I’ve never tried adding multiple subnets to the external network so my guess is as good as yours, but…. did you also add an IP address for each subnet to the EdgeGateway? Did you add a route for each subnet to the EdgeGateway. Have you added SNAT rules for each subnet? When adding SNAT rules be sure to choose the EXTERNAL interface like I did in my example. That is one of the most common made mistakes.

tommygee

I had a feeling that you would throw something Edge Gateway related at me :). No, we are not using the Edge Gateway as we do not want to use NAT in this scenario. The clients that will be living in this new Org VDC currently have direct public access and we need to keep them that way. I haven’t had much experience with the Edge itself but from my readings, there is no real way to do any “transparent” filtering without NAT. Feel free to correct me if I am wrong.

I have found in the vCloud documentation that there is only one child network pulled from the provider VDC external network that it is directly connected, I wish there was a way to pull all the external networks specified in the “network specification” tab from the provider VDCs external network.

Create an Organization vDC Network With a Direct Connection
An organization vDC network with a direct connection provides direct layer 2 connectivity to machines and networks outside of the organization vDC. Machines outside of this organization vDC can connect to machines within the organization vDC directly.An organization vDC network with a direct connection is configured as a child network of one of the external networks provisioned to the cloud by the system administrator.

http://www.GabesVirtualWorld.com Gabrie van Zanten

Hmmm Would you need to create multiple external networks? But I guess then you would run into problems with connecting the one Org network to all those external networks.

Tommygee

I tried that route but with no luck. I have engaged VMware support. I will let you know the outcome.

Tommygee

Well VMware did provide us with a big line of code for the work around (API).However, in the end, it is not going to work for us. For anyone else looking for a resolution here, expect it in 5.1.2.

http://www.GabesVirtualWorld.com Gabrie van Zanten

Thank you for reporting back!

Raj

I did exactly like you said, but i am unable to ping the default gateway from the vApp ( windows 2008 ) firewall is disabled

http://www.GabesVirtualWorld.com Gabrie van Zanten

See private e-mail.

CMAC

Three days I was stumped on why my VMs in my OrgVDC network could not talk out…….until I read this post! Thank you

http://www.GabesVirtualWorld.com Gabrie van Zanten

Great to hear. Thanks!

http://twitter.com/forbesguthrie Forbes Guthrie

Where did you get the Visio Stencil for those diagrams? I like.

http://www.GabesVirtualWorld.com Gabrie van Zanten

Visio 2013 has this as standard objects

http://twitter.com/forbesguthrie Forbes Guthrie

Time for upgrade from Visio 2010 then

vcloud new bie

This is an excellent post and it helped me alot in understanding the concepts.

I’m running into an issue and wondering if you can help out with it. I’ve setup exactly what you have with different subnets and external network etc. but the design is same as yours. The problem is that my vApp VM’s are not getting the IP address from the DHCP (vShield Edge GW) when the vApp VM resides on a different ESX host. It works fine when the vApp VM and edge appliance are on the same ESX host. For example, if edge is running on ESX1 and I spin-up two vApp VM’s from the vApp template then if one (let’s say VM2) lands on ESX2 then only VM1 (which is running on ESX1) will get the DHCP base address and VM2 will not be getting it. If I do a vMotion of the VM2 from the vSphere client to ESX1 then it gets the address just fine.

Any ideas where to look to investigate this problem?

http://www.GabesVirtualWorld.com Gabrie van Zanten

I’ve send you a private e-mail. Check your inbox.

vcloud new bie

Thanks to your directions I was able to resolve the issue which was related to the networking.

http://twitter.com/antimike Mike Way

Just curious as to whether or not your Org Network GW is an actual gateway/firewall?

Here is my real problem…I have to provide a default gateway where none truly exists. The client wants to have a “fenced” ala Lab Manager terminology where the VMs located in the Org Network but an actual gateway pingable or otherwise does not exist in this environment.

Would I just need to turn up a VM to do the routing purposes for me i.e. Linux box?

http://www.GabesVirtualWorld.com Gabrie van Zanten

In this example the Org Gateway address is: 192.168.10.254. This gateway will function as a true gateway and firewall. Depending on how fenced the VMs should be, you could attach all VMs (in a vAPP) directly on the org network or create a vApp network per vApp that has its own firewall / gateway. If you want I can help you by e-mail or skype. Sent me an email at: thegabeman on my gmail.com account.

http://twitter.com/bjornbats bjorn

can you tell what the ip nummers are of your vshield and vcloud server during the setup. its not in the picture.

http://www.GabesVirtualWorld.com Gabrie van Zanten

The vShield Manager and the vCloud Cells have completely different IP’s and are not important for the example in this network design. If you need more help, just send me an email.

a.d.

What are the use cases for NAT-routed networks ? Are they only good for setting up an isolated network segment that also needs access to the internet ? Can they be configured to have two-way communication with the rest of the virtual datacenter ?

http://www.GabesVirtualWorld.com Gabrie van Zanten

The best use case indeed is further isolation of the VM in the vApp from other vApps.

Please also see my other posts on vCloud Networking

NewBe2

Hi Gabri,
I encountered the same problem as “vcloud new bie”…The problem is that my vApp VM’s are not getting the IP address from the DHCP (vShield Edge GW) when the vApp VM resides on a different ESX host. Can you help with how to resolve?

http://www.GabesVirtualWorld.com Gabrie van Zanten

For “vcloud newbie” the issue was OUTSIDE of the vCloud. It was in his vSphere environment. How to test if your vSphere is ok:
– in vCenter Server ( NOT IN VCLOUD)
– create two very simple VMs (Windows or Linux) on the SAME ESXi host
– Give them both a fixed IP
– connect them to a VLAN (portgroup) and have them ping each other
– VMotion one VM to different ESXi host and see if the ping still works

If the ping doesn’t work, your issue is in the physical or in ESXi network, but not in vCloud.

Test this for each VLAN you are using in the vCloud (if possible)

Let me know if that worked for you

cloud

hi can you explain the design of vCenter and vCloud Director that you used to deploy this lab thanks

http://www.GabesVirtualWorld.com Gabrie van Zanten

What would you like to know? I’ll send you an e-mail

Angela

Hi Gabrie, do you have to deploy an Edge gateway for each organisation vDC? I want a client I deploy to use their own Juniper firewall pair as a DG for their virtual servers. If I do have to have a Edge gateway can it just perform routing? Thanks

http://www.GabesVirtualWorld.com Gabrie van Zanten

Yes you can have it perform routing only or even completely bypass the Edge by connecting vApps directly to the External network. Maybe I should write a blogpost on that sometime.

Nishanth D Rao

Hi Gabri,
I followed the same setup with a small deviation that Ip is not allocated from DHCP but statically configured (did this because i had issues with assigning ip from dhcp). I am not able to ping either the gateway address or the other vm. Can you please help me here

http://www.GabesVirtualWorld.com Gabrie van Zanten

See my private email.

Rohit Singh

Hi Gabrie , I am currently using Vcloud director 5.1 set up. I am working on the similar set up as mentioned above. I am applying 2 firewall rules for ICMP with source as any and destination as internal ip as 192.x.x.x.and default rule allowing internal to external traffic.
I have already added SNAT 10.x.x.x to 192.x.x.x.and DNAT rules vice versa allowing any protocol, port. The issue, I am facing is i am not able to ping the vm using public ip which is 10.x.x.x from the external machine, however when i change the destination ip in firewall rule to 10.x.x.x the traffic ping works. Am i missing something or is this the default behaviour. Note I have added this rule on external network via edge gateway

http://www.GabesVirtualWorld.com Gabrie van Zanten

I won’t be able to respond for the next two weeks Please email me on my GMail account (thegabeman).