PKS NSX-T Home Lab – Part 12: Create a K8s Cluster

December 20, 2018

Intro

In the previous post in this series we configured and deployed PKS. Now that we have a PKS API endpoint, we are going to create a kubernetes cluster.

Create a User

Before we can create a K8s cluster we need to create a user. We create and manage users in PKS with User Account and Authentication (UAA). We use the UAA Command Line Interface (UAAC) to interact with the UAA server. You can either run UAAC commands from the Ops Manager VM or install UAAC on your local workstation or jump etc.

As we are going to be communicating with the PKS API, we need its certificate we generated when performing the install. In Ops Manager, click the PKS tile, then PKS API in the left column of the settings tab. Copy the cert and then on the jump create a new file where the contents are the certificate just copied. Take note of the PKS API FQDN we configured previously.

Next we need to retrieve the PKS UAA admin credentials. In Ops Manager, click the PKS tile, followed by the Credentials tab. Click Link to Credential for Pks Uaa Management Admin Client as seen below. Record the secret returned.

And then assign their scope. Two options are 1) pks.clusters.admin where user can create and access all clusters 2) pks.clusters.manage where the user can create and access only their own clusters.

root@ubuntu-jump:~# uaac member add pks.clusters.admin keith
success

Create a Kubernetes Cluster

Now we have a user created who has permissions to create a kubernetes cluster. But how do we create a cluster? We use the PKS CLI which can be downloaded from PivNet. While there, also download the Kubectl CLI. There are Linux, Windows, MAC flavors of each. Copy both the PKS and Kubectl CLI’s to your jump.

Now we have the PKS CLI on our jump lets login in using the PKS API FQDN, the user we create earlier, and the PKS API cert. Note, you can if wished, skip cert verification by using -k instead of –ca-cert <cert>

Now that we are authenticated, lets create a k8s cluster. Here we are creating a cluster called pks-cluster-1, whose kubernetes API will be available via pks-cluster-1.lab.keithlee.ie and will use the small plan we defined in the previous post which was 1x master and 3x workers by default. The number of workers can be specified by -n, –num-nodes, up to a maximum as defined in the plan, which was 50.

If we look in our vSphere Client when the K8s cluster is being created, we see four new VMs in our Mgmt AZ. These are not our K8s cluster VMs, but actually the BOSH Compilation VMs. BOSH compiles packages on the fly. This can be confirmed by checking the Custom Attributes.

After a while, the pks cluster <cluster-name> command will return that the cluster is created. Note, rather than manually checking you could use the watch command. You will also notice the K8s API IP, 10.0.80.11, for the cluster is from our NSX-T Floating IP pool. If wished, you can create an entry in your DNS for pks-cluster-1.lab.keithlee.ie to resolve to 10.0.80.11

If we look in our vSphere Client, we can now see 4x VMs across our AZs AZ1, AZ2, and AZ3, where the master node is in AZ1 and the 3x worker nodes across the AZs, just like how we defined them in our small plan. We also see the master node IP address is from our Nodes IP Block we defined in NSX-T.

So now we have a K8s cluster, naturally we want to use it! For that we use a tool called kubectl, pronounced by some as kube-cuddle, awww cute! Earlier in this post we downloaded it and added it to our PATH on your ubuntu jump. Before we can issue kubectl command we need the credentials. Credentials can be retrieved using the PKS command pks get-credentials <cluster-name>

root@ubuntu-jump:~# pks get-credentials pks-cluster-1
Fetching credentials for cluster pks-cluster-1.
Context set for cluster pks-cluster-1.
You can now switch between clusters by using:
$kubectl config use-context <cluster-name>

Lets test we can successfully communicate with the K8s cluster with the retrieved credentials by running the kubectl cluster-info command

root@ubuntu-jump:~# kubectl cluster-info
Kubernetes master is running at https://pks-cluster-1.lab.keithlee.ie:8443
Heapster is running at https://pks-cluster-1.lab.keithlee.ie:8443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://pks-cluster-1.lab.keithlee.ie:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubernetes-dashboard is running at https://pks-cluster-1.lab.keithlee.ie:8443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
monitoring-influxdb is running at https://pks-cluster-1.lab.keithlee.ie:8443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

I’m not going to run through all the kubectl commands but another useful command is kubectl get nodes -o wide. Here we can see details of the workers such as the K8s version and the IP address of each worker.

So as you can see, its literally that easy to create a k8s cluster with one simple command.

NSX-T Objects

When we create a k8s cluster using the PKS API, there is lots of “magic” happening in the background. I’m not going to go into detail of what is happening under the covers in this post; you can attend my VMware PKS workshop for that ;). But lets have a look at what objects are created in NSX-T.

In NSX Manager, go Networking > Routers and we can see below we have 6 new tier-1 routers. Prior to K8s cluster creation, we only had T0-LR and t1-pks-mgmt.

The new object names include the UUID of the PKS K8s cluster, 910f1410-9c48-435c-8f8a-a775c6688021, which is the same UUID returned by pks clusters and pks cluster <cluster-name>. The first, lb-pks-<uuid>-cluster-router is a tier-1 for the load balancer. Second, pks-<uuid>-cluster-router is for the k8s node network where the master and workers reside. The remaining 4x tier-1’s are for the 4x default namespaces when a new cluster is created, them being pks-system, kube-system, kube-public, and default. You will also notice that these new routers have a icon beside them, whereas the T0-LR and t1-pks-mgmt don’t, this denotes that they are protected objects created by the Superuser we created previously.

In Networking > Switching, there are also 6x new logical switches, again 1x for load balancer, 1x node network, and 4x for default namespaces.

In Networking > Routing > T0-LR > Services > NAT we see 5x new NAT rules auto created. The rules (2064 to 2075) with Prioirty 1024 below. The first, rule 2064, is for our node network, and the remaining four are for the namespace pod networks

In Networking > Load Balancing we see a new load balancer, again with the UUID of our cluster

The load balancer has 3x virtual servers, again with the cluster UUID making up part of their name. The first with virtual server at the end of its name is a layer 4 virtual server for our k8s master node. You will notice the IP 10.0.80.11 is the same as returned by pks cluster <cluster-name>. The remaining two virtual servers are http and https layer 7 virtual servers.

Post navigation

Your email address will not be published. Required fields are marked *

Comment

Name *

Email *

Website

Notify me of follow-up comments by email.

Notify me of new posts by email.

About The Author

Keith Lee

Keith is a Systems Architect currently working as Staff Training Specialist for VMware on their cloud native apps portfolio including PKS (Pivotal Container Service) and VIC (vSphere Integrated Containers). He has written several whitepapers and reference architectures in the cloud native and data analytics space including most recently PRA (Pivotal Ready Architecture); a reference architecture to run PAS (Pivotal Application Service) and PKS with NSX-T on VxRail HCI.