Random thoughts and technical bits

What networks does PKS create inside each K8 cluster?

Pivotal Container Service (PKS) provides desired state management for Kubernettes clusters. ​​​​ It radically simplifies many operational aspects of running K8 in production. ​​ Out of the box K8 struggles to provide secure multi-tenant ingress to clusters. ​​​​ With PKS this gap is filled by tight integration with NSX-T. ​​​​ A simple command can be used to create the K8 cluster API and worker nodes with all required networking. ​​​​ I wanted to provide a deeper dive into the networks that are created when you issue the following command in PKS:

This command tells PKS to create a new K8 cluster with the name K8s-1 with an external name of k8s_1 using the small plan. ​​​​ My plans are defined are part of the PKS install and resizable / adjustable at any time. ​​​​ The plan​​ denotes​​ the following things:

How many Master/ETCD nodes and sizing

How many worker nodes and sizing

My command produces the following details:

Once you issue the command the ETCD and worker nodes are deployed along with all required networking. ​​​​ I’ll go into a deeper dive of NSX-T PKS routing in another post but simply put several networks are created during the cluster creation. ​​​​ All the networks include the clusters UUID so it’s simple to track. ​​ Searching in NSX-T for the UUID provided the following information:

As you can see the operation has created​​ several​​ logical routers to handle PKS traffic including:

T1 Router for K8 master node

T1 Router for the load balancer

Four T1 routers one per namespace​​ (found using: kubectl get ns -o wide)

To locate what is running inside each namespace you can run (kubectl get pods –all-namespaces)

When you add additional namespaces to the K8 cluster additional T1 routers are deployed. ​​​​ All of this is manual with traditional K8 clusters but with PKS it’s automatically handled​​ and integrated. ​​ ​​​​

About Author

Joseph Griffiths is a virtualization focused solutions architect who works with complex cloud based solutions. He currently holds many IT certifications including VMware VCDX-DCV and VCDX-CMA #143. This blog represents his random technical notes and thoughts. The thoughts expressed here do not reflect Joseph’s current employer in anyway. You can follow Joseph on Twitter @Gortees