Kubernetes and the "Platform Engineer"

One of Oracle's top 10 predictions for developers in 2019 was that developers will need to partner with a platform engineer, which will emerge as a key new role for cloud native development. Recent conversations with Enterprise customers have reinforced this, and it is becoming clear that a separation of concerns is emerging for those delivering production applications on top of Kubernetes infrastructure. The application developers building the containerized apps driven by business requirements, and the “Platform Engineers”, owning and running the supporting Kubernetes infrastructure, and platform components. For those familiar with DevOps, SRE (pick your term) – this is arguably nothing new, but the consolidation of these teams around the Kubernetes API is leading to something altogether different. In short, the Kubernetes YAML file (via the Kubernetes API) is becoming the contract or hand-off between application developers and the platform team (or more succinctly between dev and ops).

In the beginning, there was PaaS

Well, actually there was infrastructure! – but for application developers, there was an awful lot of pieces to assemble (compute, network, storage) to deliver an application. Technologies like Virtualization and Infrastructure as Code (Terraform et al) made it easier to automate the infrastructure part, but still, a lot of moving parts. Early PaaS (Platform as a Service) pioneers, recognizing this complexity for developers, created (PaaS) platforms, abstracting away much of the infrastructure (and complexity), albeit for a very targeted (or “opinionated”) set of application use cases or patterns – which is fine if your application fits into that pattern, but if not, you are back to dealing with infrastructure.

Then Came CaaS

Following the success of Container technology popularized in recent years by Docker, so called “Containers as a Service” offerings emerged a few years back, sitting somewhere between IaaS and PaaS, CaaS services abstract some of the complexity of dealing with raw infrastructure, allowing teams to deploy and operate container based applications without having to build, setup and maintain their own container orchestration tooling and supporting infrastructure.

The emergence of CaaS also coincided largely with the rise of Kubernetes as the de facto standard in container orchestration. The majority of CaaS offerings today are managed Kubernetes offerings (not all offerings are created equal though, see The Journey to Enterprise Managed Kubernetes for more details). As discussed previously, Kubernetes has essentially become the new Operating System for the Cloud, and arguably the modern application server, as Kubernetes continues to move up the stack. At a practical level, this means that in addition to the benefits of a CaaS described above, customers benefit from standardization, and portability of their container applications across multiple cloud providers and on-prem (assuming those providers adhere to and are conformant with upstream Kubernetes).

Build your Own PaaS?

Despite CaaS and the standardization of Kubernetes for delivering these, there is still a lot of potential complexity for developers. With “complexity”, “cultural changes” and “lack of training” recently cited as some of the most significant inhibitors to container and Kubernetes adoption, we can see there’s still work to do. An interesting talk at KubeCon Seattle played on this with the title: “Kubernetes is Not for Developers and Other Things the Hype Never Told You”.

Enter the platform engineer. Kubernetes is broad and deep, and only a subset of it ultimately needs be exposed to end developers in many cases. As an enterprise that wants to offer a modern container platform to its developers, there are a lot of common elements/tooling that every end developer/application team consuming the platform shouldn’t have to reinvent. Examples include (but are not limited to): monitoring, logging, service mesh, secure communication/TLS, ingress controllers, network policies, admission controllers etc… In addition to common services being presented to developers, the platform engineer can even extend Kubernetes (via extension APIs), with things like the Service Catalog/Open Service Broker to facilitate easier integration for developers with other existing cloud services, or by providing Kubernetes Operators, helpers essentially that developers can consume for creating (stateful) services in their clusters (see examples here and here).

The platform engineer then in essence, has an opportunity to carve out the right cross section of Kubernetes (hence build your own PaaS) for the business, both in terms of the services that are exposed to developers to promote reuse, but also in enforcement of business policy (security and compliance).

Platform As Code

And the fact that you can leverage the same Kubernetes API or CLI (“Kubectl”) and deployment (YAML) file to drive the above platform, has led some to talk about the approach as “Platform as code” – essentially an evolution of Infrastructure as Code, but in this case, native Kubernetes interfaces are driving the entire creation of a complete Kubernetes based application platform for enterprise consumption.

The platform engineer and the developer now have a clear separation of concerns (with the appropriate Kubernetes RBAC roles and role bindings in place!). The platform engineer can check the complete definition of the platform described above into source control. Similarly, the developer consuming the platform, checks their Kubernetes application definition into source control – and the Kubernetes YAML file/definition becomes the contract (and enforcement point) between the developer and platform engineer

Platform engineers ideally have a strong background in infrastructure software, networking and systems administration. Essentially, they are working on the (Kubernetes) platform to deliver a product/service to (and in close collaboration with) end development teams.

In the future, we would expect there to be additional work in the community around both sides of this contract. Both for developers, and how they can discover what common services are provided by the platform being offered, and for platform engineers in how they can provide (and enforce) a clear contract to their development team customers.