Borg, Omega and Kubernetes

Summary

In this article five Google engineers discuss lessons learned from over a decade of container management at Google such as

the transformation from machine-oriented to application oriented (machine and operation system agnostic) deployment

containers are the only entity supported by Google's infrastructure

log collection, telemetry (cAdvisor) and configuration is done on a per application level

bundling of application containers into pods. A pod may, for instance, contain containers for (i) the main application, (ii) log rotation, etc. This design increases modularity and allows multiple teams to develop different pieces of the service.

Reduction of complexity by standardizing APIs and metadata. Kubernetes for instance provide the following three fields for every object:

ObjectMetadata: information on objects (name, UID, etc.)

Specification: the object's desired state

Status: the object's current state

recommendations for best practices

don't let the container system manage port numbers

provide labels and metadata with the containers

don't expose raw state (e.g. force service access through a centralized API that hides implementation details and provides services for object validation, defaulting and versioning).