About upgrading clusters

Supported versions

To find the supported Kubernetes master and node versions for upgrades and
downgrades, run the following command:

gcloud container get-server-config

Downgrading a cluster is not recommended. A cluster can be downgraded to a
patch version older than the current master version. You cannot downgrade a
cluster from one minor version to another. For example, if a cluster is
running GKE 1.11.5, you can downgrade to 1.11.4 if it is still
available, but you cannot downgrade to 1.10.9. Instead, an error like the
following is generated:

ERROR: (gcloud.container.clusters.upgrade) ResponseError: code=400,
message=Master cannot be upgraded to "1.10.9-gke.7": specified version is not
newer than the current version.

To downgrade a cluster to a previous patch version, update the cluster master
version using the gcloud command line tool.

Downgrading a node pool is not possible. Instead, create a new node pool at the
desired version and migrate your workloads to it. If you enable node
auto-upgrade on the node pool, nodes are upgraded to match the master version.

Saving your data to persistent disks

Before upgrading a node pool, you must ensure that any data you wish to keep is
stored in a Pod using persistent volumes which use persistent disks.
Persistent disks are unmounted, rather than erased, during upgrades, and their
data is "handed off" between Pods.

The following restrictions pertain to persistent disks:

The nodes on which Pods are running must be Compute Engine VMs

Those VMs need to be in the same Compute Engine project and zone as the
persistent disk

Upgrading the master

In the following sections, you can substitute the word "upgrading" with
"downgrading". Before downgrading a cluster or a node pool, be sure you
understand the caveats of downgrading clusters.

Automatically upgrading the master

Google upgrades cluster masters automatically. To learn more about automatic
master upgrades, see
Versioning and upgrades.

You can initiate a manual upgrade any time after a new
version becomes available.

Manually upgrade the master

Note: You cannot upgrade your cluster master more than one minor version at a
time. For example, you can upgrade a cluster from version 1.12.x to 1.13.x, but
not directly from 1.11.x to 1.13.x. For more information, refer to
Versioning and Upgrades.

When initiating a cluster master upgrade, you can't modify the cluster's
configuration for several minutes, until the control plane is accessible again.
If you need to prevent downtime during master upgrades, consider using a
regional cluster

You can manually upgrade your masters using the GCP Console or the
gcloud command-line tool. Once you've upgraded your cluster's master, you can
upgrade the nodes to the same version. If node auto-upgrade is enabled,
Google upgrades your nodes after upgrading the control plane.

gcloud

To upgrade your cluster master's version, first run the following command
to see the available versions:

Click the arrow at the top of the screen to go back to the cluster
overview page.

Upgrading nodes

When you upgrade cluster nodes, GKE stops scheduling new Pods
onto the node being upgraded and attempts to reschedule those Pods onto
different nodes. Replacement nodes are recreated with the same name as their
predecessors, just like any other event that recreates a node.

Note: During automatic or manual node upgrades,
PodDisruptionBudgets (PDBs) are
respected for a maximum of 1 hour. If Pods running on a node cannot be scheduled
onto new nodes within 1 hour, the upgrade is initiated, regardless.

For more information about what to expect during node termination in general,
see the topic about Pods.

The upgrade is only complete when all nodes have been recreated
and the cluster is in the desired state. When a newly-upgraded node registers
with the master, GKE marks the nodes as schedulable.

Node auto-upgrade

Node auto-upgrade is enabled by default for new clusters. When node auto-upgrade
is enabled, Google ensures that the version of GKE on your
nodes is kept up to date with the version running on the master.

Manually upgrade a node

You can only upgrade a node pool version to match the version of the master or
to a previous version that is still available and is compatible with the master.
Kubernetes guarantees that masters are compatible with nodes up to two minor
versions older than the master. For example, Kubernetes 1.13 masters are
compatible with Kubernetes 1.11 nodes.

Note: Upgrading a node pool may disrupt workloads running in that node pool. To
avoid this, you can create a new node pool with the desired version and migrate
the workload. Then delete the old node pool.

You cannot downgrade a node pool. Instead, create a new node pool using the older
version and migrate workloads to the new node pool.

You can manually upgrade your nodes to a version compatible with the master, using
the Google Cloud Platform Console or the gcloud command-line tool.

gcloud

The following command upgrades your nodes to the version that your master is
running:

Rolling back a node upgrade

You can roll back node pools that failed to upgrade, or whose upgrades were
canceled, to their previous version of Kubernetes. You cannot roll back node
pools once they have been successfully upgraded. Nodes that have not started an
upgrade are unaffected.