This upgrade guide is intended for Cilium 1.0 or later running on Kubernetes.
It is assuming that Cilium has been deployed using standard procedures as
described in the Deployment. If you have installed Cilium
using the guide Requirements, then this is automatically the case.

When rolling out an upgrade with Kubernetes, Kubernetes will first terminate the
pod followed by pulling the new image version and then finally spin up the new
image. In order to reduce the downtime of the agent, the new image version can
be pre-pulled. It also verifies that the new image version can be pulled and
avoids ErrImagePull errors during the rollout.

Micro versions within a particular minor version, e.g. 1.2.x -> 1.2.y, are
always 100% compatible for both up- and downgrades. Upgrading or downgrading is
as simple as changing the image tag version in the DaemonSet file:

Kubernetes will automatically restart all Cilium according to the
UpgradeStrategy specified in the DaemonSet.

Note

Direct version upgrade between minor versions is not recommended as RBAC
and DaemonSet definitions are subject to change between minor versions.
See Upgrading Minor Versions for instructions on how to up or downgrade between
different minor versions.

When upgrading from one minor release to another minor release, for example 1.x
to 1.y, it is recommended to first upgrade to the latest micro release
as documented in (Upgrading Micro Versions). This ensures that downgrading by rolling back
on a failed minor release upgrade is always possible and seamless.

The configuration of Cilium is stored in a ConfigMap called
cilium-config. The format is compatible between minor releases so
configuration parameters are automatically preserved across upgrades. However,
new minor releases may introduce new functionality that require opt-in via the
ConfigMap. Refer to the Version Specific Notes for a list of new
configuration options for each minor version.

Refer to the section Upgrading a ConfigMap for instructions on how to
upgrade a ConfigMap to the latest template while preserving your
configuration parameters.

As minor versions typically introduce new functionality which require changes
to the DaemonSet and RBAC definitions, the YAML definitions have to be
upgraded. The following links refer to version specific DaemonSet files which
automatically

Below we will show examples of how Cilium should be upgraded using Kubernetes
rolling upgrade functionality in order to preserve any existing Cilium
configuration changes (e.g., etc configuration) and minimize network
disruptions for running workloads. These instructions upgrade Cilium to version
“v1.4” by updating the ConfigMap, RBAC rules and
DaemonSet files separately. Rather than installing all configuration in one
cilium.yaml file, which could override any custom ConfigMap
configuration, installing these files separately allows upgrade to be staged
and for user configuration to not be affected by the upgrade.

Occasionally, it may be necessary to undo the rollout because a step was missed
or something went wrong during upgrade. To undo the rollout, change the image
tag back to the previous version or undo the rollout using kubectl:

$ kubectl rollout undo daemonset/cilium -n kube-system

This will revert the latest changes to the Cilium DaemonSet and return
Cilium to the state it was in prior to the upgrade.

Note

When rolling back after new features of the new minor version have already
been consumed, consult an eventual existing downgrade section in the
Version Specific Notes to check and prepare for incompatible feature use
before downgrading/rolling back. This step is only required after new
functionality introduced in the new minor version has already been
explicitly used by importing policy or by opting into new features via the
ConfigMap.

This section documents the specific steps required for upgrading from one
version of Cilium to another version of Cilium. There are particular version
transitions which are suggested by the Cilium developers to avoid known issues
during upgrade, then subsequently there are sections for specific upgrade
transitions, ordered by version.

The table below lists suggested upgrade transitions, from a specified current
version running in a cluster to a specified target version. If a specific
combination is not listed in the table below, then it may not be safe. In that
case, consider staging the upgrade, for example upgrading from 1.1.x to the
latest 1.1.y release before subsequently upgrading to 1.2.z.

Current version

Target version

DaemonSet upgrade

L3 impact

L7 impact

1.0.x

1.1.y

Required

N/A

Clients must reconnect[1]

1.1.x

1.2.y

Required

Temporary disruption[2]

Clients must reconnect[1]

1.2.x

1.3.y

Required

Minimal to None

Clients must reconnect[1]

>=1.2.5

1.4.y

Required

Minimal to None

Clients must reconnect[1]

Annotations:

Clients must reconnect: Any traffic flowing via a proxy (for example,
because an L7 policy is in place) will be disrupted during upgrade.
Endpoints communicating via the proxy must reconnect to re-establish
connections.

Temporary disruption: All traffic may be temporarily disrupted during
upgrade. Connections should successfully re-establish without requiring
clients to reconnect.

The DaemonSet now uses dnsPolicy:ClusterFirstWithHostNet in order for
Cilium to look up Kubernetes service names via DNS. This in turn requires
the cluster to run a cluster DNS such as kube-dns or CoreDNS. If you are not
running cluster DNS, remove the dnsPolicy field. This will mean that you
cannot use the etcd-operator.
More details can be found in the kube-dns section.

preallocate-bpf-maps: If true, reduce per-packet latency at the expense
of up-front memory allocation for entries in BPF maps. If this value is
modified, then during the next Cilium startup the restore of existing
endpoints and tracking of ongoing connections may be disrupted. This may
lead to policy drops or a change in loadbalancing decisions for a
connection for some time. Endpoints may need to be recreated to restore
connectivity. If this option is set to false during an upgrade to 1.4.0
or later, then it may cause one-time disruptions during the upgrade.

auto-direct-node-routes: If true, then enable automatic L2 routing
between nodes. This is useful when running in direct routing mode and can
be used as an alternative to running a routing daemon. Routes to other
Cilium managed nodes will then be installed on automatically.

install-iptables-rules: If set to false then Cilium will not
install any iptables rules which are mainly for interaction with
kube-proxy. By default it is set to true.

masquerade: The agent can optionally be set up for masquerading all
network traffic leaving the main networking device if masquerade is
set to true. By default it is set to false.

datapath-mode: Cilium can operate in two different datapath modes,
that is, either based upon veth devices (default) or ipvlan
devices (beta). The latter requires an additional setting to specify
the ipvlan master device.

New ipvlan-specific CNI integration mode options (beta):

ipvlan-master-device: When running Cilium in ipvlan datapath mode,
an ipvlan master device must be specified. This is typically pointing
to a networking device that is facing the external network. Be aware
that this will be used by all nodes, so it is required that the device
name is consistent on all nodes where this is going to be deployed.

New flannel-specific CNI integration mode options (beta):

flannel-master-device: When running Cilium with policy enforcement
enabled on top of Flannel, the BPF programs will be installed on the
network interface specified in this option and on each network interface
belonging to a pod.

flannel-uninstall-on-exit: If flannel-master-device is specified,
this determines whether Cilium should remove BPF programs from the master
device and interfaces belonging to pods when the Cilium DaemonSet is
deleted. If true, Cilium will remove programs from the pods.

flannel-manage-existing-containers: On startup, install a BPF
programs to allow for policy enforcement on pods that are currently
managed by Flannel. This also requires the Cilium DaemonSet to be
running with hostPID:true, which is not enabled by default.

legacy-host-allows-world: This option allowed users to specify Cilium
1.0-style policies that treated traffic that is masqueraded from the outside
world as though it arrived from the local host. As of Cilium 1.4, the option
is disabled by default if not specified in the ConfigMap, and the option is
scheduled to be removed in Cilium 1.5 or later. For more information, see
Traffic from world to endpoints is classified as from host.

If you are running Cilium 1.0.x or 1.1.x, please upgrade to 1.2.x first. It
is also possible to upgrade from 1.0 or 1.1 directly to 1.3 by combining the
upgrade instructions for each minor release. See 1.2 Upgrade Notes.

ct-global-max-entries-tcp/ct-global-max-entries-other: Specifies the
maximum number of connections supported across all endpoints, split by
protocol: tcp or other. One pair of maps uses these values for IPv4
connections, and another pair of maps use these values for IPv6
connections. If these values are modified, then during the next Cilium
startup the tracking of ongoing connections may be disrupted. This may lead
to brief policy drops or a change in loadbalancing decisions for a
connection.

clean-cilium-bpf-state: Similar to clean-cilium-state but only
cleans the BPF state while preserving all other state. Endpoints will
still be restored and IP allocations will prevail but all datapath state
is cleaned when Cilium starts up. Not required for normal operation.

cluster-name: Name of the cluster. Only relevant when building a mesh
of clusters.

cluster-id: Unique ID of the cluster. Must be unique across all
connected clusters and in the range of 1 and 255. Only relevant when
building a mesh of clusters.

monitor-aggregation-level: If you want cilium monitor to aggregate
tracing for packets, set this level to “low”, “medium”, or “maximum”. The
higher the level, the less packets that will be seen in monitor output.

Due to a format change in datapath structures to improve scale, the
connection tracking table will be cleared when the new version starts up for
the first time. This will cause a temporary disruption. All existing
connections are temporarily but should successfully re-establish without
requiring clients to reconnect.

legacy-host-allows-world: In Cilium 1.0, all traffic from the host,
including from local processes and traffic that is masqueraded from the
outside world to the host IP, would be classified as from the host entity
(reserved:host label). Furthermore, to allow Kubernetes agents to
perform health checks over IP into the endpoints, the host is allowed by
default. This means that all traffic from the outside world is also allowed
by default, regardless of security policy. This behavior is continued to
maintain backwards compatible but it can be disabled (recommended) by setting
legacy-host-allows-world to false. See Traffic from world to endpoints is classified as from host for more
details.

Follow the guide in MTU handling behavior change in Cilium 1.1 to update the MTU of existing
endpoints to the new improved model. This step can also be performed after
the upgrade but performing it before the upgrade will guarantee that no
packet loss occurs during the upgrade phase.

Cilium 1.0 by default configured the MTU of all Cilium-related devices and
endpoint devices to 1450 bytes, to guarantee that packets sent from an endpoint
would remain below the MTU of a tunnel. This had the side-effect that when a
Cilium-managed pod made a request to an outside (world) IP, if the response
came back in 1500B chunks, then it would be fragmented when transmitted to the
cilium_host device. These fragments then pass through the Cilium policy
logic. Latter IP fragments would not contain L4 ports, so if any L4 or L4+L7
policy was applied to the destination endpoint, then the fragments would be
dropped. This could cause disruption to network traffic.

Cilium 1.1 fixes the above issue by increasing the MTU of the Cilium-related
devices and endpoint devices to 1500B (or larger based on container runtime
settings), then configuring a route within the endpoint at a lower MTU to
ensure that transmitted packets will fit within tunnel encapsulation. This
addresses the above issue for all new pods.

The MTU for endpoints deployed on Cilium 1.0 must be updated to remediate this
issue. Users are recommended to follow the below upgrade instructions prior to
upgrading to Cilium 1.1 to prepare the endpoints for the new MTU behavior.

The mtu-update tool is provided as a Kubernetes DaemonSet to assist the
live migration of applications from the Cilium 1.0 MTU handling behavior to the
Cilium 1.1 or later MTU handling behavior. To prevent any packet loss during
upgrade, these steps should be followed before upgrading to Cilium 1.1;
however, they are also safe to run after upgrade.

Any rules that are not compatible with 1.0.x must be removed before
downgrade.

Add or update the option clean-cilium-bpf-state to the ConfigMap and
set to true. This will cause BPF maps to be removed during the
downgrade, which avoids bugs such as Issue 5070. As a side effect, any
loadbalancing decisions for active connections will be disrupted during
downgrade. For more information on changing ConfigMap options, see
Upgrading a ConfigMap.

Follow the instructions in the section Upgrading Minor Versions to perform the
downgrade to the latest micro release of the 1.0 series.

Upgrades are designed to have minimal impact on your running deployment.
Networking connectivity, policy enforcement and load balancing will remain
functional in general. The following is a list of operations that will not be
available during the upgrade:

API aware policy rules are enforced in user space proxies and are currently
running as part of the Cilium pod unless Cilium is configured to run in Istio
mode. Upgrading Cilium will cause the proxy to restart which will result in
a connectivity outage and connection to be reset.

Existing policy will remain effective but implementation of new policy rules
will be postponed to after the upgrade has been completed on a particular
node.

Monitoring components such as ciliummonitor will experience a brief
outage while the Cilium pod is restarting. Events are queued up and read
after the upgrade. If the number of events exceeds the event buffer size,
events will be lost.

---apiVersion:v1kind:ConfigMapmetadata:name:cilium-confignamespace:kube-systemdata:# This etcd-config contains the etcd endpoints of your cluster. If you use# TLS please make sure you follow the tutorial in https://cilium.link/etcd-configetcd-config:|----endpoints:-https://cilium-etcd-client.kube-system.svc:2379## In case you want to use TLS in etcd, uncomment the 'ca-file' line# and create a kubernetes secret by following the tutorial in# https://cilium.link/etcd-configca-file:'/var/lib/etcd-secrets/etcd-client-ca.crt'## In case you want client to server authentication, uncomment the following# lines and create a kubernetes secret by following the tutorial in# https://cilium.link/etcd-configkey-file:'/var/lib/etcd-secrets/etcd-client.key'cert-file:'/var/lib/etcd-secrets/etcd-client.crt'# If you want to run cilium in debug mode change this value to truedebug:"false"# Enable IPv4 addressing. If enabled, all endpoints are allocated an IPv4# address.enable-ipv4:"true"# Enable IPv6 addressing. If enabled, all endpoints are allocated an IPv6# address.enable-ipv6:"false"# If a serious issue occurs during Cilium startup, this# invasive option may be set to true to remove all persistent# state. Endpoints will not be restored using knowledge from a# prior Cilium run, so they may receive new IP addresses upon# restart. This also triggers clean-cilium-bpf-state.clean-cilium-state:"false"# If you want to clean cilium BPF state, set this to true;# Removes all BPF maps from the filesystem. Upon restart,# endpoints are restored with the same IP addresses, however# any ongoing connections may be disrupted briefly.# Loadbalancing decisions will be reset, so any ongoing# connections via a service may be loadbalanced to a different# backend after restart.clean-cilium-bpf-state:"false"# In Cilium 1.0, all traffic from the host, including from local processes# and traffic that is masqueraded from the outside world to the host IP,# would be classified as from the host entity (reserved:host label).# Furthermore, to allow Kubernetes agents to perform health checks over IP# into the endpoints, the host is allowed by default. This means that all# traffic from the outside world is also allowed by default, regardless of# security policy.## This option was introduced in Cilium 1.1 to disable this behaviour. It must# be explicitly set to "false" to take effect on Cilium 1.3 or earlier.# Cilium 1.4 sets this to "false" by default if it is not specified in the# ConfigMap.## This option has been deprecated, it will be removed in Cilium 1.5 or later.## For more information, see https://cilium.link/host-vs-world#legacy-host-allows-world: "false"# If you want cilium monitor to aggregate tracing for packets, set this level# to "low", "medium", or "maximum". The higher the level, the less packets# that will be seen in monitor output.monitor-aggregation-level:"none"# ct-global-max-entries-* specifies the maximum number of connections# supported across all endpoints, split by protocol: tcp or other. One pair# of maps uses these values for IPv4 connections, and another pair of maps# use these values for IPv6 connections.## If these values are modified, then during the next Cilium startup the# tracking of ongoing connections may be disrupted. This may lead to brief# policy drops or a change in loadbalancing decisions for a connection.## For users upgrading from Cilium 1.2 or earlier, to minimize disruption# during the upgrade process, comment out these options.ct-global-max-entries-tcp:"524288"ct-global-max-entries-other:"262144"# Pre-allocation of map entries allows per-packet latency to be reduced, at# the expense of up-front memory allocation for the entries in the maps. The# default value below will minimize memory usage in the default installation;# users who are sensitive to latency may consider setting this to "true".## This option was introduced in Cilium 1.4. Cilium 1.3 and earlier ignore# this option and behave as though it is set to "true".## If this value is modified, then during the next Cilium startup the restore# of existing endpoints and tracking of ongoing connections may be disrupted.# This may lead to policy drops or a change in loadbalancing decisions for a# connection for some time. Endpoints may need to be recreated to restore# connectivity.## If this option is set to "false" during an upgrade from 1.3 or earlier to# 1.4 or later, then it may cause one-time disruptions during the upgrade.preallocate-bpf-maps:"false"# Regular expression matching compatible Istio sidecar istio-proxy# container image namessidecar-istio-proxy-image:"cilium/istio_proxy"# Encapsulation mode for communication between nodes# Possible values:# - disabled# - vxlan (default)# - genevetunnel:"vxlan"# Name of the cluster. Only relevant when building a mesh of clusters.cluster-name:default# Unique ID of the cluster. Must be unique across all conneted clusters and# in the range of 1 and 255. Only relevant when building a mesh of clusters.#cluster-id: 1# Interface to be used when running Cilium on top of a CNI plugin.# For flannel, use "cni0"flannel-master-device:""# When running Cilium with policy enforcement enabled on top of a CNI plugin# the BPF programs will be installed on the network interface specified in# 'flannel-master-device' and on all network interfaces belonging to# a container. When the Cilium DaemonSet is removed, the BPF programs will# be kept in the interfaces unless this option is set to "true".flannel-uninstall-on-exit:"false"# Installs a BPF program to allow for policy enforcement in already running# containers managed by Flannel.# NOTE: This requires Cilium DaemonSet to be running in the hostPID.# To run in this mode in Kubernetes change the value of the hostPID from# false to true. Can be found under the path `spec.spec.hostPID`flannel-manage-existing-containers:"false"# DNS Polling periodically issues a DNS lookup for each `matchName` from# cilium-agent. The result is used to regenerate endpoint policy.# DNS lookups are repeated with an interval of 5 seconds, and are made for# A(IPv4) and AAAA(IPv6) addresses. Should a lookup fail, the most recent IP# data is used instead. An IP change will trigger a regeneration of the Cilium# policy for each endpoint and increment the per cilium-agent policy# repository revision.## This option is disabled by default starting from version 1.4.x in favor# of a more powerful DNS proxy-based implementation, see [0] for details.# Enable this option if you want to use FQDN policies but do not want to use# the DNS proxy.## To ease upgrade, users may opt to set this option to "true".# Otherwise please refer to the Upgrade Guide [1] which explains how to# prepare policy rules for upgrade.## [0] http://docs.cilium.io/en/stable/policy/language/#dns-based# [1] http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-actiontofqdns-enable-poller:"false"

Add the new options manually to your old ConfigMap, and make the necessary
changes.

In this example, the debug option is meant to be kept with true, the
etcd-config is kept unchanged, and legacy-host-allows-world is a new
option, but after reading the Version Specific Notes the value was kept unchanged
from the default value.

After making the necessary changes, the old ConfigMap was migrated with the
new options while keeping the configuration that we wanted:

The Linux kernel applies limitations on the complexity of BPF code that is
loaded into the kernel so that the code may be verified as safe to execute on
packets. Over time, Linux releases become more intelligent about the
verification of programs which allows more complex programs to be loaded.
However, the complexity limitations affect some features in Cilium depending
on the kernel version that is used with Cilium.

One such limitation affects Cilium’s configuration of CIDR policies. On Linux
kernels 4.10 and earlier, this manifests as a restriction on the number of
unique prefix lengths supported in CIDR policy rules.

Unique prefix lengths are counted by looking at the prefix portion of CIDR
rules and considering which prefix lengths are unique. For example, in the
following policy example, the toCIDR section specifies a /32, and the
toCIDRSet section specifies a /8 with a /12 removed from it. In
addition, three prefix lengths are always counted: the host prefix length for
the protocol (IPv4: /32, IPv6: /128), the default prefix length
(/0), and the cluster prefix length (default IPv4: /8, IPv6: /64).
All in all, the following example counts as seven unique prefix lengths in IPv4:

In Cilium 1.0, all traffic from the host, including from local processes and
traffic that is masqueraded from the outside world to the host IP, would be
classified as from the host entity (reserved:host label).
Furthermore, to allow Kubernetes agents to perform health checks over IP into
the endpoints, the host is allowed by default. This means that all traffic from
the outside world is also allowed by default, regardless of security policy.

Users who are not reliant upon IP-based health checks for their kubernetes pods
may mitigate this issue on earlier versions of Cilium by adding the argument
--allow-localhost=policy to the Cilium DaemonSet for the Cilium container.
This prevents the automatic insertion of L3 allow policy in kubernetes
environments. Note however that with this option, if the Cilium Network Policy
allows traffic from the host, then it will still allow access from the outside
world.

Cilium 1.1 and later only classify traffic from a process on the local host as
from the host entity; other traffic that is masqueraded to the host IP is
now classified as from the world entity (reserved:world label).
Fresh deployments using the Cilium 1.1 YAMLs are not affected.

In cilium versions 1.2 and 1.3 DNS Polling was automatically used to
obtain IP information for use in toFQDNs.matchName rules in DNS based
policies.
Cilium 1.4 and later have switched to a DNS Proxy scheme - the
DNS Polling behaviour may be enabled via the a CLI option - and expect a
pod to make a DNS request that can be intercepted. Existing pods may have
already-cached DNS lookups that the proxy cannot intercept and thus cilium will
block these on upgrade. New connections with DNS requests that can be
intercepted will be allowed per-policy without special action.
Cilium deployments already configured with DNS Proxy rules are not
impacted and will retain DNS data when restarted or upgraded.

Deployments that require a seamless transition to DNS Proxy
may use Running a pre-flight DaemonSet to create a copy of DNS information on each cilium
node for use by the upgraded cilium-agent at startup. This data is used to
allow L3 connections (via toFQDNs.matchName and toFQDNs.matchPattern
rules) without a DNS request from pods.
Running a pre-flight DaemonSet accomplishes this via the --tofqdns-pre-cache CLI option,
which reads DNS cache data for use on startup.

DNS data obtained via polling must be recorded for use on startup and rules
added to intercept DNS lookups. The steps are split into a section on
seamlessly upgrading DNS Polling and then further beginning to intercept
DNS data via a DNS Proxy.

Policy rules may be prepared to use the DNS Proxy before an
upgrade to 1.4. The new policy rule fields toFQDNs.matchPattern and
toPorts.rules.dns.matchName/matchPattern will be ignored by older cilium
versions and can be safely implemented prior to an upgrade.

The following example allows DNS access to kube-dns via the DNS Proxy and allows all DNS requests to kube-dns. For completeness,
toFQDNs rules are included for examples of the syntax for those L3 policies
as well. Existing toFQDNs rules do not need to be modified but will now use
IPs seen by DNS requests and allowed by the toFQDNs.matchPattern rule.

Set the tofqdns-enable-poller field to true in the cilium ConfigMap used
in the upgrade. Alternatively, pass --tofqdns-enable-poller=true to
the upgraded cilium-agent.

Add tofqdns-pre-cache:"/var/run/cilium/dns-precache-upgrade.json"
to the ConfigMap. Alternatively, pass
--tofqdns-pre-cache="/var/run/cilium/dns-precache-upgrade.json" to
cilium-agent.

Deploy the cilium Running a pre-flight DaemonSet helper. This will download the cilium
container image and also create DNS pre-cache data at
/var/run/cilium/dns-precache-upgrade.json. This data will have a TTL of
1 week.

Deploy the new cilium DaemonSet

(optional) Remove tofqdns-pre-cache:"/var/run/cilium/dns-precache-upgrade.json"
from the cilium ConfigMap. The data will automatically age-out after 1 week.