Container Garbage Collection

The policy for container garbage collection is based on three node settings:

Setting

Description

minimum-container-ttl-duration

The minimum age that a container is eligible for garbage collection. The
default is 1m (one minute). Use 0 for no limit. Values for this setting can be
specified using unit suffixes such as h for hour, m for minutes, s for seconds.

maximum-dead-containers-per-container

The number of instances to retain per pod container. The default is 2.

maximum-dead-containers

The maximum number of total dead containers in the node. The default is 100.

The maximum-dead-containers setting takes precedence over the
maximum-dead-containers-per-container setting when there is a conflict. For
example, if retaining the number of maximum-dead-containers-per-container
would result in a total number of containers that is greater than
maximum-dead-containers, the oldest containers will be removed to satisfy
the maximum-dead-containers limit.

When the node removes the dead containers, all files inside those containers are
removed as well. Only containers created by the node will be garbage collected.

You can specify values for these settings in the kubeletArguments section of
the
node
configuration file (the /etc/origin/node/node-config.yaml file by default). Add the section if it does not already exist:

Detecting Containers for Deletion

Each spin of the garbage collector loop goes through the following steps:

Retrieve a list of available containers.

Filter out all containers that are running or are not alive longer than
the minimum-container-ttl-duration parameter.

Classify all remaining containers into equivalence classes based on pod and image name membership.

Remove all unidentified containers (containers that are managed by kubelet but their name is malformed).

For each class that contains more containers than the
maximum-dead-containers-per-container parameter, sort containers in the class by
creation time.

Start removing containers from the oldest first until the
maximum-dead-containers-per-container parameter is met.

If there are still more containers in the list than the
maximum-dead-containers parameter, the collector starts removing containers
from each class so the number of containers in each one is not greater than the
average number of containers per class, or
<all_remaining_containers>/<number_of_classes>.

If this is still not enough, sort all containers in the list and start
removing containers from the oldest first until the maximum-dead-containers
criterion is met.

Image Garbage Collection

Image garbage collection relies on disk usage as reported by cAdvisor on the
node to decide which images to remove from the node. It takes the following
settings into consideration:

Setting

Description

image-gc-high-threshold

The percent of disk usage (expressed as an integer) which triggers image
garbage collection. The default is 90.

image-gc-low-threshold

The percent of disk usage (expressed as an integer) to which image garbage
collection attempts to free. Default is 80.

You can specify values for these settings in the kubeletArguments section of
the
node
configuration file (the /etc/origin/node/node-config.yaml file by default).
Add the section if it does not already exist:

Detecting Images for Deletion

Two lists of images are retrieved in each garbage collector run:

A list of images currently running in at least one pod

A list of images available on a host

As new containers are run, new images appear. All images are marked with a time
stamp. If the image is running (the first list above) or is newly detected (the
second list above), it is marked with the current time. The remaining images are
already marked from the previous spins. All images are then sorted by the time
stamp.

Once the collection starts, the oldest images get deleted first until the
stopping criterion is met.