Give Codeship a try

Want to learn more?

Getting a notification that Docker containers are down in production is one of the worst ways to spend your night. In today’s article, we’ll discuss how to use Docker’s restart policy to automatically restart containers and avoid those late-night notifications.

What Happens When an Application Crashes?

Before we get started with Docker’s restart policy, let’s understand a bit more about how Docker behaves when an application crashes. To facilitate this, we’ll create a Docker container that executes a simple bash script named crash.sh.

#/bin/bash
sleep 30
exit 1

The above script is simple; when started, it will sleep for 30 seconds, and then it will exit with an exit code of 1 indicating an error.

Building and running a custom container

In order to run this script within a container, we’ll need to build a custom Docker container which includes the crash.sh script. In order to build a custom container, we first need to create a simple Dockerfile.

$ vi Dockerfile

The Dockerfile will contain the following three lines:

FROM ubuntu:14.04
ADD crash.sh /
CMD /bin/bash /crash.sh

The above Dockerfile will build a container based on the latest ubuntu:14.04 image. It will also add the crash.sh script into the / directory of the container. The final line tells Docker to execute the crash.sh script when the container is started.

With the Dockerfile defined, we can now build our custom container using the docker build command.

From the above, it appears that Docker was able to start a container named testing_restarts. Let’s check the status of that container by running docker ps.

$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS

The docker ps command doesn’t show any running containers. The reason for this is because docker ps by default only shows running containers. Let’s take a look at running and non-running containers by using the -a flag.

With the docker ps results, we can see that when an application within a Docker container exits, that container is also stopped. This means that, by default, if an application that is running within a container crashes, the container stops and that container will remain stopped until someone or something restarts it.

Changing Docker’s Default Behavior

It’s possible to automatically restart crashed containers by specifying a restart policy when initiating the container. To understand restart policies better, let’s see what happens when we use the always restart policy with this same container.

In the above command, we specified that Docker should apply the always restart policy to this container via the --restart flag. Let’s see what effect this has on our container by executing a docker ps again.

The second run shows the container has only been up for 19 seconds. This means that even though our application (crash.sh) continues to exit with an error, Docker is continuously restarting the container every time it exits.

Now that we understand how restart policies can be used to change Docker’s default behavior, let’s take a look at what restart policies Docker has available.

Docker’s Restart Policy(ies)

Docker currently has four restart policies:

no

on-failure

unless-stopped

always

The no policy is the default restart policy and simply does not restart a container under any circumstance.

Restarting on failure but stopping on success

The on-failure policy is a bit interesting as it allows you to tell Docker to restart a container if the exit code indicates error but not if the exit code indicates success. You can also specify a maximum number of times Docker will automatically restart the container.

Let’s try this restart policy out with our testing_restarts container and set a limit of 5 restarts.

We can see from the above that after 3 minutes the container is stopped. This is due to the fact that the container has been restarted more than our max-retries setting.

With success

The benefit of on-failures is that when an application exits with a successful exit code, the container will not be restarted. Let’s see this in action by making a quick minor change to the crash.sh script.

$ vi crash.sh

The change will be to set the exit code to 0.

#/bin/bash
sleep 30
exit 0

By setting the script to exit with a 0 exit code, we will be removing the error indicator from the script. Meaning as far as Docker can tell, this script will execute successfully every time.

With the script changed, we will need to rebuild the container before we can run it again.

Since the crash.sh script exited with a successful exit code (0), Docker understood this as a success and did not restart the container.

Always restart the container

If we wanted the container to be restarted regardless of the exit code, we have a couple of restart policies we could use:

always

unless-stopped

The always restart policy tells Docker to restart the container under every circumstance. We experimented with the always restart policy earlier, but let’s see what happens when we restart the current container with the always restart policy.

What’s great about the always restart policy is that even if our Docker host was to crash on boot, the Docker service will restart our container. Let’s see this in action to fully appreciate why this is useful.

$ sudo reboot

By default or even with on-failures, our container would not be running on reboot. Which, depending on what task the container performs, may be problematic.

With the always restart policy, that is not the case. The always restart policy will always restart the container. This is true even if the container has been stopped before the reboot. Let’s look at that scenario in action.

$ sudo docker stop testing_restarts
testing_restarts
$ sudo reboot

Before rebooting our system, we simply stopped the container. This means the container is still there, just not running. Once the system is back up after our reboot however, the container will be running.

The reason our container is running after a reboot is because of the always policy. Whenever the Docker service is restarted, containers using the always policy will be restarted regardless of whether they were running or now.

The problem is that restarting a container that has been previously stopped after a reboot can be a bit problematic. What if our container was stopped for a valid reason, or worse, what if the container is out of date?

The solution for this is the unless-stopped restart policy.

Only stop when Docker is stopped

The unless-stopped restart policy behaves the same as always with one exception. When a container is stopped and the server is rebooted or the Docker service is restarted, the container will not be restarted.

Let’s see this in action by starting the container with the unless-stopped policy and repeating our last example.

One important item with unless-stopped is that if the container was running before the reboot, the container would be restarted once the system restarted. We can see this in action by restarting our container and rebooting the system again.

The difference between always and unless-stopped may be small, but in some environments this small difference may be a critical decision.

Selecting the Best Restart Policy

When selecting the best restart policy, it’s important to keep in mind what type of workload the container is performing.

A Redis instance, for example, may be a critical component in your environment which should have an always or unless-stopped policy. On the other hand, a batch-processing application may need to be restarted until the process successfully completes. In this case, it would make sense to use the on-failures policy.

Either way, with Docker’s restart policy you can now rest assured that next time a Docker host mysteriously reboots at 3 a.m., your containers will be restarted.

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles. Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.

We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

Hi, I have set up a testing docker environment. In this I have dockerized nfs servers. This is a one docker container/client. Of course docker containers are run with a bunch of parameters. When setting a restart policy would these parameters also in effect when a container restarted? Also the name stays the same? It is important because after starting a container I am also giving an ip address with the ovs-docker command.

So to make the question simple: how would I make sure a container is in the same state after restarting?

thanks

Francisco Carriedo

Hi,

Do you know if there’s a way to put a notify hook to alert when max restarts `on-failure` are reached? It might be that the container talks to hardware that is explicitly malfunctioning, it would be wise to try to recover 5 times and notify if it couldn’t start it successfully again (and not keep on trying indefinitely).

Thanks for the great article.

Francisco Carriedo

Also,

How are restarts `on-failure:5` counted. For eg., if it tries 3 times and successfully runs the container on the 4th probably exiting cleanly, does it clear its restart attempts or do you have only one/two left.