Persisting Docker Volumes in ECS using EFS

– using EFS as a means to persist your Docker Volumes in your ECS cluster

Last week we faced a new challenge to persist our Docker Volume using EFS. Sounds easy, right? Well, it turned out to be a bit more challenging than expected and we were only able to find a few tips here and there. That is why we wrote this post so others may succeed faster.

Before digging into the solution, let’s take a minute to describe our context to elaborate a bit more on the challenge.
First of all, we believe in Infrastructure as Code and thereby we use CloudFormation to be able to recreate our environments. Luckily Amazon provides a working sample and we got EFS working quite easily. The next part was to get Docker to use a volume from EFS. We got lucky a second time as Amazon provides another working sample.

We managed to combine these resources and everything looked alright, but a closer look revealed that the changes did not persist. We found one explanation for why it didn’t work. It appears that we mount EFS after the Docker daemon starts and therefore the volume mounts an empty non-existing directory. In order to fix that we did two things, first we orchestrated the setup and then we added EFS to fstab in order to auto-mount on reboot.

extracted FileSystem EFS into another CF template and exported the EFS identifier so that we can use ImportValue

added -p to the mkdir command just in case

enhanced mount to use imported filesystem reference

added mount to fstab so that we auto-mount on reboot

recursive changed EFS mount ownership

restarted Docker daemon to include mounted EFS and started ECS as it does not automatically restart when the Docker daemon restarts

added ECS cluster info to ECS configuration

added ECS agent security group so that port 51678 which the ECS agent uses is open

added yum update just in case

included launch configuration into auto scaling group for the ECS cluster and added depends on ECS cluster

We were a bit surprised that EFS does not require an additional volume driver to function. It appears to work out-of-the-box and turned out to be quite straightforward. Thank you for reading and enjoy using EFS as a means to persist your Docker Volumes in your ECS cluster!