Nuts and Bolts

Everyone has opinions and thoughts. Here are some of ours.

Under the Hood of AWS Elastic Beanstalk Part 1

Posted by Tung Nguyen
on
Jul 19, 2017

Elastic Beanstalk, EB, is a bit of a
magic box. I’d thought it would be good to poke around a little bit under the
hood, see how things actually work and learn from it. Understanding just enough
of how EB works is extremely helpful for debugging. For this example I’m using a
“64bit Amazon Linux 2017.03 v2.7.0 running Docker 17.03.1-ce” EB solution stack.

What Happens When We Deploy Code

So what happens when we deploy code to a EB environment? Let’s deploy a simple
little rails app and trace the environment to find out. The source code the
simple rails app is on GitHub at tongueroo/hi
though for the sake of this post it really doesn’t matter which app you used to
deploy

EB Host Manager

EB talks about a host manager process in the documentation architecture
section.
This is a daemon process that sits there and manages the server. It listens for
events like code deployment or environmental variable updates and process those
events when they are triggered. The host manager actually leverages the
/opt/aws/bin/cfn-hup
command.

So when we run the eb deploy command or deploy via the EB api, what eventually
happens is that the AWS EB service writes to an sqs queue under our AWS account.
The host manager daemon process, in turn, listens to this queue and does its
bidding. Note: I have redacted all sensitive information.

What does cfn-hup do?

If you trace through the files and the libraries, you will find AWS EB service
goes through the following steps:

The trace jumps from Python, Bash and Ruby scripts. It is easier and quicker to
actually see what is going on with the ps command. Run this watch command watch
‘ps auxwwf | grep -A 5 "python /opt/aws/bin/cfn-hup"' during a deploy.

The important thing is that eventually a ruby
/opt/elasticbeanstalk/lib/ruby/bin/command-processor script is called.
Tracing that code you’ll see that it loads a beanstalk-core gem. Finally,
the
/opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.3/lib/elasticbeanstalk/command-processor.rb
file contains the heart of the logic for how ElasticBeanstalk handles
deployment. You can also see in here that the command-processor command logs
what it does to /var/log/eb-commandprocessor.log. Here’s an example of some
of it’s output:

You can see that it goes through a set of actions and it downloads the code,
runs some hooks, does some building and cleans up. It reports success at the
end.

There are a few interesting files in the gem to look at to get a better
understanding of how it works:

lib/elasticbeanstalk/command.rb

lib/elasticbeanstalk/hook-directory-executor.rb

lib/elasticbeanstalk/executable.rb

Looking at command.rb we can see that the hooks scripts are in the
/opt/elasticbeanstalk/hooks/ directory. Looking at executable.rb,
ElasticBeanstalk’s command processor interestingly instance_eval’s the scripts
if it is a ruby script and shells out and executes it as a subprocess it is not.

Ultimately, the trace of the code leads to the
/opt/elasticbeanstalk/hooks/appdeploy/folder:

This scripts are basically run in order and calls 03build.sh to build the
docker image and 00run.sh to run the docker container.

Most Useful Thing To Know When Docker Container Won’t Start

The biggest takeaway from this blog post is knowing where to focus your
debugging efforts when AWS EB Service does not behave in a way you expect it to.
If you cannot figure out why the environment is not working. It is very likely
that the docker container did not start up properly. Run this command as root:

$ sudo su

From there you’ll see a docker run command. Copy and paste that command and
change the docker image from aws_beanstalk/staging-app to
aws_beanstalk/current-app. For example:

In this debugging session, there are eventually 2 docker containers running
because the first container started up just fine. If there are no containers
running, the debugging technique demonstrated above should be very useful. Most
of the time, the app is misconfigured with an invalid or missing environmental
variable. If you can get the docker container to run locally just fine and are
sure you are using the same docker image, the issue is most likely with your
environmental variables.

Thanks for reading this far. If you found this article useful, I'd really appreciate it if you share this article so others can find it too! Thanks 😁 Also connect with me on LinkedIn.