"innovative technology I think people should read about"

Category Archives: Uncategorized

I’ve had the opportunity to work with a few different infrastructure automation tools such as Puppet, Chef, Heat and CloudFormation but Ansible just has a simplicity to it that I like, although I admit I do have a strong preference for Puppet because i’ve used it extensively and have had good success with it.

In one of my previous project I was creating a repeatable solution to create a Docker Swarm cluster (before SwarmKit) with Consul and Flocker. I wanted this to be completely scripted to I climbed on the shoulders of AWS, Ansible and Docker Machine.

The script would do 4 things.

Initialize a security group in an existing VPC and create rules for the given setup.

Create machines using Docker-Machine of Consul and Swarm

Use AWS CLI to output the machines and pipe them to a python script that processes the JSON output and creates an Ansible inventory.

Use the inventory to call Ansible to run something.

This flow can actually be used fairly reliable not only for what I used it for but to automate a lot of things, even expand an existing deployment.

Next, the important piece is the pipe ( `|`). This signifies we pass the output from the command on the left of the | to the command on the right which is create_flocker_inventory.py so that the output is used as input to the script.

So what does the python script do with the output? Below is the script that I used to process the JSON output. It first setups up an _AGENT_YML variable that contains YAML for a configuration then the main() function takes the JSON from json.loads() in the script initialization and creates an array of dictionaries that represent instances and opens a file and writes each instance to the Ansible inventory file called “ansible_inventory”. After that the “agent.yml” is written to a file along with some secrets from the environment.

After this processes the JSON from the AWS CLI, all that remains is to run Ansible with our newly created Ansible inventory. In this case, we pass the inventory and configuration along with the ansible playbook we want for our installation.

Conclusion

Overall this flow can be used along with other Cloud CLI tools such as Azure, GCE etc that can output instance state that you can pipe to a script for more processing. It may not be the most effective way but if you want to get a semi complex environment up and running in a repeatable fashion for development needs it has worked pretty well to follow the “pre-setup_get-output_prcocess-output_install_config” flow outlined above.

What is FIO?

fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate. – (https://linux.die.net/man/1/fio)

fio can be a great tool for helping to measure workload I/O of a specific application workload on a particular device or file. Fio proves to be a detailed benchmarking tool used for workloads today with many options. I personally came across the tool while working at EMC when needing to benchmark Disk I/O of application running in different Linux container runtimes. This leads me to my next topic.

Why Docker based fio-tools

One of the projects I was working on was using Docker on AWS and various private cloud deployments and we wanted to see how workloads performed on these different cloud environments inside Docker container with various CPU, Memory, Disk I/O limits with various block, flash, or DAS based storage devices.

One way to wanted to do this was to containerize fio and allow users to pass the workload configuration and disk to the container that was doing the testing.

The first part of this was to containerize fio with the option to pass in JOB files by pathname or by a URL such as a raw Github Gist.

The Dockerfile (below) is based on Ubuntu 14 which admittedly can be smaller but we can easily install fio and pass a CMD script called run.sh.

What does run.sh do? This script does a few things, is checked that you are passing a JOBFILE name (fio job) which without REMOTEFILES will expect it to exist in `/tmp/fio-data` it also cleans up the fio-data directory by copying the contents which may be jobs files out and then back in while removing any old graphs or output. If the user passes in REMOTEFILES it will be downloaded from the internet with wget before being used.

There are two other Dockerfiles that are aimed at doing two other operations. 1. Producing graphs of the output data with fio2gnuplot and serving the graphs and output from a python SimpleHTTPServer on port 8000.

All Dockerfiles and examples can be found here (https://github.com/wallnerryan/fio-tools) and it also includes an All-In-One image that will run the job, generate the graphs and serve them all in one which is called fiotools-aio.

Other Examples

Important

Your fio job file should reference a mount or disk that you would like to run the job file against. In the job fil it will look something like: directory=/my/mounted/volume to test against docker volumes

If you want to run more than one all-in-one job, just use -v /tmp/fio-data instead of -v /tmp/fio-data:/tmp/fio-data This is only needed when you run the individual tool images separately

Notes

The fio-tools container will clean up the /tmp/fio-data volume by default when you re-run it.

If you want to save any data, copy this data out or save the files locally.

How to get graphs

When you serve on port 8000, you will have a list of all logs created and plots created, click on the .png files to see graph (see below for example screen)

Testing and building with codefresh

As a side note, I recently added this repository to build on Codefresh. Right now, it builds the fiotools-aio Dockerfile which I find most useful and moves on but it was an easy experience that I wanted to add to the end of this post.

Navigate to https://g.codefresh.io/repositories? or create a free account by logging into codefresh with your Github account. By logging in with Github it will have access to your repositories you gave access to and this is where the fio-tools images are.

I added the repository as a build and configured it like so.

This will automatically build my Dockerfile and run any integration tests and unit tests I may have configured in codefresh, thought right now I have none but will soon add some simple job to run against a file as an integration test with a codefresh composition.

Conclusion

I found over my time using both native linux tools and docker-based or containerized tools that there is need for both sometimes and in fact when testing container-native application workloads sometimes it is best to get metrics or benchmarks from the point of view of the application which is why we chose to run fio as a microservice itself.

On my spare time I run a website for a tax accounting company. This monolithic app is a largely stateless app (not really but we don’t need persistent stores/volumes), uses one of the best online tax software, and uses ruby on rails and runs on an EC2 instance with Apache and MySQL Server and Client installed. This is what is referenced as a “monolithic app” because all components are installed on 1 single VM. This makes the application more complicated to edit, patch, update etc (even though it barely ever needs to updated apart from making sure the current year’s Tax information and links are updated.) If we want to migrate to a new version of the database or newer version of rails, things are not isolated and can overlap. Now, RVM and other mechanisms can be used to do this, but by using Docker to isolate components and Amazon ECS to deploy the app, we can rapidly develop, and push changes into production in a fraction of the time we used to. This blog post will go through the experience of migrating that “Monolithic” ruby on rails app and converting it into a 2 service Docker application that can be deployed to Amazon ECS using Docker Compose.

First things first:

The first thing I did was ssh into my EC2 instance and figure out what dependencies I had. The following approach was taken to figure this out:

Use lsb_release -a to see what OS and release we’re using.

Look at installed packages via “dpkg” / “apt“

Look at installed Gems used in the Ruby on Rails app, take a peek at Gemfile.lock for this.

View the history of commands via “history” see any voodoo magic I may have done and forgot about 🙂

View running processes via “ps [options]” which helped me remember what all is running for this app. E.g. Apache2, MySQL or Postgres, etc.

This gives us a bare minimum of what we need to think about for breaking our small monolith into separate services, at-least from a main “component” viewpoint (e.g. Database and Rails App). It’s more complicated to try and figure out how we can carve up the actual Rails / Ruby code to implement smaller services that make up the site, and in some cases this is where we can go wrong, if it works, don’t break it. Other times, go ahead, break out smaller services and deploy them, but start small, think of it like the process of an amoeba splitting 🙂

We can now move into thinking about the “design” of the app as it applies to microservices and docker containers. Read the next section for more details.

Playing with Legos:

As it was, we had apache server, rails, and mysql all running in the same VM. To move this into an architecture which uses containers, we need to separate some of these services into separate building blocks or “legos” which you could think of as connecting individual legos together to build a single service or app. SOA terms calls this “Composite Apps” which are similar in thinking, less in technology. We’ll keep this simple as stated above and we’re going to break our app into 2 pieces, a MySQL database container and a “Ruby on Rails” container running our app.

Database Container

First, we’ll take a look at how we connect the database with rails. typically in Rails, a connection to a database is configured in a database.yml file and looks something like this. (or something like this)

We need a way to let our application know where this container lives (ip address) and how to connect to it (tcp?, username/password, etc). We’re in luck, with docker we can use the –link (read here for more information on how –link works) flag when spinning up our Rails app and this will inject some very useful environment variables into our app so we can reference them when our application starts. By assuming we will use the –link flag and we link our database container with the alias “appdb” (more on this later) we can change our database.yml file to look something like the following (Test/Prod config left out on purpose, see the rest here)

Rails Container

For the ruby on rails container we can now deploy it making sure we adhere to our dynamic database information like this. Notice how we “link” this container to our “appname_db” container we ran above.

We also map a port to 3000 because we’re running our ruby on rails application using rails server on port 3000.

Wait, let’s back track and see what we did to “containerize” our ruby on rails app. I had to show the database and configuration first so that once you see how it’s deployed it all connects, but now let’s focus on what’s actually running in the rails container now that we know it will connect to our database.

The first thing we needed to do to containerize our rails container was to create an image based on the rails implementation we had developed for ruby 1.8.7 and rails 3.2.8. These are fairly older version of the two and we had deployed on EC2 on ubuntu, so in the future we can try and use the Ruby base image but instead we will use the ubuntu:12.04 base image because this is the path of least resistance, even though we can reduce our total image size with the prior. (more about squashing our image size later in the post)

Doing this we can create Dockerfile that looks like the following. To see the code, look here We actually don’t need all these packages (I dont think) but I haven’t got around to reducing this to bare minimum by removing one by one and seeing what breaks. (This is actually easy and a fun way to get your container just right because we’re using docker, and things build and run so quickly)

As you can see we use “FROM ubuntu:12.04” to denote the base image and then we continue to install packages, COPY the app, make sure “bundler” is installed and install using the bundler the dependencies. After this we set the RAILS_ENV to use “production” and rake the assets. (We cannot rake with “db:create” because the DB does not exist at docker build time, more on this in runtime dependencies) Then we throw our init script into the container, chmod it and set it as the CMD used when the container is run via Docker run. (If some of this didn’t make sense, please take the time to run over to the Docker Docs and play with Docker a little bit. )

Great, now we have a rails application container, but a little more detail on the init script and runtime dependencies before we run this. See the below section.

Runtime dependencies:

There are a few runtime dependencies we need to be aware of when running the app in this manner, the first is that we cannot “rake db:create” until we know the database is actually running and can be connected to. So in order to make this happen at runtime we place this inside the init script.

The other portion of the runtime dependencies is to make sure that “rake db:create” does not fire off before the database is initialized and ready to use. We will use Docker Compose to deploy this app and while compose allows us to supply dependencies in the form of links there no real control over this if A) they aren’t linked, and B) if there is a time sequence needed. In this case it takes the MySQL container about 10 seconds to initialize so we need to put a “sleep 15” in our init script before firing off “rake db:create” and then running the server.

In the below script you can see how this is implemented.

Nothing special, but this ensures our app runs smoothly every time.

Running the application

We can run the app a few different ways, below we can see via Docker CLI and via Docker Compose.

Running the app in ECS and moving DNS:

We can run this in Amazon ECS (Elastic Container Service) as well, using the same images and docker compose file we just created. If you’re unfamiliar with EC2 or the Container Service, check out the getting started guide.

First you will need the ecs cli installed, and the first command will be to setup the credentials and the ECS cluster name.

Next, you will want to create the cluster. We only create a cluster of size=1 because we don’t need multiple nodes for failover and we aren’t running a load balancer for scale in this example, but these are all very good ideas to implement for your actual microservice application in production so you do not need to update your domain to point to different ECS cluster instances when your microservices application moves around.

After this, we can send our docker compose YAML file to ecs-cli to deploy our app.

ecs-cli compose --file docker-compose.yml up

To see the running app, run the following command.

ecs-cli ps

NOTE: When migrating this from EC2 make sure and update the DNS Zone File of your domain name to point at the ECS Cluster Instance.

Finally, now that the application is running

Let’s back track and squash our image size for the rails app.

There are a number of different ways we can go about shrinking our application such as a different base image, removing unneeded libraries, running apt-get remove and autoclean and a number of others. Some of these taking more effort than others, such as if we change the base image, we would need to make sure our Dockerfile still installs the needed version of gems, and we can alternatively use a ruby base image but the ones I looked at don’t go back to 1.8.7.

The method we use as a “quick squash” is to export, and import the docker image and re-label it, this will squash out images into one single image and re-import the image.

As you can see, this squashed our image down from 256MB to 184MB, not bad for something so simple. Now I can do more, but this image size for my needs is plenty small. Here is a good post from Brian DeHamer on some other things to consider when optimizing image sizes. Below you can see the snapshot of the docker image (taxmatters is the name of the company, I have been substituting this with “appname” in the examples above).

Development workflow going forward:

So after we finished migrating to a Docker/ECS based deployment it is very easy to make changes, test in either a ECS development cluster or using Local Docker Machine, then deploy to the production closer on ECS when everything checks out. We could also imagine code changes automated in a CI pipeline where a CI pipeline kicks off lambda deployments to development after initial smoke tests triggered by git push, but we’ll leave that for “next steps” :).

I wrote a post not to long ago about creating a microservices architecture from scratch as part of series I am doing on modern microservices. Some colleagues and friends of mine suggested I break a portion of that post out into its own so I can continue to update it as the ecosystem grows. This is an attempt to do so. The portion they were talking about was the breakdown of layer and tools within MSA in my post here which laid the initial pass at this. This post will try and fill these layers out and and continue to add to them, there is just no way I can touch every single tool or definition correctly as I see them so please take this as my opinion based on the experience I have had in this ecosystem and please comment with additions, corrections, comments etc.

Applications / Frameworks / App Manifests

Scheduling / Scaling

Management Orchestration

Monitoring (including Health) / Logging / Auditing

Runtime Build/Creation (think build-packs and runtimes like rkt and docker)

Networking / Load Balancing

Service Discovery / Registration

Cluster Management / Distributed Systems State

Container OS’s

Data Services, Data Intelligence, and Storage Pools

To give you an idea of the tools available and technologies that fall into these categories, here is the list again, but with some of the tools and technologies in the ecosystem added. *Keep in mind this is is probably not an exhaustive list, if you see a missing layer or tool please comment!

*Note: Some of these may seem to overlap, if I put Kubernetes under Orchestration, it could easily fit into Cluster Management, or Scheduling because of its underlying technologies, however this is meant to label something with it’s overall “feel” for how to ecosystem views the tool(s), but some tools may appear in more than one section. I will labels these (overlap)

*Note: I will continue to add links as I continue to update the breakdown*

It has been a crazy past few months leading up to DockerCon in San Francisco and I wanted to share some thoughts about current events and some of the work we have been participating in and around the Docker community and ecosystem now that we’re post-conference this week.

I have been working in open-source communities for more than five years now across technology domains including software defined networking, infrastructure & platform as a service, and container technologies. Working on projects in the Openflow/SDN, Openstack, and container communities has had its ups and downs but Docker is arguably the hottest tech communities out there right now.

There are so many developments going on in this ecosystem around pluggable architectures, logging, monitoring, migrations, networking and enabling stateful services in containers. Before I talk more in depth about persistence and some of the work my team and partners have been up to I want to highlight some of the major takeaways from the conference and the community right now.

The theme at @DockerCon was “docker in production” and by this I mean docker is ready to be used in production. Depending on who you ask and how docker is being used, production and containers with microservices can either be “hell” as Bryan Cantrill put it (If you haven’t seen Bryan’s Talks on the unix philosophy and production debugging, I highly recommend any of his sessions, especially the ones from the recent Orielly conference and this past weeks DockerCon) or it can really help your applications break down into their bounded domains with highly manageable and efficient teams going through the CI/CD build/ship/deploy process efficiently well. Netflix OSS [http://netflix.github.io/] is always a great example of this done well and many talks by Adrian Cockcroft dig into this sufficiently. You can also see my last post [https://aucouranton.com/2015/04/10/what-would2-microservices-do/] about microservices which will help with some context here.

So, Is Docker ready for production? Here are a list of what you need to make sure you have a handle on before productions use, and I might add that these are also big topics at DockerCon technically and there are many projects and problems still being actively solved in these spaces, I’ll try to list a few as I go through.

• Networking

Docker’s “aquirehire-sition” of the Socketplane startup culminated in the ability to use lib network [https://github.com/docker/libnetwork] which mean you can connect your docker daemons from hosts to host to allow container to easily send IP traffic over layer 2 networks. Libnetwork is maintains outside of the main docker daemon and abstracts the networking nicely and most importantly abstracts too much network knowledge away from the end user and make things “just work”

• Security

Docker is hardening the registry and images as well as the docker engine itself. I got to chat with Eric Windisch from Docker about what he had been focusing on with docker security. The docker engine’s security and hardening is at the forefront of focus because any vulnerabilities there means the rest of your container could be compromised. There is a lot of work going on around basic source hardening as well as other techniques using apparmour and selinux. Looking forward to seeing how to security aspect of Docker unfolds with other projects like Lightwave from VMware.

• Logging & Monitoring & Manageability

Containers are great, but once your running thousands to tens of thousands of them in products the need for great tooling to help debug, audit, troubleshoot and manage is a necessity. There seems to be lots of great tooling coming out to help manage containers. First docker talked about “Project Orca”, an opinionated container stack that aims at combining Docker Engine, Docker Swarm, Networking, GUI, Docker Compose, security, plus tools for installation, deployment, configuration. This of course isn’t always the way docker will be run but would be nice to have a way to get all this up and running quickly with manageability. Other tools like loggly, cadvisor, ruxit, datadog, log entries and others are all competing to be the best options here and quite frankly thats great!

• Pluggability

Docker has given the power to its ecosystem by telling it wants a pluggable, extensible toolset that allows for different plugins to work with their network, auth, and storage stacks. This provides a way similar to openstack for customers and users to say plugin an openvswitch network driver, along with lightwave for auth and EMC ScaleIO for persistent storage. Pretty cool stuff considering docker is only just over 2 years old!

• Stateful / Persistant services

This last bullet here is near and dear to my employer EMC and we have done some really awesome work by partnering up with ClusterHQ (The Data Container Poeple) [https://clusterhq.com/] who’s open-source Flocker project can manage volumes for your containers and enables mobility and HA for those volumes when you want to go ahead and start moving around or recovering your containerized applications, really cools stuff.

We had the opportunity to host a meetup at Pivotal Labs in San Francisco to showcase the partnership and drivers, at the Pivotal labs office we had a number of people come out for a few live demos, some beer and great food and conversation. Here is a gist for the ScaleIO demo we ran at the meetup showcasing Flocker + ScaleIO running on Amazon AWS deploying a MEAN stack application that ingested twitter data and placed it into MongoDB using node and express.https://gist.github.com/wallnerryan/7ccc5455840b76c07a70

At DockerCon Clint Kitson and I along with some ECS folks had a packed house for our partner tutorial session showcasing the ClusterHQ, RexRay and ECS announcements at DockerCon. The room was pas standing room only and attendees started to fill the floor. We hoped to have a bit more time to let the folks with laptops actually get to hack on some of the work we did but unfortunately pressed into 40 minutes we did what we could!

EMC also integrates through a native Go implementation called RexRay, a way to manage your persistence volumes but without the auto-mobility Flocker gives. RexRay is really cool in the way its working on letting you use multiple backends at the same time, say EC2 EBS Volumes as well as EMC ScaleIO.

In all, persistence and containers is a here to stay and there are a number of reason why and some items to keep and eye on. First the stateless and 12-factor app was the rage, but its not realistic and people are realizing state exists and running stateful containers like databases is actually important to the microservices world. This is because all containers have state, even if its “stateless” there is in memory state like running programs and open sockets that may need to be dealt with in certain use cases like live migration. If you haven’t seen the live movement of realtime Quake playing and moving between cities across oceans please watch, its great! (I’ll post the video once I find it)

Data is becoming a first class citizen in these containerized environments. As more workloads gets mapped to container architectures we see the need for the import and of data consistency, integrity and availability come into play for services that produce state and need it to persist. Enter enterprise storage into the mix. Over the week and weekend we saw a number of companies and announcements around this including some of our own at EMC. A few offerings that caught my eye are:

As we prepare for DockerCon 2015 next week, we’re taking a very different approach than EMC usually takes at events. Most events are focused on outbound messaging, product announcements and a heavy dose of sales and marketing. But since EMC World 2015, it’s no longer your father’s EMC.

Next week, our focus is entirely on enablingthe community of developers and end-users that are moving towards Cloud Native applications (or “3rd Platform”) and container-based infrastructure. We fully believe that this is not just a trend, but a major shift in how businesses will interact with their customers, and that software is at the center of this revolution.

So what does it mean to enable the community and ecosystem of developers? We’re taking a three-legged approach. The overall theme is a focus on Data Persistence and how this evolving space is critical to those Cloud Native applications.

Compute nodes represent a potential bottleneck in an OpenStack Cloud Environment, because the compute nodes run the VM’s and Applications, the workloads fall on the I/O within the Hypervisor. Everything from Local File system I/O, RAM Resources and CPU can all affect the efficiency of your cloud.

One thing to consider when provisioning physical machines is to look at what Guest/VM flavours you are going to allow to be deployed on that machine. Flavours that eat up 2VCPUs and 32G of RAM may not be well suited with a machine with only 64G of RAM and 8 CPU Cores

The notion of tenancy is also important to keep track of, tenant size and activity factors into how your environments resources are used. Tenants consume images, snapshots, volumes and disk space. Consider how many tenants will consume your cloud and adjust your resources accordingly. Make sure if you plan to take advantage of overprovisioning think about thin provisioning and potential performance hits.

Using KVM:

KVM is a well-supported hypervisor for Nova and has its own ways to increase performance. KVM isn’t the only Hypervisor to choose, Hyper-V, Xen, VMware can also be supported in Openstack, but KVM is a powerful competitor. Tuning your hypervisor is just as important as tuning your cloud resources and environment, so here are some things to consider: