Monday, February 2, 2015

All the things your microservices should do (Part 2)

Logging

The services never write anything to disk. With immutable servers, our servers are deleted with every push, so anything written to disk is also deleted.

For CLI and local testing, the logger reverts to using stdout. For staging and production we ship our logs to CloudWatch Logging.

One great thing about this is that we can easily integrate monitoring with the log events to get notified when errors start coming in.

The last 10 log entries are also visible by viewing the health page.

Metrics

The services directly post their metrics, and these include memory, cpu, disk, and anything else you are interested in. Monitoring then can bet set to consume these metrics and alert you when things start moving in the wrong direction.

We send all our metrics to CloudWatch Metrics. The last 10 metrics are also visible by viewing the health page.

You could use an agent to send logs and metrics, but that complicates the installation and configuration process, and introduces new technologies into, what should be, a very minimalistic stack.

Profiling

Memory leaks and performance bottlenecks happen. When they do, it is important to be able to find them quickly. All our services run the net/http/pprof package so that we can remotely troubleshoot any issues.

Init

With golang, you do not have to install any dependencies or any programs. When our server boots up, it copies the single compiled file onto itself, and runs it with the -init parameter. This is done using a user data script and s3 for storage.

Like version, it is a contract between our services and our continuous delivery system. For most of the services, the init command just generates an upstart configuration file, but it can be used to do many different things.

Configuration

We do not use configuration files, they are a pain to deal with, track, and get right. You'll be too tempted to just SSH into the server and change the file instead of going through your CI/CD process.

Our dependencies are detected from various areas such as our Cloud Formation outputs, and service locators. One example is the CloudWatch Log Stream which is dynamically named by AWS. On start, the code will look up the output values from the CloudFormation outputs and configure the logging system accordingly.

Others are just hard-coded. It takes about 3 mins to change a hard-coded value and push the change to production. Not a big deal, and well worth the benefits of having such a simple deployment process.

We use IAM roles and EC2 security groups for authentication and network connectivity and haven't needed to store any passwords (yet...)