Microservice Pitfalls & AntiPatterns, Part 1

Last month I had the pleasure of attending ArchConf 2016 and hearing from some really smart people and their experiences working with microservices. As we’ve discussed before, we began our migration to microservices awhile back, so I wasn’t necessarily there to learn how to “do” microservices. What I was really interested in was hearing how other organizations have approached microservices, what struggles they’ve had, and any pearls of wisdom they may have to share. We’ve certainly learned a few lessons along the way, but as they say it sometimes takes a village.

One of the talks that really resonated with me was from the great Mark Richards entitled “Microservice Pitfalls and AntiPatterns”. Mark started off with two excellent quotes to help explain the difference:

An anti-pattern is just like a pattern, except that instead of a solution it gives something that looks superficially like a solution but isn’t one. –– Andrew Koenig

A pitfall is something that was never a good idea, even from the start. –– Neal Ford

I think this is something just about every software developer can relate to, and really sums up the struggles of software design quite nicely. Software anti-patterns, in general, are something that initially make sense. We can convince ourselves, and even our peers, that some design or approach is the best one given the knowledge on hand at the time. Eventually we learn more about the constraints or context of the system and we find out the approach is not ideal. Conversely, a pitfall is, typically, something we should recognize as bad from the start and avoid at all costs.

With this distinction in mind, Mark broke the talk into 2 sessions, one for pitfalls and the other for anti-patterns. In that same spirit, we’ll spend this first post focusing on the pitfalls he identified and how they apply to our own experience with microservices so far. Our next post will focus on the anti-patterns and how we have managed to navigate those.

Common Microservice Pitfalls

This post won’t cover every one of the microservice pitfalls that Mark discussed, but for completeness they are all described briefly below:

Grains of Sand: Too many fine grain microservices that become difficult to manage and support.

Developer Without A Cause: Decisions being made in a vacuum without consideration of business needs or priorities.

Jump On The Bandwagon: Endeavoring to create a microservice architecture without the infrastructure and processes to support it.

Logging Can Wait: Failing to consider log aggregation and correlation early on in a microservices architecture.

Static Contract: Not properly versioning APIs and interfaces between microservices.

Service Orphan: Poor microservice ownership leading to abandoned microservices that are never upgraded or maintained.

Are We There Yet: Failing to understand the full round trip time required to service a single request.

Dare To Be Different: Not using a common technology stack or base image when building microservices.

Give It A REST: Relying heavily on RESTful APIs when other messaging paradigms may be more appropriate.

In the following sections we’ll take a closer look at some of these and how they’ve played a role in our own microservices architecture.

Grains of Sand Pitfall

I suspect this is the pitfall most organizations struggle with as they attempt to move from monolith to microservice. It’s a scenario in which an organization creates microservices that are too fine grained. This leads to a proliferation of microservices that wreak havoc on testing, deployment, monitoring and just about every other aspect of the DevOps.

The underlying problem here is that defining the size of a microservice is one of the most challenging aspects of the architectural style. Some people have proposed strict line of code counts, while others have proposed more abstract ideas such as “no bigger than your head”. At HomeAdvisor, we don’t have a set guideline for how we divide our microservices. They tend to be along data and functional boundaries, but we certainly don’t enforce any arbitrary line of code counts or endpoint limits. For example, we have some microservices that only contain a single HTTP endpoint or listen to a single Kafka topic, but we also have some microservices with a dozen or so endpoints.

The really important take away for this pitfall is to understand that you may not choose the right level of granularity your first time through. In fact, it’s almost a guarantee that you won’t. As a rule of thumb you should start out with bigger, more coarse-grain services. It will always be easier to further decompose a single service than it will be to combine multiple services. Always stress quality over quantity, and if you find your services are still doing too much, then continue to break them apart.

Jump On The Bandwagon Pitfall

This occurs when your organization starts building microservices before the tools and processes are in place to support them. One of the recurring themes from all of the speakers at ArchConf was despite all the benefits and appeal of a microservice architecture, if you’re organization is not ready or willing to change it’s testing and deployment mentality, then you likely won’t get all the benefits of microservices.

While I don’t necessarily agree with this sentiment (moving towards a more modular architecture, even if it’s not microservices, can certainly be beneficial to building and testing a system), at HomeAdvisor we have made big strides in our DevOps practices. We still do our traditional monolithic deployments every 2 weeks, but we’ve also started deploying our microservices on their own schedules.

To achieve this, we have started building a suite of integration and automated tests around each microservice, and those tests are integrated into the build process. This gives us a high level of confidence that each change does not introduce a regression. We also use Jenkins to manage microservice builds and deployment, so we can move each microservice through its lifecycle independent of the rest of the system. The net result is we can deploy our microservices on a different schedule from both the monolith and other microservices. This reduces the amount of work for DevOps during the main deployment activities, and if something goes wrong we can roll back individual microservices without perturbing the rest of the system.

Another area where we fell a little behind early on was with monitoring. Similar to the Logging Can Wait pitfall, monitoring should not be an after thought in your microservices architecture. We had some existing infrastructure in place for our monolithic applications that were based on Nagios, and it was really simple to configure it to point to our few instances of the monolith. It would perform the same basic checks (heap usage, CPU usage, etc) and alert if anything violated the limits we setup. Once we started deploying microservices that were running on new hosts and each having their own monitoring needs, it became clear that our one size approach was not sufficient. Some microservices need Kafka monitoring, some need Elasticsearch monitoring, etc.

Our solution goes a little into the Dare To Be Different pitfall. Our standard microservice stack includes health checks that every microservice will want (CPU, memory, etc) and optional health checks that can give more detail about specific functionality when it makes sense (Kafka, Elasticsearch, Database, etc). We also make it very easy for microservice owners to write their own health checks (for example, knowing when a 3rd party system is unavailable) that are rolled up into overall microservice health. We still use Nagios to monitor and send alerts about our platform, but now it can get more tailored health information from each microservice instead of a generic status.

Logging Can Wait Pitfall

Logging in this sense refers to a couple of things: log aggregation between applications and the ability to correlate messages as they hop across multiple applications. Typically log aggregation and correlation are planned into later releases because they are viewed as less important than business logic and easy to implement at any time. But the truth is they should be planned up front and as part of your common microservice technology stack.

Admittedly, this is one area where we fell victim. Despite our best intentions when we began building microservices, we were guilty of overlooking this aspect. In terms of aggregation, we’ve just recently begun to use Sumo Logic to collect logs from all of our critical applications and services in a single place for searching and analyzing. But this is still a fairly new concept for us, and many people still resort to manually logging into multiple servers and running some sort of complex find and grep commands. As we become more comfortable with the platform, we’ll have some more concrete thoughts to share in the future.

As for correlation, definitely do not overlook the importance of this when you build microservices. Being able to trace messages through the system can really help troubleshoot problems and identify performance issues. For example, the developer in charge of Service A may know that she needs to communicate with Service B, but she might not know that Service B turns around and makes 10 calls to other services. In general she probably doesn’t need to know this when the system is behaving properly, but as soon as the responses from Service B slow down, having some correlation framework in place can quickly help identify the root cause.

At HomeAdvisor, we have multiple tools to help with correlation. The first is another third party platform, AppDynamics, that can dynamically instrument an application and track messages between them. In our example above, it would detect when Service A makes an HTTP call to Service B, and would also see that this caused Service B to make those additional calls. We can configure App Dynamics to detect any number of remote calls: HTTP, database query, Kafka message, distributed in-memory cache, etc. In short, we can see the call graph from each microservice to every other service in the system, even the ones we don’t write ourselves. When one of the edges of this graphs starts experiencing delays, we can tell pretty quickly which component is the problem.

Example App Dynamics dashboard for a microservice.

App Dynamics is a great tool, but it’s not practical to run in every environment we maintain. Another tool we use for message correlation is an extension of our open source Robusto API client that uses HTTP headers to track messages. There are two pieces:

Anytime an HTTP request is received by a microservice, we make note of any correlation ID using a ThreadLocal.

Anytime an HTTP request is sent by a microservice, if the correlation ID is present we simply propagate it. If it’s not present, we generate a new one using a pluggable factory mechanism.

Essentially, the first message in a chain of requests will create a new correlation ID, and all subsequent requests will simply propagate it. Luckily when we started building microservices, we already had a mechanism to asynchronously log our existing API requests to a database. When we decided to add in this custom correlation ID later on, we simply added it into the existing log message. While this logging isn’t as robust as a system like App Dynamics because we only see HTTP messaging between our applications, it does provide a quick way to see the interactions of our microservices using basic SQL queries. It also provides a nice framework to build on later. For example, we could easily start including the correlation ID in our application log messages, which would then be searchable from Sumo Logic. We could include the correlation ID in our Kafka message payloads, allowing us to track interactions of HTTP messages through our distributed messaging system.

Static Contract Pitfall

Versioning in a microservice architecture is something you should be thinking about from day one. The version is an integral part of the contract that a microservice provides, and it would be naive to think that the first API a microservice offers will be its last. Building microservices that can support multiple versions will make your life much simpler. When a new version of the API is created, you can support both the old and the new while the dependent services are migrated. Instead of having to upgrade all services at once, each service is free to upgrade to the new contract at its own pace, or never at all. What if those dependent services is really a mobile application? You can’t force people to upgrade on the schedule you want, so you will likely be stuck supporting multiple versions for long periods of time. Luckily, since you now know about the Logging Can Wait pitfall, you’ll have an easy way to track which versions are being called over time.

The problem with versioning APIs is that there are a lot of different approaches, in particular when it comes to HTTP:

Using a custom HTTP header to specify the version. This is a fairly common practice in the real world, but there are several arguments against this practice.

Using the Accept-Type header along with vendor definitions. This approach seems to be picking up some momentum and certainly favored over using a custom header.

Using the URL path. This is another common practice, but it flies in the face of core REST principles. The URL should uniquely identify the resource, not the contract.

Using query parameters. Another common practice that is compatible with a variety of clients.

Using the request body. This should definitely be avoided. Not only are you directly violating REST principles, not all HTTP methods expect request bodies so you may find some frameworks don’t support this scheme.

At HomeAdvisor we use an HTTP query parameter for versioning our APIs. This was driven by the fact that prior to building microservices, we had a very robust set of internal APIs that were already using this scheme. When it came time to start converting these APIs into microservices, using query parameters to express API versions was already second nature to our developers and was generally the least amount of friction.

All that to say, there is no right or wrong answer to versioning. Each approach has it merits and drawbacks, so it likely boils down to the technology stack you plan to use and what makes sense for your organization. For example, Spring MVC makes it really easy to create request mappings based on any combination of HTTP method, request path, request parameters, headers, and more. This means if you’re using Spring Boot for your microservice stack that it’s trivial to create two versions of the same API based on query parameter (in this example, we use the parameter “v”):

This allows us to maintain two versions of the same API, each one returning a different response body, driven by query parameters. For comparison, a framework like Spark only allows routes based on HTTP method and URI, so you would be limited to using the URI to embed version information. Of course you could have a single method with conditional logic that examines the headers or query parameters manually, but this bloats the method and potentially makes it more difficult to deprecate a particular version of the API:

Java

1

2

3

4

5

6

7

8

9

10

11

get("/:username/address",(request,response)->

{

if(request.queryParams("v").equals("1")

{

// Lookup and return AddressDTO

}

else

{

// Lookup and return EnhancedAddressDTO

}

});

Clearly the microservice stack can help guide how you version your APIs. But as long as API versioning is part of your evaluation, you’re already headed in the right direction.

Dare To Be Different Pitfall

If you want your microservice architecture to be maintainable, it’s imperative that everyone in your organization standardize on the same technology stack (or what Mark calls a “base image”). The same could probably be argued for monolithic systems as well, but the burden of testing, deploying, and monitoring your microservices will be far greater if every team uses their own implementations. Even if you standardize on the communications protocol (REST, SOAP, JMS, etc), there are common features of the architecture that you don’t want every team to implement on their own:

Service registration and discovery

Health and monitoring

Authentication, authorization, and auditing

Logging format

Database connectivity and object-relational mappings

Messaging provider

HTTP client/server library

To avoid this pitfall, one of your first tasks, before you build a single microservice, should be to standardize the technology stack for everyone. Imagine trying to build and manage a set of Jenkins jobs or Docker containers that can support Spring Boot, Spark, Jodd, DropWizard, and more. Each of these frameworks provide their own mechanisms for security, logging, configuration, embedded HTTP server, UI template engine, dependency injection, and more. And those are just the popular Java frameworks. If you have teams that decide to use Ruby, Python, or .NET things get even more chaotic. Trying to create a standard environment to support all of these would be nearly impossible.

At HomeAdvisor, we did a fair amount of prototyping and investigation before we settled on our current microservice stack, which is based on Spring Boot. Once we picked that, we were able to build common components that every team could leverage instead of building from the ground up. For example, we provide auto configurations that automatically register a microservice with ZooKeeper, connect them to the database, configure health checks appropriate for that microservice’s environment, and much more. We’ve pushed as much common functionality and boilerplate into the base image as possible so that anyone creating a new microservice can focus on business logic instead.

Conclusion

Clearly microservices come with their own cautionary tales and dangers. They’re not the single grand unifying software architecture that everyone should adopt as quickly as possible, but they’re certainly a growing trend and will likely be around for a long time. One of the speakers at ArchConf mentioned that just two years ago they only had two sessions geared towards microservices. This year over half of the presentations were geared towards microservices. As more and more organizations start moving towards microservices, it’s important to understand where others have gone wrong and how we can avoid some of these pitfalls.

Remember that pitfalls are things that are never good ideas. If your organization is considering a move to microservices, heed the advice of those who have gone before you and avoid at all costs. In the second part of this post, we’ll discuss microservices anti-patterns, many may sound good at first but can quickly create problems.

Based in Golden, CO, HomeAdvisor’s technology group is comprised of nearly 100 Java ninjas, front end gladiators, QA warriors, U/X experts and other rock stars. We build the technology that helps make HomeAdvisor the best place for homeowners to connect with home service professionals.

Download our Free Apps

About Michael Pratt

Software Developer / Technical Architecture Team

Trackbacks

[…] Microservice Pitfalls & AntiPatterns, Part 1 An anti-pattern is just like a pattern, except that instead of a solution it gives something that looks superficially like a solution but isn’t one. A pitfall is something that was never a good idea, even from the start. (from The Microservice Weekly #31) […]

[…] we talked about in our microservices pitfalls post, one of the most important things you should do before building microservices is settle on a […]

About Us

Based in Golden, CO, HomeAdvisor’s technology group is comprised of nearly 100 Java ninjas, front end gladiators, QA warriors, U/X experts and other rock stars. We build the technology that helps make HomeAdvisor the best place for homeowners to connect with home service professionals.