A New Era of Application Services at Puppet Labs

Breadcrumb

Recently at Puppet Labs, we’ve been putting a lot of thought into how to make our server-side applications faster, smarter, and more modular. Today, we’re excited to give you a sneak peek into the future of some of this work.

Trapperkeeper

Trapperkeeper is a new Clojure framework we’ve written for hosting long-running applications and services. It takes some of the lessons we’ve learned about high-performance server-side architecture over the last few years and combines them with patterns for maximizing modular, reusable code. It’s entirely open source, and we’re looking forward to seeing how the community helps it grow and evolve.

We’ve actually been thinking about this for a few years; the fruits of our original efforts became available in the form of PuppetDB, originally released in May of 2012. PuppetDB has been an overwhelming success for us, so since that time, in addition to improving PuppetDB itself, we’ve been thinking about how to take some of the technology choices from the project, and take advantage of them elsewhere in the Puppet ecosystem. Let’s take a quick dive into some of those technology choices.

The Java Virtual Machine

The JVM is an ideal platform to use for long-running server side applications. It's been battle-tested over the last 20 years, and it’s an extremely fast, stable, and reliable platform on which to run applications like web servers, message queues, and other services. Additionally, it has a wealth of existing tools, libraries, and frameworks that serve as tremendously valuable building blocks to enable rapid development and allow developers to focus their efforts on domain-specific problems.

Several great profiling tools and debuggers available in the JVM ecosystem give developers and users unparalleled insight into the behavior and performance of their applications. Sophisticated instrumentation and metrics APIs allow us to track the most important characteristics of a running application, and expose them in a highly visible, user-friendly fashion. And because JVM-based applications are portable across most of our supported platforms, we don’t need to spend as much time building and packaging things for each individual platform.

Clojure

Given that Puppet and many of our other projects are written in Ruby, Puppet Labs has been mainly a Ruby shop until recently. The programming language we chose for PuppetDB is Clojure, and we’ve been moving towards adopting Clojure for most of our new back-end projects.

Clojure is a relatively new (and, if you ask us, super-cool) language. Like Ruby, Clojure is a dynamically-typed language, but beyond that it differs from Ruby in many ways, and has a few important features that make it an ideal choice for back-end services development:

Clojure is a functional programming language from top to bottom. This means that code written in Clojure is very modular, composable, reusable and easy to reason about. The vast majority of your code is written in a way that is free of side effects (it’s idempotent, like Puppet!), so you don’t have to worry about hard-to-reproduce bugs caused by state changing out from underneath you.

Clojure has simple, powerful concurrency primitives. This means that you can write multi-threaded code that can take full advantage of all of the processing power available in a server, without all the complexity of traditional concurrency primitives like locks.

Clojure provides high-performance immutable data structures. This is another huge boon for writing concurrent code; you don’t have to worry about putting locks around your objects because it’s impossible for them to be modified by another thread.

Clojure has excellent interoperability with the Java language, which means we have access to all of the powerful, well-tested libraries available in the Java ecosystem.

Clojure also has a thriving and very active community — a recent study found Clojure’s popular has increased drastically over the last 6 months.

Clojure is really fun to write!

Broadening Our Horizons

OK, so, Clojure is awesome, and the JVM is a great platform for building our server-side applications, and PuppetDB has been enormously successful in the field. Next, we needed to decide how to take the best parts of the PuppetDB architecture and mold them into a state that would serve as a foundation for other new products and services at Puppet Labs.

We found that many of our applications needed a lot of the same functionality that has become commonplace in today's application frameworks:

the ability to configure and control which parts of the system are loaded at run-time

the ability to compose modular bits of functionality

a way to cohesively manage the lifecycle of the various components making up an application

a common logging mechanism that wouldn't need to be set up and configured by each component

a web server flexible enough to load multiple web apps, and to support a rich set of options for configuring SSL

Of course, our first inclination was to use an existing solution for this; after all, we’re not in the business of re-inventing wheels when we can avoid it, and we’re certainly not the first shop who has had a need for something in this space. So we allocated some time to survey the landscape and see if any existing solutions were a good fit for our needs. Here are some of the options we explored.

Alternatives to "Build it Here"

OSGi

OSGi has some features that were very attractive to us; in particular, it provides a Service Registry that can be used to define services, load them together into the JVM, and manage dependencies between them. Services are defined via Java interfaces, so it’s possible to write different implementations of a given service and swap them out without affecting other services they depend on, or provide functionality to.

However, OSGi can be a fairly heavyweight solution: It’s easy to drag in a lot of dependencies that result in a very large file size for your application’s redistributables. Also, OSGi provides a lot of very powerful — but very complex — functionality to allow you to hot-swap service implementations in a running JVM. It relies heavily on very advanced usage of Java’s classloaders to manage this, which can often lead to the phenomenon known as classloader hell. The flexibility that this usage of classloaders provides comes with a cost in terms of complexity, and one of the main tenets of Clojure is that avoiding complexity is key to writing maintainable software.

Furthermore, we don’t really need a lot of the hot-swapping features provided by frameworks like OSGi. These kinds of features are critical for SaaS applications, but not always useful for on-premise applications like those we build at Puppet Labs.

JBoss / Immutant

JBoss is a popular Java enterprise application server. There is a related project called Immutant that provides Clojure bindings for most of the JBoss functionality. Immutant brings a lot to the table, and we gave it strong consideration… but in the end, we decided that there were a few details that prevented it from being an ideal fit for us.

The Immutant/JBoss runtime is a pretty large, full-featured deployment artifact. The "slim" version is in the ballpark of 50MB, and the “full” version is in the ballpark of 125MB. Those sizes are perfectly reasonable for SaaS applications, but felt a bit unwieldy for Puppet Labs’ on-premise use cases, partially because we have to be careful about bloating the size of our installers too much, but also because in most of our applications / services, we’d only be using a small subset of the functionality.

Immutant ships with some predefined technology choices: JBoss web server, HornetMQ message queue, Infinispan key/value cache, etc. While these choices are all perfectly reasonable, we were looking for something that would allow us to choose the specific technologies for individual subsystems. We were also hoping we could find a solution that would allow us to swap out other technologies for the subsystems if the need should arise — for example, a solution that would allow us to decide to switch from one message queue to another without necessarily changing much of the consuming code.

Since we package all our own software at Puppet Labs, we need something that is lightweight, modular, and easy to fit into our packaging paradigm. We also need to be able to easily package our application code along with the framework as a single distributable. Immutant and JBoss are more amenable to the model where you install them first and separately, and then load your app into them afterwards.

Clojure Frameworks and Libraries

We also evaluated a few other Clojure-specific frameworks, including Jig and Component. These both look fantastic and useful, but neither seemed like quite the right fit. That said, the authors have done great work, and their projects provided useful inspiration for us, so we highly encourage you to check them out! They may be a perfect fit for your project, even though they didn’t seem like quite the right fit for ours.

At the time we evaluated them, Jig felt more targeted to developers (as opposed to administrators / end users), and not necessarily intended to be something you would ship with an on-premise application. Component required you to define your application structure in code, and we needed something more dynamic and configuration-driven. Neither had built-in solutions for things like logging and service configuration, and neither seemed to provide a solution for specifying an "interface" for a service that would allow you to swap out alternate implementations.

Enter Trapperkeeper

After careful consideration, we decided it would be worth our while to build our own Clojure services framework. The result: Trapperkeeper, a new open source project from Puppet Labs. You can think of Trapperkeeper as a sort of "binder" for Clojure Ring applications and other modular bits of Clojure, Java, or JRuby code. It is a Clojure framework for hosting long-running applications and services. Of course, the word "service" means very different things in different contexts. In Trapperkeeper, we define a service as:

a set of functions

a set of dependencies on other services

a simple lifecycle

To use these services, Trapperkeeper also provides a basic application framework. At startup, the framework decides which services to load based on a configuration file, and it also provides a common configuration mechanism to each service. This enables the user to configure the list of services that should be loaded into the framework, as well as providing a way to configure each individual service.

So, what does all this mean to you? That depends on whether you’re wearing your Clojure developer hat or your Puppet user hat. For those of you in the former camp, check out the github repo, and stay tuned for another blog post in the near future where we’ll explore Trapperkeeper in depth, in all of its Clojure-y glory. For those of you in the latter group, here are some of the reasons we’re really excited about the project:

It will help us accelerate our move to a more robust and scalable platform for all our new server-side projects.

It will make it much easier for us to re-use the various architectural components of our server-side stack across a broad range of Puppet Labs applications and services, so that we can focus our energies on the core features that matter to you.

It includes some of the core technologies that have been successful in the field with PuppetDB, so we have confidence we’re building our new products on a solid, battle-tested foundation.

It will mean that we have consistent, simple configuration formats (including logging configuration) across all our new applications.

In short, we’re excited about the new applications and features we have in store for you in the near future. We’re excited about building on a robust and scalable platform, and we wanted to give you a sneak preview of some of the related technology. Stay tuned for more good things to come!

If you are interested in more technical detail about Trapperkeeper and how we're using Clojure, check out our follow-up blog post.

Tell Us What You Think!

We love hearing from our users and our community. At the end of the day, you are what keeps the lights on over here at Puppet Labs — so, your opinions are very important to us. If you have thoughts on any of our current features or suggestions to help us shape and prioritize features for upcoming releases, we really want to hear from you! Here are some great ways to get in touch with us:

Sign up to be a Puppet Test Pilot! You'll get free goodies as a reward for your participation if you are able to volunteer a few minutes of your time to tell us what you think about prototypes of upcoming features.

I've the strong sensation that we'll hear a lot about Trapperkeeper in the future. And maybe that it's time to start to learn Clojure.
Best of luck, if the quality of this project and the others that will evolve around it will be similar to the one of PuppetDB, we will surely see Great Things happening in the future.

It's sad that the level of technical ability is so low that people adopt languages like clojure because they think its "concurrent". The JVM is not concurrent, therefore clojure is not concurrent therefore you're just signing up for a world of hurt.

The Erlang VM has been around longer than the JVM, it really is concurrent, and it really is battle tested.

First clue: the "we're so awesome we have to build our own infrastructure even when we're probably complete n00bs in our new hipster language" syndrome. Most of the dings that were laid out as "justification" were *operational nice-to-haves*; if your new environment isn't mature enough to have rock-solid operational support (and anything on the JVM really *should*), then you are fundamentally misunderstanding something.

Second clue: looking up to my address bar (in Opera 20.0/Mac) and seeing <a href="http://www.whitefirdesign.com/server-details">Server Details</a> reporting that the page was being served using an outdated version of PHP (5.4.4-14+deb7u2, which has numerous CVEs reported against it).

Third clue: no visible "Log in" or "Sign in" link, yet entering name and email in the "Add New Comment" fields produces a message stating that "The name you used belongs to a registered user." Score -1 for welcoming, openness, and transparency, guys!

And yes, the Erlang VM is very sweet. Call me back when it has the same depth and breadth of language and tool support as the JVM. Until then, I've more than enough Ruby projects (with various bits of Groovy, Ada, and Fortran thrown in for flavour) to keep me busy for a decade or six.

I gotta love the claim that, you shouldn't use a platform that actually works because there are more half assed attempts to get the platform that doesn't work to work. (aka "language and tool support".)

What do you mean when you say the JVM is not concurrent? The JVM has the concept of threads with higher level concurrency models built on top, such as the actor model with Akka. How can you say the JVM is not concurrent?

>The JVM is not concurrent
What do you even mean by "not concurrent virtual machine"? You can have concurrency in any language or platform you want, green threads are not that hard to implement. You don't need any "concurrent VM" for them.

>clojure is not concurrent
False premise → false conclusion.

And then there is the Erlang itself.
— dog slow "linear" code. You need to parse one MB of JSON? Wait… wait… wait… wait… done!
— oh, I know, there are NIFs! But now latency goes through the roof, because schedulers can't reschedule your thread while it's in native code. Boom, it's not at all better than good-ol-Apache. And you need to write in C, as if it's 80s or something out there.
— oh, you need to parse a date? No luck for you, 'coz there is nothing like strptime/strftime in standard library.
— you need to render some Markdown? Too bad. Please whip up some Python port just to render Markdown.

It's just plain incompetence to market Erlang as an universal purpose language, especially compare it to Java or Clojure. Hype like this hurts Erlang, because more and more engineers look at Erlang, get bitten by the lack of libraries and principal tradeoffs and abandon it with disgust.

Erlang is amazing, wonderful, brilliant technology. But it's very niche. If your app consists primarily of IO without any heavy data crunching, it fits. If not, Erlang will be a constant PITA.

> It relies heavily on very advanced usage of Java’s classloaders to manage this,
> which can often lead to the phenomenon known as classloader hell.

I disagree completely. OSGi actually manages to prevent classloader hell, since from the start every bundle is kept separate from each other. Ocassionally that leads to problems if you need the classloader from another bundle, but thats actually pretty rare. The big problems come from components which are doing their own dark classloader magic, and these are often difficult - sometimes even impossible - to integrate in a OSGi environment. In most cases it is a bad idea for those to do this anyway.

* w/rt OSGI: I definitely did not mean to imply that OSGi isn't a great technology. It's awesome for solving certain types of problems, and if you're working on an application that drags you into the realm of "jar hell" or "classloader hell", OSGi is clearly a good choice for helping to reduce or eliminate those issues. For our applications, we're trying to keep them small enough and modular enough that we can use the "uberjar" deployment approach, which is a different way to avoid all of the classloader issues. In addition, Clojure's normal interactions with classloaders don't quite align with the model that OSGi uses. In the end, OSGi just wasn't quite the right fit for us, but it's an excellent and mature tool that certainly has a legitimate, expansive domain of applications that it is well-suited for.

* w/rt Clojure: we've been using it heavily at Puppet Labs for over 2 years, and have several engineers on staff who've been using it for production applications for even longer. PuppetDB is written in Clojure and has been very successful in the field. We've done lots of performance testing and benchmarking against our Clojure applications and they've held up extremely well. Performance will continue to be a major focus for us in the coming year, and we are very confident that Clojure can deliver. Hopefully in the not-too-distant future we can put together a blog post comparing some of our Ruby server-side code with some of the Clojure versions.

* Scala and Erlang are great languages as well!

* Just to make sure that it's clear: this new framework is something that we intend to use for many of our new server-side products. For our client-side products (such as the Puppet agent), we recognize that the start-up time and somewhat heavy-weight nature of the JVM would not necessarily make it an ideal fit. So, don't worry, we won't be requiring the JVM on all of your Puppet nodes!

I'm a bit interested as to why Groovy wasn't picked. It's possible to do Functional Programming in Groovy, there's a lot of tools/IDE support, etc around and it would adoption as there's plenty of precedent in build ecosystems eg: Grails and Gradle.﻿

Hi! Thanks for the comment. Groovy is a great language, there's no technical reason why it wouldn't suffice. It's mostly a matter of us already having some momentum behind Clojure. Also, Clojure is a bit more opinionated about writing code that is purely functional and does not contain mutable state, and we think that those features will help us make sure that we have a very modular, maintainable codebase that is very amenable to multi-threaded application development.

At the end of the day, though, that's more of a matter of personal / stylistic preferences than any sort of shortcoming in Groovy, Scala, or other JVM languages. And who knows, in the future if we expose customisable extension points for the server-side apps, maybe we'll choose Groovy as a language for that--it's definitely more approachable to most developers who haven't spent much time in a LISP language!

it is ridiculous you mentioned JVM is the best, As a sysadmin, I found JVM is most horrible applications we have to take care of. each time JVM starts, it will use up all my servers' memory and drain the servers toooo slow. I am wondering why you have such a bad choice of programming languages. Why can't you code server side with C/C++, it is too hard to learn or too hard to code?