cprop

This week we learned of a recent alien hacking into Earth. We have a suspect but unsure about the true source. Could be The Borg, could be Klingons, could be Cardassians. One thing is certain: we need better security and flexibility for our space missions.

We’ll start with the first line of defense: science based education. The better we are educated, the better we are equipped to make decisions, to understand the universe, to vote.

One of the greatest space exploration frontiers is the Hubble Space Telescope. The next 5 minutes we will work on configuring and bringing the telescope online keeping things secure and flexible in a process.

The Master Plan

In order to keep things simple and solid we are going to use these tools:

As you can see the initial, default mission is the “Eagle Nebula”, Hubble’s state is stored on tape, it uses a mono (vs. color) camera and has in internal server that runs on port 4242.

Another thing to notice, Hubble stores an audit/event log in a Hazelcast cluster. This cluster needs environment specific location and creds. While the location may or may not be encrypted, the creds should definitely be.

All the above of course can be, and some of them will be, overridden at startup. We are going to keep the overrides in Consul, and the creds in Vault. On Hubble startup the Consul overrides will be merged with the Hubble internal config, and the creds will be unencrypted and security read from Vault and used to connect to the Hazelcast cluster.

Environment Matters

Before configuring Hubble, let’s create and initialize the environment. As I mentioned before we would need to setup Consul, Vault and Hazelcast.

Consul and Vault

Consul will play two roles in the setup:

a “distributed configuration” service

Vault’s secret backend

Both can be easily started with docker. We’ll use cault‘s help to setup both.

The only obvious thing to override is hubble/log/hazelcast/hosts since creds need to be later overridden securely at runtime, as well as the hubble/log/auth-token. In fact if you look into Consul, you would see neither creds nor the auth token.

The less obvious thing to override is the hubble/vault/url. We need this, so Hubble knows where Vault lives once it needs to read and decrypt creds at runtime.

We will also override hubble/log/enabled to enable Hubble event logging.

So let’ override these in Consul:

hubble/log/hazelcast/hosts to ["127.0.0.1"]

hubble/vault/url to http://127.0.0.1:8200

hubble/log/enabled to true

We can either go to the Consul UI to override these one by one, but it is easier to do it programmatically in one shot.

Envoy Extraordinary and Minister Plenipotentiary

Hubble relies on envoy to communicate with Consul, so writing a value or a map with all overrides can be done in a single go:

Secrets are best kept by people who don’t know them

Two more things to solve the puzzle are Hazelcast creds and the auth token. We know that creds are encrypted and live in Vault. In order to securely read them out we would need a token to access them. But we also do not want to expose the token to these creds, so we would ask Vault to place the creds in one of the Vault’s cubbyholes for, say 120 ms, and generate a temporary, one time use, token to access this cubbyhole. This way, once the Hubble app gets creds at runtime, this auth token did its job and can no longer be used.

cault, the one you cloned at the very beginning, has a script to generate this token. And supporting documentation on response wrapping.

We saved Hubble Hazelcast creds under secret/hubble-audit, so let’s generate this temp token for it. We need to remember the Vault’s root token from the “Vault init” step in order for cault script to work:

eda33881-5f34-cc34-806d-3e7da3906230 is the token we need, and, by default, it is going to be good for 120 ms. In order to pass it along to Hubble start, we’ll rely on cprop to merge an ENV var (could be a system property, etc.) with existing Hubble config.

In the Hubble config the token lives here:

{:hubble{:log{:auth-token"OVERRIDE ME"}}}

{:hubble {:log {:auth-token "OVERRIDE ME"}}}

So to override it we can simply export an ENV var before running the Hubble app:

The first step is made

Most of the tools I push to github are created out of something that I needed at the moment but could not find a good alternative for. cprop was one of such libraries. It sat there on github all alone for quite some time, and was used only by several people on my team, until it was integrated into Luminus.

Suddenly I started talking to many different people who found flaws in it, or just wanted to add features. I learned a couple of interesting usages from Heroku guys, as well as the importance of merging creds with Vault, coexisting with configs from other fault tolerant and external services such as Consul and more.

One of the useful cprop features is merging configs from various sources. Which is quite an open extension point: i.e. once cprop does its work and comes up with an app config, you can decide how and what will be merged with it before it really becomes a thing. It can be a local map, a .properties file, more ENV vars, more system properties, more configs from anywhere else, including remote/external resources, result from which can be converted to an EDN map.

To enable this merge extension point cprop has several tools that in practice could be really useful on its own: i.e. can be used outside of the (load-config) scope.

Loading from various sources

Could be used as OS, file system and edn oriented I/O tools. Also quite useful in the REPL.

Converting for other sources

Most Java apps store their configs in .properties files. Most docker deployments rely on ENV variables. cprop has some open tools it uses internally to work with these formats to bring EDN closer to non EDN apps and sources.

EDN to .properties

(require '[cprop.tools :as t])(t/map->props-file config)

(require '[cprop.tools :as t])
(t/map->props-file config)

Converts config map into a .properties file, saves the file under temp directory and returns a path to it.

.properties to one level EDN

Besides the from-props-file function that converts .properties file to a map with hierarchy, there is also a slurp-props-file function that simply converts a property file to a map without parsing values or building a hierarchy.

The first programming language I learned was Basic. It was sometime ago, and my first computing beast at the time was ZX Spectrum.

I was in school, then another school, then college, then another college, etc. All that time application configuration was not really a thing I cared about. Who needs to create programs for different environments? Who needs pluggable resources? Why would you ever care about it?

A program that plays checkers, I wrote in Pascal in 1993, certainly did not need it. A text editor written in C a couple years after did not need it. A program in AI class in college, that created travel plans for robots with various sensors did not care about dev/qa/prod, since it was always “just prod”.

Then I suddenly started to make money creating programs. Well, not that suddenly, but it was a really strange feeling at first:

I can just do what I love, and also get paid?

Sweet.

All these people, so much care

But something did change in the way I approached creating software. All of a sudden it wasn’t just me who cared about programs I create, but other people too. In fact they cared so much that they were ready to give me their money.

But not just people, businesses too. Which meant I could no longer empower those businesses with my daily brain dumps the way I created programs before. At this point I should have had organized my thoughts in these polished blocks of software that had to be confirmed by other people before these blocks see the light.

Creating vs. Nurturing

Usually developers do not take configuration seriously. A large number of developers “are here to create software not to nurture it”. Some people prefer one set of patterns, others another set. We jam our ideas in one of those sets, and then “have to go” through this “bleak” DevOps phase to make sure our creation can survive. Sure Docker made it more fun, Consul made it more convenient, Ansible made it saner, but it is still “so far remote” from what developers love the most: creating.

Since besides programming I’ve always liked hardware and operating systems, DevOps intrigues me not a bit less than programming. But most of the software developers I’ve interacted with prefer others, specially dedicated, people to do DevOps work for them. Not just developers, all the larger organizations I’ve worked at, in addition to development, have build, test and operations teams.

Configuration. Connecting People.

It is interesting that the only true common ground that technically connects developers to build, test and operations people is configuration. Great documentation and communication help, but configuration connects.

The strength, quality and flexibility of this connection amplifies happiness of all the people involved in this enthralling journey from inception to production.

Development and Structure Factors

12 Factor clearly states that configuration should live in environment variables:

“The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.”

Makes sense. But for me:

* it makes development harder
* it is simply not needed for certain apps I work on
* environment variables lack structure

But it does not mean nothing can be done to have both: 12 factor and the solution for these three points above. We are programmers, we create things.

The Golden Goose

I’ve done quite a few apps in Java over the last decade, and I know a thing or two on how to bridge the gap between applications and configurations.

In Clojure universe, the most popular library to manage configs is environ. Normally apps would have their properties live in lein profiles / .lein-env / .boot-env and could be overridden by environment variables and system properties. Currently this is the way to manage configs in Clojure.

This is not to say environ got it wrong, or it is no good, it is really useful and works great for a lot of people, but I feel I need an alternative, because of the 4 things above, and simply because I like to have alternatives. So I wrote one.

12 Factor + Some Love

My take on configuration and properties is called cprop. It is young and brave, and currently is used by me in several applications, a few others, and it is a default configuration approach in Luminus which is a micro-framework that is based on a set of lightweight libraries.

Instead of repeating cprop documentation, it would be important to notice that the factor number III out of 12 factors, which is called “Config“, is fully supported by cprop, while still allowing for more flexibility.

For example let’s take a concept of “structure” or “hierarchy”. Say you have a config in development (i.e. somewhere in dev-resources):

Looks like it passes the 12 factor litmus test which says that “the codebase could be made open source at any moment, without compromising any credentials” while preserving the structure.

There are several ways to provide real values for this config with cprop, such as other maps merged at runtime, system properties, runtime arguments, etc.. but the most interesting way, from the 12 factor point of view, is “environment variables”. Which can be simply done with cprop as: