https://blog.venanti.us/https://blog.venanti.us/favicon.pngVenantiushttps://blog.venanti.us/Ghost 2.9Tue, 18 Dec 2018 11:06:49 GMT60Every year I try to write a little something - a manifesto for the year. I did so in 2014, and again in 2015. I skipped last year, largely due to being busy and having too many irons in the fire.

It will be a year of focusing on a single thing: getting the new venture off the ground. This means fundraising (a lot), building a founding team, establishing a foreign subsidiary, moving abroad, and much, much more.

It will be the biggest and hardest thing I have ever done in my life, and it will require my complete attention. There's nothing else I want to do more, and I've spent a long time preparing for this.

Public toilet seats

I just got back from a month of traveling in the UK and in Italy. What follows are some miscellaneous thoughts about traveling, Italian culture, and Italian cities.

Italy

Public toilet seats

It turns out that Italians have what I would describe as an insane fear of public toilet seats. This causes people to stand on the toilet seat to use the bathroom (I'm not making this up), which ends up breaking the toilet seat. Eventually the maintainer of the bathroom stops replacing the toilet seat, which means most public Italian bathrooms in restaurants, bars, etc do not have toilet seats.

Note that a few Italian bathrooms actually do have proper squatting toilets. I don't love squatting toilets, but I think they are preferable to having normal toilets with broken toilet seats in which you have to squat. The latter use case is not what the instrument was built for.

Trash

The Italian garbage system is complicated, and particularly complicated in places like Venice. Recycling is collected every other day. Food waste is collected every other day - the complement set of the recycling days. Normal garbage is collected every day. No garbage is collected on Sunday. Why couldn't you just have one day for collecting garbage?

The answer seems to be that the trash bags used are so small that they need to be frequently taken out. This relates generally to a lack of space, and certainly to a lack of space for garbage. I'm a fan of using less space, but not of having to remember complicated garbage rituals. Also, the use of small and somewhat weak trash bags invariably means getting trash on your hands, which is hardly a charming experience.

Hot water

The only place we stayed that had enough hot water for us to take a shower lasting longer than about 8 minutes was in a mansion maintained by the British. In Venice, this was understandable, but it was harder to rationalize at the two different places we stayed in Rome.

If you are traveling as a couple, this means that in practice you end up taking showers at very different times of day.

Food

The food is really good. People had told me before that pasta was different in Italy and they were correct, although in ways that are hard to explain. I think the best I could do would be to say that pasta in Italy is way less starchy, and that has profound implications for how it tastes. I left feeling like pasta in America sucks and we should stop serving it to people and acting like what we're doing is okay.

Food was also, generally speaking, very cheap - though considerably more expensive in Rome. Also, this didn't warrant a longer section below, but I had the best pasta Carbonara of my life in Rome. I did not realize you were expected to drown the pasta in pepper but boy, did it make for an amazing meal.

Alcohol

I don't know what's going on in the Prosecco market these days but even while we were in England shops were basically fighting to give us cheap Prosecco. I think the average price we paid for a glass of Prosecco was probably something like 4‎£/€ and there were quite a few places in which it was cheaper.

I drank a lot of Prosecco.

I also did not realize how huge Campari / Aperol spritzes are in Italy - you see people drinking them everywhere, particularly Aperol spritzes.

Clothes

Everyone seems to wear clothes that actually fit them. I don't think this is a consequence of stores selling clothes of many sizes, but rather think this is a consequence of living somewhere where it is simply an expectation that you will have your clothes tailored.

Regardless of the ultimate cause, it makes Italian people look much better dressed than their counterparts in the U.S. and U.K.

Mosquitos

Coming from the San Francisco Bay Area, I try to act like mosquitos don't actually exist until one gets trapped in my bedroom. This is, unfortunately, not an option in Italy.

For some reason Italy has not adopted the use of screen windows, so if a window is open then it is a potential attack vector for mosquitos. The Italians compensate for this somewhat by aggressively using electric mosquito coils, which do an okay job of remediating the problem, but not great.

We purchased some of these mosquito sleeping bag liners from REI before going and they made a huge difference at night (though I did have one nightmare scenario where a mosquito was trapped on the inside of the liner and bit my legs about 5 times).

Venice

Logistics

While I was obviously aware that most of Venice's transport happens by boat, I was not aware that there are literally no roads for cars in Venice.

The mail, and packages particularly, are seen being carted around by people using dollies everywhere. If you order a package through Amazon.com, it will arrive by boat, be loaded onto a hand cart, and someone will walk that cart through the streets of Venice until they arrive at your home.

The lack of roads also means that custom machinery ends up being used that can either (a) move along Venice's narrow streets or (b) be completely dis-assembled, transported by boat, and re-assembled at the destination. As construction obviously takes place in Venice using modern means, we saw examples of both of this - the former on the renovation work at the Rialto bridge, and the latter at various construction sites.

One of the interesting implications behind both of these is the level of additional expense that must be layered onto all logistics activity in Venice - no standard logistical infrastructure can be used, so everything ends up being a custom solution that the Venetians have come up with for the task at hand.

Food

In general, I was a fan of Venetian chiccheti. I was a particular fan of the local way of serving sardines - in oil, with raisins and grilled onions. It sounds like a bizarre combination but it works together very well - a savory dish with a slightly sweet tone.

There are a lot of touristy restaurants that both can and should be avoided. The best and most interesting meals we had in Venice were ones that were somewhat off the beaten path and in some cases came recommended by locals.

Aesthetics

Venice is stunningly beautiful. It's basically impossible to point a camera at anything and not have the photo come out looking gorgeous.

While the more touristy parts of the city happen to include some of the most exceptionally beautiful parts, I really loved the quieter neighborhoods like Castello, Dorsoduro and Cannaregio as well. There isn't as much to "do" there, but they're no less beautiful than the more central parts of the city.

Rome

Driving

Do not drive in Rome. I do not know what they make Roman drivers out of, but it is harder stuff than is available over the counter in the United States and I cannot recommend it. Roman streets have no lanes, they just have cars doing whatever they feel like and miraculously not experiencing consequences.

Aesthetics

It took me a while to figure out why central Rome seemed somewhat austere at times and I eventually nailed it down to a combination of low amounts of graffiti coupled with an almost complete lack of advertising billboards. The latter has a pronounced effect but takes a while to figure out, and you don't really notice it until you leave and start seeing advertisements everywhere again.

Unlike Venice, which is uniformly beautiful, Rome is divided into somewhat unattractive buildings and works of extreme awe and majesty. As you walk around central Rome the frequency with which you encounter awesome works goes up until you are constantly looking at works of such grandeur that your mind cannot really comprehend it. Photographs do not work so well in these circumstances - there is, for instance, no real way to take a photograph that captures the scale of the Altare della Patria, nor the fact that the Roman Forum lies immediately behind it.

One of the things that occurred to us as we were walking around Rome is that being surrounded by buildings of this sort would undoubtedly contribute to a shared sense of manifest destiny. It's not hard to imagine how the literal structural environment of Rome could have contributed to the 20th century rise of Italian Fascism.

One of the things I've been struggling with lately is how to go from being a senior or somewhat experienced engineer[1] to being an exceptional engineer[1:1].

To be clear, I'm not talking about reaching a level of global fame (i.e. on the same level as Linus Torvalds, Guido van Rossum, Rich Hickey, or whoever else you idolize). What I am referring to, more generally, is how to get to a place in which one is recognized as having extraordinary skill.

I started thinking about this more recently in response to a passage in a book I've been reading:

With some tasks, we stop short of our highest level of proficiency on purpose. The calculus we perform in our heads suggests that the added effort it’ll take to find and learn something new will probably yield a diminishing marginal return, so we stop learning. For instance, we learn how to make use of a word processor or Web server by mastering the most common moves, but we never learn many of the additional features that would dramatically improve our ability.

When this same pattern of arresting our development is applied over an entire career, it yields fairly unsatisfactory results. For example, most professionals progress until they reach an “acceptable” level, and then they plateau. Software engineers, for instance, usually stop progressing somewhere around five years after entering the workforce. Beyond this level of mediocrity, further improvements are not correlated to years of work in the field.[1:2][1:3]

I was eventually able to track the source of this statistic to the research of [Dr. K. Anders Ericsson](Dr. K. Anders Ericsson), but in trying to locate it I ran into another interesting finding: that the correlation between IQ and performance in a domain decreases over time, and that after more than five years of professional experience the correlation ceases to be reliable.[1:4]

Put together, the two suggest the following: continuing to do what you've done in the past will eventually fail to net any further improvement. Similarly, intellect alone will only help you get so far. For reasons that are as likely to be co-incidence as anything else, both intellect and what I think of as the brute force approach to skill acquisition both cease to be meaningful contributors to one's growth at around the same point in time.

I like to call everything after this point the "endgame".

I'm not using the term in the chess sense of the word, in which there are relatively few pieces left on the board. Rather, I'm using the term in the massively-multiplayer online gaming sense of word, referring to a class of gameplay only available to characters who have otherwise completed the game's level progression.

The endgame's unprecedented difficulty is accompanied by structural differences from what has preceded it. It introduces new modes of (self-contained) progression, in which advancement requires spending time in a very different manner from what brought you to the endgame in the first place. The endgame is typically impossible to reasonably approach until you are prepared for it.[1:5]

For many people, the endgame is where the true meat of the game lies - it's a place in which simply logging time is necessary but no longer sufficient. Actual progression within the endgame requires both skill and deliberate focus on an entirely new set of activities.

In the real world, most people seem to level off when they encounter the skill endgame. They're either unwilling to invest further, satisfied with their level of expertise, or they're unable to adapt their methods of practice and learning in a way that will enable them to grow further.

It's hard to know what the exact breakdown of motivations is. For most people, I would hazard a guess that the individual in question is presenting themselves with a false choice: "work harder, or keep the status quo". Presented with such a choice (and, often, competing interests for time - at this point in their career the person in question may well have started a family), they make what seems to them to be a rational decision and choose to invest their efforts elsewhere.

For those that don't fall victim to the false choice, some number will want to advance but won't know how (either because the information is unavailable, or the resources are). Eventually, demoralized, they'll likely switch to a different focus area within the field - or perhaps transition to a different field entirely - in order to continue experiencing intellectual growth.

Some final number will have both the motivation and a clear idea of what would be necessary to progress, but will find themselves blocked by the structure of the institution they're operating in. This tends to happen in less flexible corporate cultures, or in cases where the individual has wound up on the losing side of a larger political battle.[1:6]

Measuring Skill

Part of the reason so many people plateau may be that it is simply not clear at what point one has entered the endgame. There are multiple reasons for this, but the most important is the difficulty in getting a clear indicator of one's own skill. Skill estimation is a notoriously hard problem to solve, as a considerable body of research has shown.

Assuming such a thing as one's "true" skill exists, the associated challenges entailed in measuring it (let alone decomposing it into constituent parts) are non-trivial. For starters, individual traits may play a very different role in one's overall skill, depending on the nature of the skill itself. To what degree does intellect play a part? Emotional intelligence? Experience? Environment? Economic incentive? How different do the answers to these questions look when we talk about the skills of an athlete vs. a software engineer?

To begin, we should dispatch with the notion of a single linear skill tree - even the term "tree" is misleading, or only relevant insofar as the roots of the tree are just as vital as its branches. When thinking of skill, it is important to be considerate of tracking progression in many directions at once. Part of the difficulty in thinking about skill lies in how diverse talents can coerce into measures like speed or effectiveness.

I'm not anything close to an expert in this area. An exhaustive review of the literature here would be well beyond the scope of this post (as well as my knowledge). Skill measurement is widely understood to be an objectively hard problem.

That having been said, there are certain heuristics that can be helpful. When you're first starting out with a new skill, you know next to nothing, and so learning almost anything makes you better. For a diverse skill arena like software engineering, there are many different parts of the ladder you can begin climbing, any of which will have a material effect on your overall skill. Over time, you begin to acquire enough of a common lexicon (due to shared patterns across specializations) that you're able to reason about the difficulty of issues in a different focus area from your own.

It's not too long after this point that measuring your skill in relation to others becomes more complicated. This tends to happen most in the early-to-middle ground of most people's careers; a point at which they've acquired some specialized knowledge, and a fair amount of generalized knowledge, but not enough context to understand the depth of other people's experience in their own specializations.

This balkanization into specializations is a particularly hard problem, and one I don't have any prescriptions for. My focus here is not on how to compare skills across specializations but rather to focus on narrow depth; I am not trying to tackle the problem of comparing a database expert to a compiler expert.[1:7]

My experience with attempting to understand my position on a skill tree has been that it becomes easy to see and define the nodes behind you, but only possible to grok one or two nodes beyond you. Nodes greater than one or two away tend to fall into one of two categories: they either seem unapproachable, or seem to be not so different from nodes you already understand.[1:8]

This sort of relative heuristic offers us little by way of absolute identity, but turns out to have extremely practical implications as far as choosing habits that enable us to continue to grow.

Growth Habits

In thinking about how to reason about growth habits in the endgame, a good place to start is on habits that are equally valuable in the endgame as outside of it. With this in mind, the best advice I have come across is to find someone who is just slightly further along than you are and to ask them for help.[1:9]

It's important that any mentors not be too many nodes away from you, otherwise they won't know or be able to remember how to approach the problems you're facing with the appropriate context. Conversely, it's only so helpful to ask the advice and help of people currently at your level - they're trying to figure out the same things you are!

If most people plateau at a certain point, however, then by definition the further you go up the ladder the fewer people there will be above you who can help you. Further complicating things is the reality that people well into the endgame may have significantly different specializations from yours. This is where being in a well-networked community (e.g., Silicon Valley) or a large company (Facebook, Google, Microsoft, etc.) can help you. Most populations are normally distributed, so if you want to find someone in the tail, your likelihood of finding them increases as the size of the population goes up.

Of course, people who are well into the endgame typically do not possess enough time to help all of the people who approach them for help. They are likely to be asked for help and advice by many people, a good number of whom will be considerably below them on the skill ladder. You can only imagine how many emails Linus Torvalds gets from college students.

This means that not only are there very few people at the higher rungs of the ladder, but they're also overburdened by requests for help. Getting around this is itself a hard problem, and begs the question of what obligation experts have to share their knowledge as opposed to deepening it.

I don't have a generalizable system for identifying people with only one or two rungs above you and approaching them for help. I wish I did, because then I would spend less time thinking about this and more time trying to learn from them.

If you can't get time with people who are at the right skill level, the next best thing you can do is to read things they have written.

Blogs are probably good when it comes to people high on the skill ladder - if they're at the right point in their career they won't be spending that much time writing, and the things they publish will be fairly high quality. If they're publishing something on their blog more than once a week, they're probably more focused on writing than they are on the skill you're trying to learn from them.

Books are better than blogs, for three reasons. First, the format of a book allows for considerably more depth of engagement. There are more things in heaven and earth than can be covered in a blog post, and while a book has its own limitations, it gives more time to examine a subject more seriously.

The second advantage of books is in their mode of engagement. Reading a book forces you to disengage from other distractions to focus more fully on it. Studies indicate that not only are we all generally terrible at "multi-tasking", but that the path to true mastery of a subject demands full attention for regular, brief intervals.

The final advantage of books is that we have an established social mechanism by which we can assign and estimate the value of their content - the book review. While you can try to judge the value of a blog post by its comments, it's not quite the same as having (sometimes long-form) reviews to read to contextualize the quality and content of a book.

There are other decent forms of written media as well. Depending on the nature of the skill, there may be guild publications you can subscribe to. Journal articles can be an exceptional source of new insights and depth, though their specialization of focus can make discovery and accessibility a problem.

If you're lucky, you can find curated lists of the best articles and papers for your domain area (for instance, this page has the Best Paper awards for Computer Science in various focus areas going back to 1996). This sort of resource is invaluable, and also enables you to see what depth looks like in closely-related specializations to your own.

The final thing you can do is to deliberately take on projects that you know to be hard. Hardness here is contextual - what you regard as hard may not be hard for others, and similarly there are tasks you once regarded as hard that may no longer be so.

Growth under this strategy requires you to attempt something you consider a considerable stretch of your known capabilities. This is a different sort of brute force approach to skill acquisition; it isn't the same as continuing to do the same things you've done in the past is, but it is still a naive approach and much more likely to result in failure.

The danger here is that you may not be able to learn from your failures without some sort of guidance, which is why this approach in isolation is likely to be less valuable. When paired with a good mentor or other supporting resources, however, taking on extremely difficult projects can have a multiplicative effect on your rate of growth.

One key benefit to taking on the right project is that the outcome can lead to feelings of accomplishment and satisfaction - even in the failure case! By contrast, mentorship and education don't always trigger the same reward centers, which can make them less motivating.

Deliberate Practice

The quote that inspired this article comes from a book that counsels that deliberate practice is what enables you to keep growing when others stagnate. Deliberate practice is the research focus of Dr. K. Anders Ericsson and has been covered somewhat extensively in U.S. media. Deliberate practice is defined as having the following components:

Requires full attention for brief intervals

Provides immediate feedback against a clear standard

Breaks mastery into small goals

Prepares for setbacks and builds in resilience

The biggest takeaway from the research into deliberate practice is the insight that neither more time nor more effort are necessary (though you do need to be able to commit to dedicated periods of focus). To reduce it to its most trite and banal: "work smarter, not harder!". If you're logging tons of time trying to improve and you're not noticing a change, you're probably doing something wrong.

Extreme skill - or at least, extreme skill as it currently exists on a normal distribution - should be within the grasp of almost anybody. If this sounds unlikely, there's actually a fair amount of historical evidence that suggests that we as a civilization have become much better at deliberate practice over time, shifting the entire curve to the right. For instance, in Ericsson's book Peak, he talks about this in the context of music:

In the early 1930s Alfred Cortot was one of the best-known classical musicians in the world, and his recordings of Chopin’s ‘24 Études’ were considered the definitive interpretation. Today teachers offer those same performances — sloppy and marred by missed notes — as an example of how not to play Chopin, with critics complaining about Cortot’s careless technique, and any professional pianist is expected to be able to perform the études with far greater technical skill and élan than Cortot. Indeed, Anthony Tommasini, the music critic at the New York Times, once commented that musical ability has increased so much since Cortot’s time that Cortot would probably not be admitted to Juilliard now.

Of course, it's easy to take that as a feel-good soundbite and go about one's day, but much harder to actively push the edges of the curve right yourself. For instance, how does one take the abstract requirements of deliberate practice and turn those into concrete activities for a skill such as software engineering?

Some of the things covered in this post clearly fit the requirements. Others, e.g. mentorship, do not. Some of the requirements themselves are hard to apply to technical work. For instance, what does it mean to break software mastery into small goals? What clear standards exist to measure oneself against?

I've been thinking about the skill endgame in one form or another for quite some time, and in a near-obsessive manner for the past few months. Sitting down and starting to write about this has led to a lot of thoughts coalescing (it has also, unsurprisingly, unearthed much more consensus than I had anticipated). Something to be said there for deliberate focus, after all.

At this point, for all of my musing, the one thing I'm left wondering is what else I'm missing. Not the obvious things - absent skills or missing practices - but the unknown unknowns, to quote Donald Rumsfeld; the structural insights about the nature of improvement that haven't surfaced yet.

If you've also been thinking about this for a while and feel that you're in a good position to provide guidance on the skill endgame, I'd love to hear your thoughts. Whether you're an expert yourself, a veteran in the industry, or someone coming from a completely different background - let's get a conversation going. I'd love to know what I'm missing.

]]>https://blog.venanti.us/what-im-reading-may-2016/5a846bf2ee01a700184f02b9Sat, 07 May 2016 07:14:34 GMTMy friend Keith Ballinger used to post on his blog about once a month with a list of what he was reading.

Keith recommended this to me when I asked what he thought the best book on distributed systems was. It's a little dated at this point but most of the patterns and practices are still very applicable. Funnily enough, this is also one of the current books in a small infrastructure engineering book group at Airbnb that I'm a part of.

This was recommended to me by my manager, Clint. It's written by the same people who wrote Crucial Conversations, which is probably the best book on communication I've read so far.

I'm about halfway through it - it's definitely quite good. I'd say Crucial Conversations is about how to get what you want out of personal interactions, and Influencer is how to accomplish systemic change. The books are very complementary.

What can I say? I rented and watched the recent movie with Tom Hiddleston and I thought it was so weird and fascinating that I wanted to read the book.

It's not going to change your life, but it's fun reading. TL;DR - a novel about a high rise apartment complex falling into total anarchy.

]]>I was working on a pretty big post last year when I was sidelined with a new job and started writing a book, Learning ClojureScript. Working on the book has given me a lot of opportunities to explore areas of the ClojureScript ecosystem that I wouldn't otherwise have explored, and]]>https://blog.venanti.us/using-transducers-with-core-async-clojurescript/5a846bf2ee01a700184f02acThu, 21 Jan 2016 17:41:49 GMT

I was working on a pretty big post last year when I was sidelined with a new job and started writing a book, Learning ClojureScript. Working on the book has given me a lot of opportunities to explore areas of the ClojureScript ecosystem that I wouldn't otherwise have explored, and some of the learnings that have resulted have been pretty cool.

Specifically, two of the things I've been learning about lately are core.async and transducers. Core.async is a highly popular Clojure(Script) library for embedding CSP-style concurrent program design in libraries and applications; transducers are composable functions useful for performing transformations on data.

There are a number of articles about core.async already floating around on the internet, but transducers appear to remain largely misunderstood. Speaking personally, I've have had quite a bit of trouble wrapping my mind around them, and one of the things that has helped me tighten up my understanding has been to get a better understanding of their specific use cases.

In general, I think the fundamental insight for understanding transducers is to realize that their application is not so different from other higher-order functions, but that they're applied to slightly different (though no less valuable) contexts. Thus far, the most awesome examples I've seen have been ones in which transducers were used in combination with core.async to build data transformations into asynchronous application message queues, so that's what I'll focus on here.

Let's get into it.

Error Handling Patterns

One of the advantages of core.async is that it encourages programs with a clear separation of concerns. Data is produced in one part of your application, and consumed and processed in another.

This separation of concerns has a drawback, however. If we have code that expects to be able to read from a channel, what happens when errors occur on the input side of the channel? We could catch and handle errors on the input side, but there are a number of circumstances in which that might be undesirable. For instance, what if there are many possible places in the codebase in which we put value into this channel? It is not unreasonable to imagine that we might want a central pipeline for handling errors.

There are a few different ways of going about this. One possibility would be to create a separate channel for error messages and to have an independent listener for that channel (a pattern discussed on the ClojureScript mailing list here).

Let’s assume we have an application where we're regularly polling the kitten factory to make sure production is running smoothly. If things are running smoothly, we'll create a report, and if they go badly, we'll log that to our exception tracking service.

Side note: What even happens at the kitten factory? Is it a factory that makes kittens, or do kittens run the factory? If they’re running the factory, what are they making – balls of yarn? I have trouble imagining this is a very efficient factory.

An alternative pattern that we could embrace would be one in which we passed the errors themselves into the channel and checked to see if a message was an error when consuming. This is a pattern that David Nolen has written about previously on his blog here.

If you read his post, he refers to a <? macro, the source code for which is available on his blog's GitHub repo. The <? macro just checks to see if the value that’s been pulled off of the channel is an error, and if it is, throws. Without this, we’d just be passing an error around by value and the compiler would have no reason to actually throw the error (thereby preventing us from catching it, as well).

We'll copy the relevant code from his blog's GitHub repo into our app, modulo a few changes:

This is a perfectly good solution, but do we really need that <? macro? This is a great chance to use a pure function (a transducer!) instead. Transducers can be passed as an optional argument to the chan constructor function, and they'll perform the given transformation on any values passed through the channel.

This means that we have an opportunity to write a transducer for our channel that just maps our helpful throw-err function over values being taken from the channel. Let's see what that looks like:

Let's take a look at another example - this one courtesy of my friend Allen (@arohner), with a few modifications.

Batching Data

Let's say we've got a similar setup to earlier, but instead we want to batch up at least 10 events before we send a report back to cat hq. If we were going to design this without transducers, we'd probably do something like the following:

This could probably be cleaned up some more, but at the very least it feels like we're doing more work than we should be. By contrast, if we were to use a transducer to batch these up, it would look like the following:

This is unquestionably much cleaner than the version without transducers. As Allen put it: "You put one kitten in, and take ten out. You can't explain that."

Conclusion

The thing that stands out to me about transducers is that they empower us in a way that is simple. As a language feature, it's easy to imagine an implementation where performing data transformations on asynchronous streams could have been over-engineered and a nightmare to write code for. Instead, Clojure(Script)'s transducers are easy to read, write and reason about.

There are plenty of other applications for transducers that I haven't talked about in this post, but for me, the core.async tie-in is where their value was first made clear. If you think transducers are n.b.d. and can explain why I can't find that image of Rich Hickey pasted onto the guy from Ancient Aliens on Google, feel free to let me know.

Thanks to Keith Ballinger (@keithba), Bill Cauchois (@wcauchois), Katherine Geenberg (@MyWolfyMaw), and Allen Rohner (@arohner) for not only reading drafts of this post, but for adding to it.

Finally, a shameless plug: if this is the sort of thing that interests you and you're interested in learning more, pre-order a copy of the book!

]]>I spent most of my labor day weekend experimenting with ClojureScript. The last time I took a serious look at ClojureScript was about 18 months ago, and the language and its ecosystem have unquestionably come a long way. There are some really, really cool projects being worked on - Figwheel]]>https://blog.venanti.us/clojurescript-blues/5a846bf1ee01a700184f02a7Tue, 08 Sep 2015 15:01:42 GMT

I spent most of my labor day weekend experimenting with ClojureScript. The last time I took a serious look at ClojureScript was about 18 months ago, and the language and its ecosystem have unquestionably come a long way. There are some really, really cool projects being worked on - Figwheel automagically recompiling and pushing updated client-side code to the browser on write and some of the work around maintaining globalapp state come to mind.

At the same time, there remains much about where ClojureScript currently finds itself that I find deeply, deeply frustrating.

This post is a documentation of some of my frustrations. My hope is that perhaps some of those of you reading this will have enough expertise in the matter at hand to show me that my frustrations are misdirected. Failing that, perhaps this post will serve as an inspiration to address some of the problems I've encountered.

Compilation Issues

I encountered an extremely irritating issue with the ClojureScript compilation step of my project's Uberjar compilation, wherein cljsbuild wouldn't re-compile my ClojureScript even though my project's :uberjar profile included a number of different compilation options. Critically, this resulted in shipping development JS (i.e., figwheel websocket connection logic) in the uberjar's javascript, with no clear way of avoiding the problem outside of manually deleting the file in question prior to attempting to compile the uberjar (which does work).

In many ways this first issue is the most damning of all of these, because it's the sort of thing that should have been resolved by now. Failures in really basic use cases of your build system are a very bad sign.

Lack of Templating Support

This is an ecosystem issue: I find it very irritating that there isn't greater flexibility in templating languages for use with ClojureScript currently.

That is to say: most of the libraries in the ClojureScript ecosystem expect you to use their custom DSL for rendering DOM elements. For instance, Om has om.dom, Reagent has a Hiccup-style templating structure, etc. To be sure, in these cases this emerges as a function of the need to work with React, which means less interaction with traditional client-side templating languages like Jade.

Even still: there exist libraries like sablono and kioo that provide convenient template intermediation via Clojure-familiar enlive or hiccup-style data structures, but if I wanted to keep my templating logic apart from my application logic there's still little support for that.

Maybe I'm crazy for wanting to have a _partial.jade that compiles to enlive data structures that I pass to Om and populate with my application data, but some part of me thinks that sounds like a better separation of concerns than the way things currently stand.

The Figwheel REPL

Let's get one thing out of the way first: Figwheel is totally awesome. The entire reason I chose to spend my weekend fiddling in ClojureScript was that I found myself last week having dinner with Daniel Woelfel and he told me that Figwheel (which didn't exist back when I was first fiddling with ClojureScript) had made the experience radically better. He was right - Figwheel is great. All the same, there are no sacred cows in this post.

In general, the Figwheel REPL is a big improvement over past ClojureScript REPLs, which were prone to hanging irrevocably. However, it has a number of attributes that remain frustrating, including the following:

Lack of tab-completion

Keyboard interrupts kill the entire REPL (and thus the entire Figwheel process), though in a normal Leiningen REPL a keyboard interrupt can be used to just clear the current REPL input and get to a new input line

I am also sad that it doesn't provide a convenient way for me to inject middleware into the ClojureScript REPL process, which would enable me to do things like syntax highlighting, etc.

Documentation Re: Server-Client Paradigm

I encountered an almost total lack of documentation and examples for realistic client-server applications. Most behaved as if either (a) you were expecting to write both client and server code in compiled ClojureScript, or failing that (b) you had little to no intention of interacting with a server whatsoever.

There is some really fascinating thinking going on in the industry at the moment around abstracting away from REST as a particular implementation of client-server relationships. Netflix' Falcor and Facebook's GraphQL start to get us to a place in which you don't necessarily have to care about including RESTful client-side examples. Similarly, the DataScript ethos of "load our application state once, then deal with it all client-side" is similarly intriguing for a surprisingly large range of use cases.

Even so, I think it's naive to presume that we're at the point where we can act like the server doesn't exist. If you're working in ClojureScript today, please include in your documentation examples that show me how I can write clients that are good actors in a modern client-server world. Don't make me have to figure this stuff out for myself.

Vim / REPL Blues

I'm always going to be in the minority as a Lisp/Clojure person working in Vim. All the same, it sucks to see really second/third-class support between ClojureScript, the Clojure REPL, and Vim. Especially when Figwheel's involved, the whole thing becomes a mess quickly:

Add a :nrepl-port option to your :figwheel key in project.clj

But wait! Figwheel doesn't handle creation/deletion of an .nrepl-port file the way Lein does, so now you need to create it yourself since otherwise you'll be manually :Connect-ing from Vim.

Okay, cool. Well now reloading Clojure files work when you're running Figwheel.

But not ClojureScript. No, for that you'll need to install Austin.

Or is it Piggieback?

And then you do what to configure it? You need to run a manually patched version of vim-fireplace? Add some extra lines to your .vimrc?

Why isn't there a single straight answer for this?

And why doesn't vim-fireplace just support this out of the box again?

On some level these complaints seem a little superfluous when all of your client and server code is configured to automatically hot-reload (either by pushing the client-side code to the browser or automatically reloading the server-side namespaces with each request). Even so, it's a real pain in the ass when you're a highly interactive developer and you value the power of being able to work from REPL.

Chestnut: Template v. Plugin

I have a lot of appreciation for the creators and maintainers of Chestnut for providing an excellent scaffolding upon which to build a ClojureScript application. However, I can't say that I'm a fan of the greater pattern of providing templates over plugins. It punishes those who might already have a significant Clojure application and are interested in migrating to a ClojureScript frontend for by forcing them to manually merge the two codebases, and also forcing them to add a considerable amount of configuration boilerplate to their project.clj

To that last point, this is the basicproject.clj for a new Chestnut app:

Doesn't that seem like a lot of new configuration to you? Not to mention the fact that you now have all of this weird additional source code in your application in weird places (env/dev, env/test, env/prod).

My strong opinions on this subject are the result of having gone down this path before - it was the whole reason I wrote Ultra, after all. To quote myself:

...or, why didn't you just put all of this stuff in your ~/.lein/profiles.clj?

In short, my :user profile was starting to become bloated. It was difficult to tell whether plugins were interfering with each other, and my :injections key in particular was starting to look a little unwieldy.

At some point I realized I was up to my neck in alligators and that it was time to push things into a standalone repository.

I think that Chestnut, and the ClojureScript community at large, could really benefit from a single Leiningen plugin that just handles all of this stuff for you and provides simple, sane, obvious defaults, that you have the power to overwrite yourself in your project.clj, if you find you need to. Adding ~55 new lines to my project.clj just to handle the base ClojureScript development case sucks.

Om Boilerplate

I'm going to spend the least amount of time on this, because David Nolen's most recent talk at EuroClojure made clear that a much more syntactically beautiful version of Om is forthcoming, but suffice it to say that it amazes me that the library has accomplished the adoption it has in spite of its incredible verbosity in a community and a language that prides itself on taking simplicity and elegance so seriously.

JavaScript Interop

This is another area I'm going to be a little light on, but only because I read the relevant section on the ClojureScript wiki and then promptly decided that was too much of an immense hassle to be worth dealing with. Seriously, take a look at that Wiki page and tell me that's not incredibly hostile to someone new to the language ecosystem. I don't want to have to write a custom externs file for every JavaScript dependency I'm interested in consuming.

Fortunately, there's some really good work being done on this by Cljsjs, but I worry it's too much of a nascent project and/or that long-term maintenance may prove to be too much of an issue. In either case, it's not an immediate panacea to one's desire to use a more obscure JS library. In my mind, first-class support for other JS libraries in the same style as Clojure's first-class Java interoperability are absolutely critical to the long-term success of the language.

Conclusion

I'm not a big fan of being someone with a negative voice in a larger ecosystem (particularly one I happen to love so dearly), but I've found my experiences with ClojureScript to be deeply frustrating so far. It's particularly upsetting, I think, because I see some really incredible ideas at work - Clojure macros available to the client-side, bidirectional routing, tight integration with immutable frontend frameworks (with only a short hop to iOS and Android deployment via React Native!), better application state management, and generally some really fascinating potential work with websockets.

But with the current state of things, I'm not sure I can justify the frustration.

If you disagree, have feedback, or can show me what I'm doing wrong, hit me up on Twitter: @venantius.

Today I'm happy to announce the release of Glow, a very small library for syntax highlighting strings of Clojure source code.

Motivation

You may be thinking to yourself: why did you write this? After all, Ultra and Whidbey already provide syntax highlighting at the REPL, and you can pretty-print most EDN and Clojure objects using the underlying pretty-printing engines, Fipp and Puget.

The short answer is that I really, really wanted a way to get a syntax-highlighted drop-in solution for clojure.repl/source which I could inject into Ultra. The current functionality (in ultra.repl) relies on the (somewhat questionable) use of read-string to accomplish syntax highlighting via Puget, which unfortunately also does macro expansion and hides metadata.

Most of the time, macro expansion isn't a problem, but if I want to quickly look at the source code for something (e.g., a macro), I don't want to be looking at what could potentially be an extremely large macro-expanded form.

Similarly, I don't want any attached metadata to be hidden from me either. I want to look at the code as it was written.

]]>Over the past year I've been working on building my first serious web application from scratch. The experience has taught me a lot that I didn't know previously, particularly when it came to security and user experience.

Over the past year I've been working on building my first serious web application from scratch. The experience has taught me a lot that I didn't know previously, particularly when it came to security and user experience.

It's worth noting that the last time I tried building anything of any reasonable complexity was in 2005, so - in my defense - there was a lot for me to catch up on.

Even outside of what I knew about or had seen before, the laundry list of details to remember when making a web application can make it easy to forget something important - especially when you're just starting out.

This checklist isn't in any way exhaustive, and if you're an experienced developer I doubt there's anything here that'll surprise you, but I hope it'll prove helpful to those whom might otherwise have missed something.

Security

Confirm emails: When users sign up, you should e-mail them with a link that they need to follow to confirm their email. If a user updates their e-mail address at some point later on down the road, that same workflow should be triggered again.

Identity Management: When storing passwords, salt and hash them first, using an existing, widely used crypto library. If you can get away with it, outsource identity management to Facebook / GitHub / Twitter / etc. and just use an OAuth flow.

Encryption: For all of its problems with certificates, there's still nothing better than SSL. Use it. Use HSTS as well.

Eng: Animations

For the love of all that is holy, don't animate everything on your app. Most CSS animations will trigger a layout redraw; you're best off limiting yourself as much as possible to transforms and opacity.

Avoid lazy transition calculations, and if you must use them, be sure to use specific properties (e.g., "transition: opacity 250ms ease-in" as opposed to "transition: all 250ms ease-in")

UX

Forms: When submitting a form, the user should receive some feedback on the submission. If submitting doesn't send the user to a different page, there should be a popup or alert of some sort that lets them know if the submission succeeded or failed.

Login redirects: If a user attempts to access a page on your site but isn't logged in, they should first be sent to a page where they can log in, and after that should be redirected to the page they were originally trying to access (assuming, of course, that they're authorized to do so).

If they try to log-in and provide an incorrect password, provide a visual cue to remind them that they have the option to re-set their password if they've forgotten it.

Email

Subscription settings: Any emails you send a user should include, at the very least, a link to a page on your application where they can modify their email settings, and preferably a separate link that unsubscribes them from all emails as well.

Don't make them e-mail you to unsubscribe.

Mobile

You don't have to develop for mobile...but whether or not you do, you should make sure it's an active decision, since it'll have a material impact on the design and engineering of your application.

The following notes assume that you've chosen to target mobile as one of your platforms. I happen to use Grunt as my build tool, so I've included some Grunt-specific plugins that I use, but something analogous probably exists for whatever JavaScript build tool you're using.

Engineering

Single Page Apps: These days the single-page application (SPA) is king. The key advantage to an SPA is fewer full page loads - you only load resources as you need them, and you don't re-load the same resources over and over. If you're just starting out with a new web application, it should probably be an SPA.

UI

Resolutions: While developing your MVP, you probably don't need to make sure your UI works on every possible mobile device out there - but you should make sure it works on a basic range of phone and tablet resolutions. Leave worrying about every device until after you've hit product-market fit.

UX: Bandwidth

One of the big themes with mobile is that bandwidth is a more precious resource than it is on the desktop. Accordingly, you should look for every opportunity to decrease the number of requests being made, make them asynchronous where possible, and decrease the size of the resources being requested.

All assets - Use a CDN: There are two major advantages to using a CDN. The first applies to all hosted assets, and that's localization - a CDN makes sure that the resources served are located in a region that's geographically close to the user requesting them, reducing load times.

The second applies more to your dependencies (i.e., non application-specific styles and JS code). Here, using a CDN for your dependencies has the potential to greatly reduce loading times for your users by relying on their caches. For instance, since many sites will rely on Angular.js, using a CDN to link to the core Angular code will trigger a cache hit, and the user's mobile device will retrieve it from the device cache rather than making another HTTP request.

CSS - Reduce your footprint: Most developers starting out will probably use some sort of UI framework (e.g. Bootstrap, Foundation, etc.). These frameworks can be quite large, and while minified versions of their stylesheets are usually available on most CDNs, it's unlikely that you'll need all of the included styles. Consequently, a tool like uncss (typically paired with something like processhtml) can be incredibly valuable at removing the styles you don't end up using.

It's important to note that the uncss parser can't pick up dynamic styles (styles that only appear in response to, say, JavaScript events), so you'll have to be rigorous in your browser testing to make sure you don't cut styles that are in fact in use in your application.

CSS - Put crucial things in the head: Styles that need to be seen before the app is even finished loading should go in the head; less important styles can afford to be loaded later.

JS - Reduce your footprint: Since your JavaScript code doesn't need to have any of its internal variables make sense to a human reader once you're in production, it can be helpful to rename all those user.email vars to u.e to reduce your file size. Fortunately, there's a tool to do this - the aforementioned uglify, which makes JS code completely unreasonable, but usually much smaller.

UX: Forms

While it's generally good advice to keep your forms and workflows simple, this is especially important if you're also targeting mobile as a deployment platform. Nobody wants to fill out a 5-page form on their iPhone.

I hope this list proves useful to those of you just starting out on your first web application - or even for those of you who've maybe worked on backend or design before but weren't as familiar with some of the frontend optimization tricks. If you can think of other hints or things to remember when starting out, let me know and I'll see about adding them to the list.

Thanks to Chris Dean (@ctdean), Danny King (@dannykingme) and Allen Rohner (@arohner) for not only reading drafts of this post, but for adding to it.

]]>I'm writing this post in anticipation of the upcoming release of version 0.1.2 of Yagni. I've written about Yagni here before, and if you're interested in this sort of thing I'd recommend reading that post first to understand the methodology behind the analyzer.

I'm writing this post in anticipation of the upcoming release of version 0.1.2 of Yagni. I've written about Yagni here before, and if you're interested in this sort of thing I'd recommend reading that post first to understand the methodology behind the analyzer.

TL;DR: Yagni is a static code analyzer for Clojure that traverses your codebase to find variables and declarations that are unreachable (that is to say, will not be referenced at runtime).

Today's post will be about the subtleties of getting a static analyzer in Clojure to work well with Java interfaces and classes. I should note up front that, at least for the moment, the focus here is on Yagni's specific use case: analyzing Clojure code that includes special forms for Java interoperability. There will be no analysis of Java code.

A Primer on Java Interop in Clojure

While detailed writeups of Java interoperability in Clojure can be found in a number of places, I'm going to limit myself here to a specific subset of interoperability forms, namely: defprotocol, deftype and defrecord. These forms are used within Clojure to generate Java interfaces and classes.

defprotocol

defprotocol, as the name suggests, is used to define protocols, and generates Java interface code. An example protocol might look like this:

Any class seeking to satisfy this protocol must satisfy the interface of having the two methods (bar and baz) and their signatures implemented. In this case bar takes two arguments, and baz is a polymorphic function that can take either one or two arguments.

If this protocol was specified in lib.ns, the bytecode for the interface would sit on the JVM at lib.ns.AProtocol.

deftype

deftype is a Java class generation form. Philosophically, deftype is intended for use in declaring and defining classes that are unique data structures rather than simple data holders. An example, implementing the protocol we defined above, looks like this:

As with AProtocol, if defined in lib.ns the generated class bytecode here would be referenced at lib.ns.Foo.

defrecord

Like deftype, defrecord is a Java class generation form. However, the underlying philosophy of defrecord is a little different, and is focused instead around the notion of a custom data holder class. In this regard, defrecord forms function much like traditional Clojure maps, but with generated class bytecode. Like deftype, defrecord forms can also satisfy interfaces and have additional methods.

The Problem Statement

Okay, now that we've covered the basics of JVM interoperability, let's talk about some of the things that our static analyzer will need to address when dealing with these special forms.

No Direct Vars for Classes

One of the most immediate problems we have is that deftype and defrecord forms don't actually intern a var for those classes in the namespace in question. That is to say: while there's JVM bytecode for lib.ns.Foo and lib.ns.WereWolf, there are no corresponding lib.ns/Foo or lib.ns/WereWolf vars. Since Yagni builds its reference graph by looking for definitions using ns-interns (which returns a map of interned vars in a namespace), the lack of vars means Yagni won't know a class exists.

defprotocol in this case is also tricky. On the one hand, there is an interned var for the protocol. However, references to protocols in deftype, defrecord forms are actually references to the class, not the var.

This creates two problems: first, how do we know if there's a deftype or defrecord form in a namespace, and second, when references to all of these forms are references to the class, how should we track them?

Generator Functions

While deftype and defrecord forms don't intern a var for themselves directly, the macroexpansion of the forms does intern generator functions in the namespace of the declaration. This means the following functions are interned in our namespace automatically: lib.ns/->Foo, lib.ns/->WereWolf and lib.ns/map->WereWolf.

We don't actually want Yagni to report on the use of these constructors directly since that would be rather noisy. For instance, if you only ever used the ->WereWolf constructor, having Yagni warn about the lack of usage of the map->WereWolf constructor isn't actually something you're likely to care about. You might not even use one of these generator functions at all, and instead use...

Class Constructors

In addition to the generator functions above, Clojure has additional special syntax for class construction. This takes one of the following two forms:

;; these do the same thing
(WereWolf. "Abraham" "Lincoln")
(new WereWolf "Abraham" "Lincoln")

The first form will macroexpand to the second...sort of. At the lowest level the macroexpand works, but when macroexpanding from an outer form, it won't. To give an example:

Unfortunately, due to the way Yagni's form walker works, we can't recursively call macroexpand each time we look at a new form, otherwise we end up trying to call macroexpand new, which throws a RunTimeException since Clojure can't resolve new. Irritating.

This means Yagni needs to recognize both WereWolf. and WereWolf as references to the lib.ns.WereWolf class.

Ultimately, our problem space looks something like this:

The Solution[s]

When one looks at the problems described above, they boil down into one meta-problem, which is that we can't tell whether or not a given protocol or class is actually being used. In the case of protocols, the standard syntax involves a reference to the class, rather than to the protocol's var, and in the case of the classes we have between 3 and 4 possible inbound reference possibilities between the class' generator functions and class constructors.

Identifying Classes and Extending the Graph

The path I've taken with the 0.1.2 release has been to leverage the existence of the interned generator functions as a proxy for identifying possible class definitions. Specifically, Yagni now checks to see if functions named lib.ns/->X or lib.ns/map->X can resolve to classes named lib.ns.X. When they can, Yagni adds a new node to the graph with the name lib.nx/X (where a var would exist if deftype and defrecord created self-named vars) and adds an edge from the generator function to that node.

When Yagni encounters a reference to an interface class / protocol, it adds an edge from the form it's walking to the protocol's var rather than to the protocol's class.

Similarly, while Clojure's resolve function will correctly resolve a direct reference to the WereWolf class, trying to resolve the syntactic sugar for a new WereWolf (i.e., (WereWolf.)) won't. So, as with the generator functions, now when Yagni hits a symbol that ends in a period, it checks to see if that symbol can otherwise resolve to a class. If it can, Yagni knows that its looking at a class constructor, and adds an edge to the var-like node (the lib.ns/X mentioned above - in this case lib.ns/WereWolf) created when Yagni traversed the generator function.

Compressing the Graph

The methodology described above serves us handily in the graph traversal phase of Yagni's analysis, but ends up being a little too noisy at report time. For instance, looking at that last diagram in the previous section, Yagni would warn us that we're not using any of its generator functions. Of course, we could simply remove the nodes for the generator functions, but then deftype and defrecord forms that had generator function references but not class constructor references would show up as parents rather than children, which is incorrect.

The solution here is to compress the graph by first changing any edges pointing to the generator functions to point directly to the nodes for the corresponding deftype or defrecord forms. We then remove the generator function nodes from the graph, leaving us with a graph that now looks like this:

Perfect.

Conclusion

Funnily enough, when I initially wrote about Yagni, I had some ideas about what the project's roadmap would be, but improving the Java interoperability wasn't on my radar at all. One of the really cool things about open source work is that sometimes your projects take you in a direction you hadn't considered (even if you should have)!

There's still quite a bit more work to be done on Yagni, but this week's release will hopefully be a major step forward for teams working in Clojure with generated Java classes.

As always, if there are additional features you'd like to see implemented, feel free to file an issue - or better yet, a pull request!

Background

Another day, another project. Today I bring you Yagni: a static code analyzer for Clojure, designed to find the parts of your codebase that aren't in use.

Background

Given the general acceptance of "You aren't going to need it" as a principle, it seemed reasonable to want an automated way of identifying vestigial features. Since we're in lisp-land, code and data share the same abstract syntax tree - why not leverage that strength to programatically traverse a codebase and see which parts are unused?

Per Martin Fowler's article on the subject, this plugin won't get rid of the cost of building or repairing an unnecessary feature, but it can at least eliminate the cost of carry by letting you know that you can remove the pertinent section[s] of code without fear of repurcussion.

How It Works

Yagni works by first walking all of the findable namespaces within your project's :source-paths, and identifying all of the interned vars within those namespaces. Like Eastwood, it will require your codebase's namespaces, so if importing a namespace fires the missiles, this will fire the missiles.

Pro tip: don't have import side effects.

Once Yagni has identified all of the interned vars in your project's namespaces, it walks the forms of those vars (Clojure, being a Lisp, makes this relatively easy). As Yagni walks the forms, it looks for symbols that resolve to the known interned vars, and builds a directed graph of references between those vars.

It then repeatedly searches the graph, starting each time from one of a set of entrypoints. Entrypoints here should be thought of as the things that someone would invoke from outside your codebase - if we're talking about an application, you could think of a main method, while if we're talking about a library, a set of functions or a public API would be a better mental model.

By default, Yagni assumes the only entrypoint is the :main value in your project.clj, but you can include other entrypoints by listing them as namespace-qualified var names in a .lein-yagni file in your project's root directory.

As Yagni searches, it prunes the nodes it finds from the graph, ultimately leaving behind a subgraph of the original where the only remaining nodes are vars that were unreachable from the provided entry points.

From here, Yagni reports on these "orphaned" vars as being in one of two classes: children and parents.

A child in Yagni is a function that is called somewhere within the codebase, but there's no actual path from an entrypoint to it.

By contrast, a parent in Yagni is called by nothing, anywhere. It has no inbound references at all.

As the definitions of parents and children should suggest, parents call children, and not the other way around. Attempting to remove a child from your codebase without first making sure that its parents have been removed will cause your program to fail to compile (since the reference from parent to child will still exist), so if you decide to prune your codebase, it's a good idea to begin with the parents.

Weaknesses

Everybody's a critic. Including me.

As you might imagine, while Yagni represents a reasonable approach to the problem at hand, it's also easily tricked. Circumventing some of those tricks is within the realm of possibility, while addressing the others is a much harder problem. Still, here are a few known weaknesses with the approach I've taken.

eval, read-string, et al - ah, yes; the bane of all static analyzers...eval. If you're dynamically creating code from some external input, or input that is only valid at runtime, there's just no easy way for Yagni to know what you're going to ask it to do.

The same goes for non-idiomatic type usage - if you're referencing vars by casting strings, keywords, hex, bytecode, etc. to symbols before turning them into vars, Yagni isn't going to know what's intended to be a reference and what's not. It assumes that Clojure's idiomatic usage of symbols as references to vars is the only one it needs to concern itself with.

branch logic - let's consider a somewhat malicious example: (defn foo [] (when false (bar))). Although (when false ... doesn't come up that often in practice, hopefully you get the idea: there's a single deterministic outcome here (unless you're redefining false, in which case you're a monster), and it's that bar isn't going to get called.

Yagni's form-walker is naive to this sort of branch logic, and assumes the fact that a reference to bar exists within foo means bar's worth holding onto. And on some level, it's right - attempting to remove bar would cause the compiler to fail. Even so, there's an argument to be made for some smarter branching logic in the walker.

multi-arity functions - another straw man: (defn x ([] foo) ([y] bar)). If x is only ever called with no arguments, then the reference to bar is unimportant. As before, the same caveat exists with regard to compiler exceptions.

non-interned var references - as a general statement, this is probably more your problem than mine (see the pro tip, above). That having been said - if you've got inlined code that isn't sitting in an interned var, Yagni will never know about it, nor any references it makes to the rest of your program.

Roadmap

I believe the project in its current form and version (0.1.1) satisfy the bare minimum for the stated objectives of the project. Even so, there are aspects to it that are interesting but currently under-leveraged. For instance:

Further Graph Analytics

Given that Yagni goes to all the trouble of constructing a graph of the program's references, it feels like there's quite a bit more there that could be of interest. For instance, with regards to the orphaned vars, which parents call which children?

I'm not yet sure I have strong preferences about how such information should be reported, but I'm interested in exploring it further.

REPL Invocation

I'm generally a fan of being able to invoke core Leiningen plugin logic from the REPL - in the past I've leveraged Eastwood and Cljfmt's abilities in this regard to write vimplugins, and while I'm not sure I'd want to write a vim plugin for Yagni I can certainly see the value in being able to call it from the REPL easily.

Namespace Re-writing

For the truly lazy, some Slamhound-style functionality could be very cool - something to simply prune unused code for you as required, re-writing the relevant namespaces in the process. Potentially dangerous, but fun nonetheless.

Conclusion

As should be well established by now, I'm veryenthusastic about developmenttools. In an ideal world, I'd hope that my fellow Clojurists find Yagni to be as valuable as their current linter - the sort of thing that should be run as part of your continuous integration process for any branch that's a contender to be merged into master.

Whether or not that actually happens, I'm looking forward to finding ways to make Yagni even better. In that vein, I'm excited to see what issues, pull requests, and features people create and are interested in.

Last February I penned a manifesto outlining my frustrations with my old way of life. I resolved to compromise less; to build more; to filter for that which brings me growth, joy, and love.

Looking back on what I wrote then, I feel a sense of great pride in the degree to which I held true to my own resolutions. As is often the case, it wasn't perfect - I didn't end up making many new friends, and I wrote far less than I wish I had.

Even so.

The past year has been one of intense struggle; of risk; of foundation; and of the occasional setback. If I had to condense it into a single word, it would be this: strive.

A good friend of mine recently told me that he had a particular attraction to creative types.

This might seem unremarkable to you, and, for what it's worth, I don't think he's alone in feeling this attraction. Anecdotally, I've heard similar desires expressed by quite a few people, in a way that makes me believe that this sort of thing is common (on both sides of the gender aisle).

I've seen it take on slightly more aggressive forms as well; a tendency, say, towards lines like "only creative types need apply" that go beyond more abstract expressions of desire for a creative partner.

I can't claim that this is a recent trend, but it is something I've been more aware of lately. In either case, let's unpack it a bit.

On the one hand, this is the sort of thing that evokes a certain amount of civic pride. Living in San Francisco, it is a reminder that I'm fortunate enough to be immersed in a culture where the arts are held in high regard - one in which at least a few individuals have formed romantic (perhaps even erotic) associations with the notion of the creative person in abstraction.

On the other, it troubles me.

For starters, it's not clear to me that popular conventions around what it means to be a "creative type" are accurate. I'd wager that those of you reading this probably summoned some mental image of what you think constitutes such a person; a hippie with a guitar, perhaps, or someone working with watercolor in her studio. If I wanted to paint with broad strokes, I'd classify these people as working in the "fine arts."

I'd put money down, however, that your mental image wasn't of a cook, a gardener, or a woodworker. I'd wager that a graphic designer didn't make your list either - and a software engineer certainly wouldn't be in there. Indeed, you might find the notion of regarding more technical craftspeople as creative individuals to be offensive or even obnoxious.

In part, I suspect that - at least in the Bay Area - the desire to be with a creative type is part of a larger backlash against the perceived monoculture of the technology industry. Those seeking more artistic partners are, on some level, trying to ensure that their lives and relationships are diverse, colorful, and have at least some escape from the cultural imperialism of tech.

While this is - on some levels - fairly reasonable, I worry that this sort of approach is the result of unexamined notions around what art and artists even are.

So, let's get into it - let's talk about what it means to be an artist. In doing so, I'd like to sidestep the question of how to define art in favor of examining the human beings involved. Although people are usually more complicated than the things they create, in this case I believe the former is easier to talk about.

I assert that a reasonably broad definition of an artist is an individual who is driven by an expressive compulsion. While plenty of people produce art in order to make a living, artists share an internal desire or need to engage in the form[s] of expression that they've chosen.

Going one step further, such an individual is likely to not only be driven by a need for expression, but by a need to express themselves in a fashion that is unique or otherwise novel. I know of few artists who seek only to reproduce works that already exist (at least, not without intending some additional expression through such an activity). Such novelty can take many forms - engaging in a different style, using a different technique, with different partners...an exhaustive list would be impossible.

Obviously, not all cooks, gardeners, or woodworkers would necessarily qualify as artists under this definition. There are those who have no particular compulsion around their work, and are rather more focused on the steady paycheck. At the same time, it becomes more apparent that many people outside of the fine arts can and should be considered artists. For instance, I can think of several people for whom their technical work is an expressive compulsion - one without which they would be miserable.

. . .

To be clear, the point I'm driving at isn't some wishy-washy "everybody's an artist!" kumbaya moment. Rather, I want to draw attention to the fact that popular notions of what artists look like don't map neatly onto the reality of creativity and art itself. The heuristic of "fine arts" is convenient, but highly exclusive - and does a real disservice both to artists and to the rest of us.

For artists, only having the fine arts be clearly classified as art results in an unspoken hierarchy of social value around the arts. This isn't a problem that is unique to San Francisco by any means, but I suspect that SF is unique in the extent to which this hierarchy bleeds over into the dating scene and casual society at large.

Even more worrying is how perversely casual we can be in our designation of what is and isn't allowed to be art. If an unspoken gradient of value is problematic, surely a binary true/false is even more so. Especially disturbing is the way the people most inclined to such casual judgements seem, at the same time, to be the most uncomfortable engaging in substantive discussion as to what such tricky notions as art or creativity mean in the first place.

For those of us who don't identify as artists, the most sinister implication of all of this lies in the inferred valuation of actual human beings; the inverse of an "only creatives need apply" mentality is that those deemed less creative or non-creative are implicitly less valuable.

. . .

Some readers may find it curious to hear such rabble-rousing coming from someone working in the technology industry, where such implied systems of merit hold powerful sway. Debbie Chachra penned an excellent article, Why I Am Not A Maker, on the subject earlier this year, making the argument that the technology industry's worship of creation and creators devalues everybody else.

Although Ms. Chachra and I come from very different places, we've arrived at a very similar conclusion. We're both disturbed by the devaluation of our peers in roles that are seen as being not creative enough - especially when such judgment comes from a society that seems incapable, unqualified and unwilling to value such skills in the first place. If anything, I would argue that her article doesn't go far enough, limiting itself as it does to the technology industry. I believe the fears she raises extend far beyond the borders of tech.

Something of a dark message, to be sure.

. . .

What to do about such a situation? I'm not so naive as to expect people to spontaneously begin to engage more seriously and individually with the question of art - and yet, I am optimistic.

Over the past decade, and especially in the last few years, I've noticed a general trend amongst my peers towards accepting people on the basis of self-identity, rather than on more traditional constructions of self. I'm especially cognizant of this trend as it relates to gender, sexual identity, and race, but I don't think it stops there, and nor should it.

I would argue that - considering the nature of the artist as an identity-driven creation - this course of events is likely to work against the system of unexamined value that I've been describing so far.

Personally, I'd love for people - including you, dear reader - to consider and reconsider the nature of art and artists on a regular basis. But if I can't have that, I'd like to encourage you instead to engage with the people you meet who consider themselves artists on their terms, rather than your own.

After all, people are messy. They're complicated creatures with layers of identity that are, at times, contradictory. Perhaps most importantly, their ideas of themselves may not map neatly to the convenient boxes that we've all become accustomed to using.

I believe that in engaging with people (including those in the fine arts), it is worth abandoning our ever-so-convenient heuristics around art. It can be exhausting - using them does make things easier - but it both frees and forces us to engage with people in true terra nova. My hope is that by doing so, it'll be harder to easily discard people - and that if you do still end up deciding someone isn't worth your time, it'll at least be because you met them on their own terms and found them wanting.

This post originally appeared on the Standard Treasury Engineering Blog, here. It has been re-published here with permission.

Today I'd like to talk about software testing - a subject dear to my heart. In particular, I'd like to talk about testing a modern microservice architecture, using Standard Treasury's software stack as an example.

Let's begin with my assertion that this is a both hard and unsolved problem. If you disagree with that, I'd love to talk to you about what you know that I don't. That's a serious offer, by the way.

Okay.

I'm going to begin at the beginning and work my way up.

Unit & Functional Tests

In the beginning, the engineer wrote source code and unit tests. The unit tests were self-contained, had no side effects, and were fast. The engineer said, "let these tests run", and saw that the passing tests were good, and divided the passing tests from the failing tests.

And the engineer said, let this be reviewed by my peers. And verily, the peers did review, and the source that was good was accepted, and the source that was poor received some additional revisions. And the source and tests were checked into version control on the second day.

On the third day the engineer wrote more source, and with this source wrote functional tests. And although there was action at a distance, and many moving parts, the engineer was able to compose guarantees around performance and behavior by complementing the functional tests with more unit tests, and it was good.

And however the engineer may have felt about this process, the fact remains that it was easy, and well-understood, and accepted. There was a process, and it was followed, and the development team was on the same page.

As the engineer wrote more applications, they sought to have those applications talk to each other through widely accepted standards. And the engineer thought: how should I test that my applications will work each other?

Genesis, please exit stage left. You're a cute allegory, but we're about to get real serious up in here.

Integration Tests

Integration tests are hard. There's a lot to think about, many components to keep in sync.

For instance:

How do you ensure all of the versions of your applications and underlying services are in sync?

How do you configure your networking? Is it the same in your test environment? Staging? Production?

How do you debug when something doesn't work as expected?

Are all of your logs in the same place?

Are you making sure your local environment isn't polluting your configuration?

Lacking static analysis tools, are you sure the integration test coverage is adequate?

These are all manageable issues when there's only one application and a few underlying services. As additional applications and services are introduced, however, things can quickly get unwieldy.

For example, we have a microservice architecture built on top of Postgres and Storm (with Kafka as the message queue). Putting security aside for a moment, a call to our API to make a transfer involves calls to our customer information service, our ledger service, as well as records being written to the database and messages potentially being enqueued and dequeued via Storm.

And, of course, let's not forget that these applications are all under active development and need to be kept in sync with each other as their interfaces evolve.

All that having been said, and in spite of the difficulties, integration tests are not optional. Our applications and services need to talk to each other, and while tests are not a fully effective vaccine against bugs, without them we're just flying blind. At the very least, our deployment days would be likely to contain some very nasty surprises.

The Naive Approach

A naive approach to integration testing might involve running all the different services locally in different windows, and having a test suite that ran against that. There are obvious advantages to such an approach - it's very easy to set-up, fits nicely into one's existing development flow, and makes testing different branches trivial.

On the other hand, there are a lot of downsides. Configuration can be messy, and can vary significantly if your development team isn't all using the same operating system. If one isn't paying attention, one can end up with local version mismatches by accident. If you're the sort of place that's into the twelve-factor app, you might accidentally pollute your tests by not paying close attention to your environmental variables. Debugging problems can be a headache as your log output is spread across many windows and locations. Finally, this approach doesn't port well to modern hosted CI environments well without some major hackery

A Better Way

I'd like to propose a more modern, devops-driven approach: use Docker! In particular, I'd recommend using the recently unveiled Compose (née fig) to standardize your integration testing.

Quoth the website:

Compose is a tool for defining and running complex applications with Docker. With Compose, you define a multi-container application in a single file, then spin your application up in a single command which does everything that needs to be done to get it running.

Let's start by taking a look at what a configuration file looks like for Compose. In doing so, I'm going to show you what a subset of our configuration file looks like.

At the lowest level, we have Docker containers running Postgres 9.4.1 and ZooKeeper 3.4.6:

There's not too much that needs explanation here, though the ports might be a little confusing - for instance, what's the significance of "5433:5432" for Postgres?

The answer is that we want to be able to access Postgres for debugging purposes, and so we need to map the container's port (5432, the default Postgres port) to a known host port (5433, in this case) for convenience. I've chosen to use 5433 as a host port instead of the default 5432 because I often also have a Postgres service running on my dev box, and I don't want these ports colliding.

It's worth keeping in mind here that our Docker host in this case is running on a Vagrant VM, so we also need the following lines included in our Vagrantfile to forward the relevant ports from Vagrant to OS X.

Now let's look at something slightly more complicated - getting containers to talk to each other. Kafka is a good example here, because it registers itself with ZooKeeper and uses ZK as a central manager for the various brokers.

Sidebar: in our case, we only have one broker instance, and our configuration reflects that. However, Compose is capable of scaling up services to arbitrary sizes, and so if we wanted to we could be using Compose to run our tests with multiple Kafka brokers all running on the same Docker host, with coordination handled by a ZooKeeper also running on that same host. Cool, right?

This section of our configuration has a few new things. First, you'll notice the hostname field - in most cases, you won't actually need to worry about this, but we ran into a particular networking problem with Kafka, ZooKeeper, and our CI server that required us to be explicit about the Kafka service hostname. As the key name suggests, this sets a value in your /etc/hosts file for the current service with the provided name.

The links section is where we tell Compose to network two containers together. In our case, we're linking to our ZooKeeper service (which we defined above) and giving it the alias "zk". Compose will now take care of (a) exposing ports for the two services to communicate with each other and (b) adding an entity in the Kafka container's /etc/hosts file for the zookeeper service, which will use the alias "zk".

At the risk of sounding too excited, this is a huge deal. A cursory look at the Docker networking documentation should suffice to give an idea of how complicated container networking can be, and Compose makes it unbelievably simple to get things to "just work".

The rest of this configuration is fairly self-explanatory - we've seen port configuration previously, and the 'environment' section should be apparent - it sets environment variables inside the container. In this case we're setting some basic configuration fields that Kafka expects and will use to communicate with ZooKeeper.

Last, but certainly not least, is what configuration for a Standard Treasury application looks like:

The beauty of this is that there's really nothing special here that we haven't seen before. At the moment, our Ledger doesn't rely on Kafka and ZooKeeper, but it does have a direct link to our Postgres service, as well as some environment variables to make the actual database connection easy for our Ledger application. We store our application Docker images on a private registry within our VPC, which explains the pretty lengthy image name. And that's basically it!

The above is but a subset of the Standard Treasury architecture, but it gives a good idea of what our greater configuration looks like. Compose gives us an easy way to layer and network services together to create a larger infrastructure that can be easily deployed, particularly in a development or testing environment.

On both our development machines and in our CI environment (we use CircleCI), we use Compose to first bring up all of our services, and then we run a test suite against the exposed services that Docker's running for us in the background.

Pros and Cons

First amongst the advantages of this is main value-add of Docker at large: it works on multiple platforms without any additional configuration. No more worrying about the underlying OS or having environment variables polluting your tests - if you can run Docker, you can run your applications with a preconfigured environment.

Convenience is fairly high on the list here; we've got a few Make targets that are a couple lines long, but by and large you can do almost anything you want to with Compose in a single instruction. For instance, I can boot the entire Standard Treasury architecture by simply being in the right directory and typing docker-compose up.

Running Compose allows you to easily nab all container output in Foreman-style colored logs, and to attach to individual container or selections of container logs as needed. You can make sure underlying service dependencies, such as your database or message queue, are pinned to specific versions. Lastly - and perhaps, most importantly - you should be able to get Compose working in your CI environment without having to jump through any further hoops.

Of course, there are some downsides.

The initial configuration can take time to get right, and while the Docker community has done some incredible work over the last two years in making container networking much easier, debugging networking problems can still be a real bear. For instance, we encountered a bug that only occurred in our CI environment, but had to do with the fact that the Kafka container didn't include itself in its own /etc/hosts file.

If your development team is using OS X, they'll need a separate Docker host running on their machine. In that capacity, boot2docker running directly on OS X can be a little rough around the edges. We've been using a Vagrant VM with boot2docker running inside the VM, which has definitely helped, but comes with its own trade-offs - additional teardown/rebuild time if the VM has problems, and difficulties accessing and debugging problems with Docker volumes.

Lastly, testing dev branches for multiple services simultaneously requires either (a) editing your compose.yml file, or (b) overwriting Docker images with specifically tagged builds for your dev branch. Neither of these are particularly ideal, and - speaking personally - I'm not sure I could even tell you what a better workflow should look like.

Conclusion

There are other frustrations I'll admit to - pulling Docker images from our private repository can take a while, and in general I want all of this to work faster. but there's no question that using Compose has allowed us to engineer a solid integration test suite that's portable across platforms, easy to deploy and run, and gives us a great starting place for debugging problems.

This is still the early days for Compose, and I'm looking forward to seeing what the Docker team and community come up with as more companies use tools like these to address the sorts of technical issues we've faced.

If you have thoughts on what we could be doing better, or have tried solving this from another angle, we'd love to hear from you (did I mention we're hiring?) - just tweet at us at @standardapi.

Oh, hi. You're back. Well, since you apparently didn't get sick of me talking about Clojure tools last time, let's keep going.

Vim is my terminal-based text editor of choice. It's highly configurable, but for me, it looks like this:

If you want to get into what's going on there - my .vimrc file isn't particularly organized, but if you're curious it looks like this. I use Solarized as my color scheme.

Why Use Vim?

This is one of those subjects that has inspired religious wars for decades. The short version goes like this: if you're a systems engineer, at some point you're going to need to familiarize a terminal-based text editor, and it should be one that you can expect to find on any reasonable UNIX system. Often, this boils down to a choice between Emacs and Vim (those being the two most mature and widely distributed such programs).

My reasons for using Vim are simple: several years ago, when the universe was still forming in the white heat of the big bang, I had the choice to learn Emacs or to learn Vim, and I chose to learn Vim.

...there isn't anything more to that story.

Honestly, at this point I'm familiar with Vim's tricks and idiosyncrasies, and just don't feel inclined to learn a new editor. There's no denying that Emacs has an inherent and [literally] built-in advantage when it comes to Lisps, but the best tool is often the one you know.

Vim Plugins

The world of Vim plugins is both huge and awesome. In addition to the Clojure-specific plugins listed below, I use vim-airline, Syntastic, and NERDTree, among others.

You didn't come here for me to talk about Vim in general, though, so let's get down to brass tacks: Vim plugins for Clojure.

vim-fireplace

Fireplace is the killer plugin for Vim and Clojure. It starts what @tpope calls a "quasi-REPL" that can be used to connect to a running nREPL server. By default, Leiningen will write a .nrepl-port file in the project's root that contains the integer port that the nREPL server is running on. When you open a Clojure file in Vim, Fireplace reads the port from the .nrepl-port file and opens a connection as a nREPL client.

You can also manually connect to a REPL server from Vim with the :Connect command, which is handy if you need to connect to a REPL that wasn't initialized from the current project directory.

If for some reason you don't have a nREPL server started (either through Leiningen or the application itself), there's also vim-leiningen which will shell out to Java to initialize one for you based on what the plugin can infer of your classpath. However, this tends to be a bit on the slow side, and in general it's much faster to just have an nREPL server already running in the background for you to connect to.

I use the fireplace REPL for a few major things:

code traversal

grabbing docstrings

running tests

hot-reloading the file I'm editing into the JVM

Some of these - in particular, docstrings and tests - are the sorts of things I'd be doing in a REPL anyways. The others are less so, but are hugely valuable for a code editor to have.

Code traversal

Let's start with code traversal - entering gf (goto file) when the cursor is over a namespace will take you to the namespace file, which is particularly great when you're wondering how the hell one of your dependencies works. Otherwise, you'd have to dig up the jar on your classpath and navigate its contents (or worse, look at the code on GitHub, which would involve - gasp! - leaving the terminal). Once within another namespace, hitting CTRL+w backs you out of the namespace into the original file.

Docstrings

I'm of the opinion that docstrings (and documentation at large) are underrated. Since I'm essentially an idiot at a typewriter, I tend to look up function documentation frequently.

When I'm at the REPL, this means a lot of (doc x). Fireplace gives me the ability to do that with :Doc x, and provides an even better shortcut by setting K as a hotkey that looks up the docstring for the symbol under the cursor.

Running tests

If you're in a test namespace, you can use the :RunTests command to kick off a conventional (clojure.test/run-tests), with the results loaded into a quickfix file

I also have a little Vim function (courtesy of my friend David Lowe) for running the test form under the current cursor. All this power and more can be yours for the low, low price of adding the following to your .vimrc (or wherever you store your Vim functions)!

Hot-reloading code into the JVM

This is, as far as I am concerned, the most important of Fireplace's features, and one I use pathologically. Using the :Require command is the equivalent of entering (require ... :reload) for the namespace in the current buffer, and I find myself doing this so often that I have it hotkeyed in my .vimrc as follows:

au Filetype clojure nmap <c-c><c-k> :Require<cr>

My development flow looks something like this:

Within Vim, make a few changes, write a new function, etc. Reload the changes into the current JVM with CTRL + C + CTRL + K.

Switch tabs to a REPL, where I interactively play with and test my changes. This is done quickly by cycling through iTerm windows; ? + ? or ? + ? for windows in my current tab, and ? + ? or ? + ? to change between tabs.

Switch tabs back to Vim, rinse, repeat.

Every half hour or so, run some tests. You have been writing tests, haven't you? ;)

When one is just starting out on a project and/or building very small and composable functions, the REPL alone can be a sufficient development environment. However, as the size of the codebase grows, I've found it easier to iteratively build functions by working on them in their target namespace within my editor, where I can more easily visualize and maintain context for what I'm working on. Fireplace is an incredible tool for supporting such a workflow

paredit.vim

Paredit.vim is another absolutely critical plugin to have for the Vim Clojure developer's toolbox. A component of the larger slimv ("Slime for Vim"), Paredit attempts to maintain the balanced state of matched characters for parenthesis, brackets, double quotation marks, etc.

If that were all it did, however, it would only be passingly useful. Its true value lies in its support for what Emacs users refer to as "Slurpage" and "Barfage" - the ability to, with a keystroke or two, move existing arguments into or out of a given form.

For instance, let's say I have the following in my editor:

(println (+ 1 1) 1)

What I really want, however, is to have that third one inside the form doing the addition. Well, with Paredit.vim, I can do that easily with two keystrokes, <Leader> >:

Buh-whaaaaa? So spice! I have my <Leader> mapped to ,, so for me the keys are right next to each other , >. This is really handy because Paredit doesn't naively push parenthesis around simple words, but rather around lisp forms. As a result, if I were to try to do the same command a second time, it wouldn't have any effect, because the inner parenthesis cannot logically be pushed outside of its containing form.

In addition to that, Paredit is great for wrapping forms in different wrappers, which is handy for when you've written something that needs to be refactored into a function, or needs let bindings added at some point.

Extra Credit: If Paredit is a part of slimv, why not just use Slimv? Great question, reader! To be totally honest, I don't have the best answer, and in fact, slimv might be great for you. However:

It comes with its own REPL, and I think the current consensus among Vim + Clojure nerds that the Fireplace REPL is better.

Paredit.vim is pure vimscript, and is extremely lightweight. slimv is much larger, and is all over the place: some common lisp, some Clojure, some Scheme, some Python, some VimL.

Ultimately, I prefer simplicity, and at this point I've been working in Vim long enough that I really trust Tim Pope's taste over most other plugin developers. But hey, you do you.

vim-surround

Next in the list of extremely useful functions for dealing with parenthesis is another Tim Pope plugin, vim-surround. This one probably deserves a place in your list of plugins whether you're working in Clojure or not, especially for when it comes to working with strings. What I find it most useful for, though, is for getting rid of stuff. As the good lord Paredit gives, vim-surround taketh away.

For example, check this out: ds( ("delete" "surrounding" "parens"):

Manually wrangling parentheses and other surrounding markers is a huge pain in the neck. Paredit.vim and vim-surround give you a large selection of key bindings to make working with such enclosures fast and easy.

vim-eastwood

You know I have a habit of hyping my own plugins, right? Well, if not, you do now! In my last post on Leiningen, I talked about Eastwood, the Clojure lint tool. vim-eastwood is a Vim plugin for Eastwood bindings that sits on top of Syntastic.

It requires you to have Eastwood on the Clojure classpath (i.e. preferably in your ~/.lein/profiles.clj) and to have a running nREPL server that you've connected to with Fireplace. After that, it feeds quickfix info on your current file to Syntastic, which generates handy markers in the left gutter and shows lint messages at the bottom of the screen when you move your cursor over the area in question:

vim-cljfmt

Continuing down the road of shameless self-promotion is another of my Vim Clojure plugins, which provides convenient bindings for cljfmt, the Clojure formatting tool. vim-cljfmt allows you to format the contents of the current Vim buffer through the :Cljfmt command, which by default is run every time you write the buffer to a file, much like how gofmt works.

Observe:

Now, isn't that nice?

Honorable Mentions

The truth is that I don't give much thought to vim-clojure-static since it's been shipping with Vim ever since 7.3.803, which was released over a year ago. As for rainbow parentheses - you know, I can see the value, but ever since installing Paredit.vim I haven't really felt particularly confused about what depth of nested parentheses I was working in. Personal preference, perhaps - if it works for you, I wish you all the best in using it.

This isn't a plugin, but it still deserves a shoutout - the hotkey, %, lets you find the starting/terminating parenthesis/bracket/brace for the one you've currently got your cursor on top of:

Conclusion

Anybody who has had the questionable privilege of having worked with me knows that I am peculiarly fanatical about tooling. I'm easily frustrated by poor tools, and I suffer from the engineer's afflication of always searching for a more perfect workflow.

One of the things I like about Vim is that it doesn't attempt to constrain me. If I want a new feature, I can add it - in a pretty wide range of languages - and I can rely on the editor's rich plugin ecosystem to provide painless ways to bind that functionality into the program.

Admittedly, writing plugins in VimScript isn't as lovely as writing everything in Clojure, but now that I've written two plugins in it I can't say that it's the worst language, either. It's just one that needs some better documentation and introductory material, which smarter people than I are already working on.

People tend to be fanatical about their editors, and so I don't anticipate anybody switching to Vim on basis of what I've covered here. However, for those of you already using Vim, I hope that this discussion of my workflow has been helpful. And for those of you who are curious about what the world of Vim is like, I hope this post shows that choosing Emacs for Clojure doesn't have to be a foregone conclusion.

I've written previously on this blog about my love for Clojure. When working in Clojure, I use Leiningen as my build tool and general Swiss army knife, and Vim as my editor. I use a healthy number of plugins for both. This post will focus on

I've written previously on this blog about my love for Clojure. When working in Clojure, I use Leiningen as my build tool and general Swiss army knife, and Vim as my editor. I use a healthy number of plugins for both. This post will focus on Leiningen, with a second post to follow that will focus on Vim.

For the record, I consider the state of my current toolchain to be good, but not great. That having been said, the focus of this post will be descriptive rather than normative.

This was written for people who either haven't used Clojure and are curious what a Clojure developer's toolchain looks like, or for people who currently use Clojure and are interested in learning about how other developers work within the language's development ecosystem.

Leiningen

Leiningen is, by far, the most popular Clojure build tool. However, it does a lot more than just serve as a build tool - so much so that instead of trying to enumerate it's features I'm going to cheat and just quote the authors directly:

Leiningen is for automating Clojure projects without setting your hair on fire

It's obvious what they mean by that, right? Of course it is. Don't mind my avoiding eye contact with you, that's just my way of saying we agree.

Why Use Leiningen?

Having to access a language via a Java jar is not ideal. Nobody wants to have to type this on a regular basis:

java -cp clojure-${VERSION}.jar clojure.main

So, what are our other options? Clojure code is (or at some point will be) Java code, so we could rely on traditional Java build tools like Ant or Maven. Speaking personally, however, I've yet to meet anybody who was particularly enthusiastic about either of those.

Fortunately for us, Leingingen actually is a decent multi-tool - it has a good build system, convenient hooks into core project aspects like testing and application initialization, is highly extensible via hooks, middleware and plugins, and has strong ecosystem buy-in. It's not going away any time soon.

At the moment, I'm only aware of one really compelling reason not to use Leiningen, which is if your team is primarily a Java shop. If you're only dipping your toes in the Clojure ecosystem, chances are that you'll get a lot more mileage out of continuing to use your existing tools.

Extra Credit, Part I: The name "Leiningen" comes from the famous short story Leiningen Versus the Ants. The observant reader can hopefully put two and two together here.

Extra Credit, Part II: Why not Boot? The very short answer is just that I haven't tried it, but it's still fairly young and I don't have a compelling reason to leave the land of Leiningen. I'd be eager to hear from developers who've had experience working with it.

How You Use It

Leiningen handles all of its configuration through a set of core config files, with different files sitting in different locations, according to their corresponding scope. For the current project, there's a project.clj file that sits in the project root. For user-level configuration, you can also store a profiles.clj file in the project root (that's usually left out of version control). Cross-project user-level configuration goes in ~/.lein/profiles.clj.

As mentioned earlier, Leiningen has quite a few core features, some of which I use on a daily basis. Here's the hall of fame for me (which, unsurprisingly, has a 1:1 mapping to the Basic Usage section of the Leiningen README):

lein new [TEMPLATE] $NAME

Creates a new Clojure project. If you don't provide a template, creates a library-style proejct, with /doc, /resources/, /src, and /test directories, as well as project.clj, README and LICENSE files.

...this obviously doesn't get used on a daily basis, but is still extremely handy for when you just want to bootstrap a handy skeleton. The optional [TEMPLATE] parameter is fantastic for when you want to make something a little more complicated that has to plug into someone else's ecosystem (e.g. making a Leiningen plugin with lein new plugin $NAME), or for when you're starting out with an application (lein new app $NAME).

lein test [TESTS]

Runs (clojure.test/run-all-tests).

It's actually a little cleverer than that. Leiningen lets you set metadata on tests so that you can filter for particular test sets in addition to configuring which directories contain test code. Don't feel like waiting for all of your integration tests to complete every time? Flag those puppies with a ^:integration metadata flag and configure test selectors to only run when you do lein test :integration (or better, lein test :all)

The optional [TESTS] parameter is useful for when you're not sure what the state of your application is, and just want to poke at a single namespace, or even a single test, e.g.:

lein test :only api.test.controllers.ach

or

lein test :only api.test.controllers.ach/valid-ach?-works

lein repl

Launches an nREPL (networked REPL)! Probably my favorite, because - in case I haven't said this before - I love REPLs. Clojure nREPLs, unlike most REPLs I am familiar with, launch not only an interactive client but also a running server with an exposed port that other nREPL clients can connect to.

One server, many nREPLs - with state shared between them. Dun dun dunnn...

This is handy for many reasons, but I get the most mileage out it by using an editor with a built-in nREPL client. You'll have to read my toolchain post on Vim to learn more. Seriously, it's so cool. You're going to be so excited. I almost want to tell you right now, but I won't ruin the surprise.

lein run -m my.namespace

Initializes and runs your application from the target namespace. Typically, most applications will have specified the main namespace in their project.clj, and so will only need to pass lein run.

An aside on nREPLs and mains: When I'm developing, I almost always want both the application and an nREPL server running. Why not do both at once? I've gone back and forth on whether it's better to develop by writing code within your application that starts an nREPL server or to launch your application by invoking the main method from an nREPL session.

I have a soft preference for the former, but I've been trying out the latter and it's growing on me. If you do go the latter route, it helps to have log output going to a file, so your nREPL output and application logs don't go through the same terminal window.

lein uberjar

Compiles a Java jar with all of its dependencies - perfect for when you want to deploy in a Docker container.

An aside on the subject of jar compilation - compiling a Java jar forces you to pick a main namespace to be compiled into a Java class, which will be loaded by the Java class loader. The differences between the Clojure and Java class loaders aren't trivial, and can result in some unexpected (and probably undesired) changes to an application's initialization process.

All of which is to say that if you've been running your application in production using Leiningen, switching to using an uberjar is definitely the sort of thing one should test rigorously before trying to get it going in some sort of container. Trust me on this one, I'm an expert.

lein deploy clojars

Okay, this is really my favorite, because it's the moment when you take all of your code and wrap it up in a pretty box and send it out into the world to play with all of the other little libraries. Don't they just grow up so fast? *sniff*

Word at the soda fountain is that public key cryptography is a thing these days. Leiningen lets you configure your deployment step to automatically sign your jars with your public key, which is nice if you care about someone not maliciously injecting code into the projects of all those poor saps who trusted you as a dependency.

Leiningen Plugins

The maintained list of Leiningen plugins is long, and there are even more plugins out there in the wild that aren't on the list. Personally, I only use a few plugins, and they largely relate to maintaining code quality. At work, we use Phabricator, and I've written hooks for some of them to run automatically when submitting a diff for review.

Cljfmt

At long last, Clojure finally has its own gofmt-like tool, thanks to the ever-industrious @weavejester. Cljfmt both checks (lein cljfmt check) and formats (lein cljfmt fix) your code nicely in adherence to the generally accepted Clojure style rules, and has options for configuration in case you have your own feelings on the subject.

Eastwood

Eastwood is a Clojure linter, invoked with lein eastwood. As a general request, please use a linter. Some of my favorite moments in the last year have come from people trying to merge code with lint errors and insisting that the linter was wrong. It's like an open admittance that they've made a mistake but are unwilling to do the work to understand what it is.

I make no claims to perfection, myself - Eastwood regularly catches minor mistakes like misaligned parenthesis in tests or out-of-place docstrings - but Eastwood at least prevents me from quietly sneaking such errors into my code.

I'm not particularly strict in my enforcement of their recommendations, but they're great tools to have available.

Ultra

Is it shameless to hype your own plugin? Probably, but I'm going to do it anyways.

Ultra is a plugin for a superior development environment (especially when your workflow is oriented around the terminal). It adds pretty-printed diffs to your test output, syntax highlighting in your REPL, more legible stacktraces, and functions to make interactive Java usage more friendly.

If you're curious, I've got a more extensive writeup on Ultra that you can read here.

Conclusion

As of the State of Clojure 2014, 98% of the respondents were using Leiningen as their build system - the ecosystem buy-in is almost total, so if you're writing in Clojure chances are pretty good you're using Leiningen and are probably familiar with it.

Even so, hopefully the above has been helpful for those of you who aren't familiar with the Clojure ecosystem, or who are still sitting mostly in Java-land and are wondering what things look like on the other side.

As I noted at the beginning, I'll have a follow-up post to this coming up shortly on how I use Vim as part of my Clojure toolchain, which will also touch on how I get my different tools to play nicely with each other.