Posts Tagged ‘design’

People always seem to want to make IT / Enterprise / Software architecture much more complex than it actually is. There are a plethora of frameworks, tools, approaches, methodologies etc. that all claim to offer the solution. I’m sure you’ve seen them. Maybe you might find of them are useful, but in general they distract from the point that architecture is actually really simple.

If you’ve ever worked in a Ruby environment, and especially if you work with the excellent Ruby on Rails framework, then you will probably have seen the emphasis put on convention over configuration.

This philosophy has worked wonders in the Ruby world, but I’m going to argue that it is hitting the limits of it’s usefulness as we engineer more sophisticated applications that need to span across multiple frameworks and tools.

I believe that there is a more important principle that we should adopt. Rather than letting a framework handle everything for us by convention, we should be using higher order techniques from functional programming to explicitly compose our solutions from pluggable building blocks. Let’s call this principle composition over convention.

In Lisp, you don’t just write your program down toward the language, you also build the language up toward your program. As you’re writing a program you may think “I wish Lisp had such-and-such an operator.” So you go and write it. Afterward you realize that using the new operator would simplify the design of another part of the program, and so on. Language and program evolve together. Like the border between two warring states, the boundary between language and program is drawn and redrawn, until eventually it comes to rest along the mountains and rivers, the natural frontiers of your problem. In the end your program will look as if the language had been designed for it.

I’ve personally experienced the power of bottom-up design in my work on my current hobby-project clisk, which is a library for procedural image generation. By building up a set of basic image generation operations, I’ve created a DSL which is proving to be extremely expressive for generating some quite sophisticated procedural images. As an example, the following code:

This generates a nice fractal landscape texture map with a bit of shading:

Note clisk isn’t a fractal landscape generator (the only thing that is landscape-specific is the hand-crafted colour gradient used for the landscape colours) – it’s just a general purpose procedural image generation DSL which has sufficient expressive power to generate fractal landscapes in just a few lines of code. This is the power of bottom-up design in a language like Clojure.

So what’s the downside?

The issue I encountered was that of managing dependencies. If you build bottom up, you are necessarily creating dependecies on the fundamental building blocks that form the foundation of your bottom-up design.

Dependencies aren’t necessarily a bad thing as long as they are stable. But that isn’t always the case. In clisk, for example, I discovered I needed some extra functionality which necessitated changing the fundamental data structure that I had used to represent the nodes in my DSL. Refactoring this was a *huge pain* which took perhaps a whole day of work because I had to rewrite and re-test 50-75% of the higher-level functions in the DSL that depended on the code node data structure. Not pretty.

But then I noticed something – my sample code using the DSL still worked (e.g. the landscape map generating code above). Despite all the refactoring pain I had gone through, none of the samples needed any refactoring. This was a lightbulb moment…..

So what is the solution to managaing dependencies in bottom up design?

Build against an abstraction

Observe the difference:

In my library code, I had built everything against a concrete data structure. When that changed, all the dependencies broke.

In my sample image code, I had built against an abstraction – the clisk DSL itself – which insulated me from the changes in the underlying data structure. When I rewrote the internals of clisk, everything still worked.

I think the principle applies in general to bottom up design. You should aspire to build in layers, each depending on an abstraction which is as simple as possible but no simpler to enable you to use the capabilities of the layer below.

In the clisk case I had three layers:

Sample code

Clisk DSL functions

Base data structure

But this probably needed an extra layer:

Sample code

Clisk DSL functions

Primitive operations (on data structure)

Base data structure

If I had done this in the first place, then the refactoring work would have been minimal as I would only need to rewrite the primitive operations – not the whole extensive library of Clisk DSL functions.

So this turned into a bit of ramble…. but I think I’ve learned something valuable and I hope it is useful in providing some insights into the pros and cons of bottom up design. And as often seems to be the case in Clojure, abstractions turn out to be the hero that saves the day…..