Archives

The Unix philosophy of Lego bricks

As a child I spent an inordinate amount of time playing with Lego (was
it just me or did anyone else invent names for the bricks? I had
Long-ers, Square-ers, Tippees, Flat Tippees, Very Flat Tippees, Tipping
Up Tippees...) The fact that a tubful of basic components could become
the most fabulous spaceship I was capable of imagining had a huge
influence on me and was probably a large factor in why I became a
programmer.

I really think I'm onto something with cmeScribble but I've had a
hard time coming up with a way to explain exactly what I think I'm
onto. Part of that is because, obviously, I'm writing largely for an
audience that isn't familiar with cmScribe, the philosophy it tries to
embody, the strengths of its approach or the limitations we're starting
to hit. Perhaps I should let my ten-year-old self explain by analogy.

We needed a powerful layout system so we came up with Lego.
Then we needed flexible hierarchical navigation so we built in Duplo.
Then we needed some place to put the content so we made a Rubik's Magic.
We built an eCommerce system out of Meccano.
We built a document repository out of Rubik's Snakes (those were great, weren't they?)
We made a general-purpose form builder out of MegaBloks.
We made a permissions system out of whatever that cool thing was that
had all the translucent octagonal pieces with slots in the sides.

And every time we add a new feature we get a different type of construction toy to make it out of.

We've done a good job of making all these things work together. To
display products in the nav we built an adapter that allowed Meccano to
attach to Duplo. We built adapters so that octagons could be attached
to anything. We built adapters to connect Meccano to Rubik's Magic. We
built adapters and adapters and more adapters.

What we've ended up with is something that's extremely flexible and
powerful, but in doing so we've solved the same problem lots of times -
making one kind of toy talk to another, or making one toy appear as if
it's a different kind of toy, or solving a problem for one kind of toy
that was already solved for one of the others (notably, our
Rubiks-Magic search is entirely separate from our Meccano search).

The idea behind cmeScribble is "what would happen if we just did it all with Lego[1]?"

The Lego philosophy and the Unix philosophy have a lot in common. In
cmScribe we got the bit about "Do one thing and do it well" right,
whether the one thing is "match a regular expression" or "be two by
four, full height and red" or "display a product list". But just as
important, if less expressable in a pithy slogan, is that the reason
Lego and Unix work as well as they do is that every part has a common
interface. In Unix it's the plain text file with each line representing
some kind of value. In Lego it's the little round bump and the little
bump-sized hole. In both cases the power comes from the fact that any
piece's bumps can attach to any other piece's holes without having to
create a custom Long-er-To-Very-Flat-Tippee-Adapter.

In the next thrilling installment: What do I propose as cmeScribble's little-round-bump-equivalent?

Are you mocking me?

Before I wrote a line of code for cmeScribble I decided that I'd code
it the right way. Retrofitting "right-way-ness" onto an existing
project is almost impossible, so I was determined not to miss the
chance to build it in from day one.

The particular "right way" I had in mind was using Test-Driven
Development, or TDD. For anyone unfamiliar with the TDD methodology,
the idea is that before you write any code or fix any bug you first
write a test or two to make sure the code works or the bug is fixed.
The test should fail when you first run it, because the code doesn't
work and the bug hasn't been fixed. Once you've written the test and
verified that it fails, then you write the code to make it pass.

The advantages of TDD are well-documented elsewhere, but in a nutshell:
writing the test forces you to know in advance exactly what you want
the code to do, and also as you develop more and more code you have a
library of tests to make sure you don't break anything.

I could easily see how to write tests for self-contained functions, but
I couldn't figure out how to write tests for user interfaces or
database-backed logic. Since cmeScribble is all about building user
interfaces from database data, this posed something of a challenge.
Maybe I can get away without testing the UI layer, but I can count
features that won't need database data in some way on the fingers of
Captain Hook's right hand. Okay, I thought, Google to the rescue... but
searching for databases and TDD provided lots of people with similar
issues and no straightforward answers.

A common suggestion was to use "mock objects". The idea is that you
create a "fake" version of your database layer which returns plausible
values without actually needing a database. This solves the problem but
the downside is that you have to actually write the mock version of
each database table, and since cmScribe has over a hundred tables at
last count[1] I didn't relish that thought much either.

Facing a potential deal-breaker for something you don't want to
compromise on tends to focus the mind, and eventually I hit on a
solution. A full SQL database is hideously complex, but cmeScribble
will be doing it's database access through nrdo (I'm biased, but I
honestly don't see how people can work any other way...) and for 99% of
uses, nrdo provides a fairly simple object model over the top of the
database. Simple enough that if you don't care about persistence (which
in a test situation is actually a bad thing) or performance, it's
possible to implement the vast majority of it very simply in memory, as
part of the nrdo runtime library and generated code, backed by simple
List<T>s. A couple of days of hacking later and that's exactly
what I've now checked into nrdo's CVS: If you use the C# 2.0 template
and set the configuration setting "NrdoMock" to "True", all your code
will run purely in memory, with a clean "database" for each run.

There are a few things missing - at a wild guess I'd say it covers 60% of
what you'd want to do right now, and I should be able to get that up
to about 90% without too much trouble by filling in things as I need
them (after writing tests!). Most of the rest can't be handled
automatically (because they'd require a full SQL engine in the mocking
layer) but can be done with a little manual intervention (by the
programmer providing a C# equivalent of the SQL clause in question; the
SQL clause itself doesn't get tested but all the logic around it does).

The most satisfying part of the whole thing was watching it pass the tests I'd prepared in advance :)

cmeScribble - coming soon!

Last night I came up with the seed of a way to implement some things in
cmScribe (the CMS that I work on as a day job and happens to run
my website too) that I've wanted to do for ages but not known how - and
get a bunch more cool features that I never even thought of before for
free. That was the good news. The bad news was that the ideas are a
radical departure from the way cmScribe works now and would require an
almost complete rewrite - only a few pieces of code could be salvaged
anything like intact. It's taken us three years of team development to
get cmScribe to where it is now - which is, in my not-entirely-unbiased
opinion, a pretty good place in lots of ways - so the idea of throwing
all that away and starting over based on a mad programmer's wild and
unproven blue-sky ideas was probably not something I could sell to the
boss ;)

(Basically, the entire architecture of cmScribe today revolves around
building templates and pages out of building blocks that we call
panels. The idea I had leaves no concept of a panel left at all, and
templates and pages don't have any independent existence but just
become particular ways of combining lower-level ideas...)

But I think I found a way to turn the bad news to my - and NetReach's,
and maybe your - advantage. Because it does mean a complete rewrite,
there's no need to use any existing cmScribe code to implement it. So I
suggested to my boss that I'd be happy to work on this in my own time
as a hobby project / technology testbed, and wouldn't it be cool to do
it as an Open Source project? As a hobby project it wouldn't even
attempt to provide the "enterprise" features that cmScribe does, but if
it successfully proves the concept, it would also be possible to use it
to rebuild those enterprise features and get a next-generation cmScribe. My boss agreed.

So without further ado, I give you...

Well, actually, I don't, because all cmeScribble is right now is a
logo, a pronunciation (see-me-scribble!) and some partly-formed ideas.
There are parts of the design that are defined as being done "by
magic". And while some incredibly rich and powerful scenarios are
enabled by this architecture, I'm a little fuzzy on how to implement
obscure things like, say, pages.

When I have some code (I've started, but I don't want to release until
it can do something that at least gives a hint of the power to come)
I'll put it online for people to try out. In the meantime I'll blog a
bit about the process by which I arrived at the new design, where I
think I'm going with it, and why you should care. Or at least give you
enough information to decide that in fact you don't ;) As I write more
about the ideas I have I'll also be interested in feedback on whether
what I'm doing makes sense, whether there are important scenarios it
misses, and what I could do to make you interested in using it.

Ok, at this point I'm announcing vaporware. Sorry; I'm excited about
this and don't have the discipline to wait until it really works.
Besides, I've noticed that when I see an announcement of a new project,
if it's provided as a fait accompli with lots of code and
documentation, I'm not terribly likely to invest the time to delve
through all that to learn what it's all about. If someone on one of the
planets I'm subscribed to introduces their project over a series of
blog posts, though, I find myself learning quite a lot about the
project - and maybe wanting to use it - without feeling like I've
invested any time or effort at all (after all, reading blogs is what
you do to avoid putting effort into something else). If I cmeScribble
can benefit from that effect in others, everyone wins.

Oh, and I guess I have to say that it's very Web 2.0. I'm not sure
what that means except that all the cool kids are doing it these days.
As far as I can understand of what that particular meaningless buzzword
means, though, it does actually fit.