Dialing back the cleverness

Last night I ran into Damian Conway at a speaker’s dinner for this week’s GOTO conference. He’s one of the people I had in mind when I said I enjoy hearing from the Perl community even though I don’t use Perl.

We got to talking about Norris’ number, the amount of code an untrained programmer can write before hitting a brick wall. Damian pointed out that there’s something akin to the Norris’ number for skilled programmers. He said that a few years ago he took this quote from Brian Kernighan to heart:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

He said he’d realized that he’d written some very complex code that was approaching his level of cleverness and that couldn’t move forward himself, much less get anyone else to take it ownership. “That’s when I decided to dial back the cleverness a bit.”

8 thoughts on “Dialing back the cleverness”

I remember, once upon a time, I would try to minimize the size of my code, or make it faster, or that kind of thing. After a certain point, I imagine my code was “clever”? I know for certain that squeezing the last 5% of speed or size or whatever out of my code would almost invariably result in something that was painful to work with when I discovered something that had to change.

Anyways, the issue here, I imagine, is abstractions — our code represents concepts which (in a complex system) we are discovering as we go along. (If we could start with a perfect specification, that properly addressed all ambiguities, and we could start with a complete understanding of that specification, and if we could guarantee that the specification never changed and we could prevent new requirements from coming in…. we would probably have to stop all technological progress to achieve that.)

So, anyways, we have discovery process. processes. discovery without process. new stuff…

So that means we need to build systems that can change.

Breaking that down:

1. Before the change, our system is inadequate (broken in some new way but perhaps not broken in the old way).

2. After the change, our system is different (perhaps not broken in some new way, but if things were perfect before they are not that way now).

So, from my point of view, this process of adaption is at cross purposes with concepts of perfection. And, ironically, in my experience, the worst possible way to deal with this change process is to “design for the future”. The problem here, is that I never have the requirements that are going to arrive later. And when I try to design for them I invariably pick the wrong design. So, instead, the best thing to do is go for the simplest possible thing that works.

Simplicity brings with it some other advantages: it’s usually relatively short (since I’m leaving out all the complex stuff), and it’s usually rather fast (since I’m leaving out a bunch of unnecessary stuff).

Note also that some judicious use of mathematical abstractions can be very helpful when simplifying. (But please do not assume that by “simple” I mean “rote” or “uninformed” or “obvious to someone who does not understand”. Nothing is obvious to someone that does not understand unless they are looking at the wrong thing — obviousness is a concept used to describe stuff we do understand.)

So, anyways… simplicity is good. And when cleverness takes you someplace that’s not simple? That is not so good.

But I’m not prepared to say that cleverness is bad. After all, my feeling is that making things simple also requires cleverness. My hunch is that cleverness (whatever that is) is not the problem but that the directions we give our cleverness are the problem.

Speaking of Perl, Perl 6 turned into a morass of cleverness and second-system effect. It not only includes everything everyone wanted, it does it in torturously clever ways. I think this has a lot to do with the lack of adoption by real software developers.

@rdm I think there’s a particular type of cleverness that this refers to that is hard to describe but not so hard to recognize.

It’s the code where when someone looks at it they ask “why does this work” (which is already a bad sign) and you answer with “well you have to think about it like ” and the asker starts furrowing his brow and rubbing his eyes. Maybe when this is all done, if you’re lucky, he’ll say “oh I do understand… yeah, that’s quite clever.”

…but sometimes that’s justified. Using quaternions and/or eigenvalues in graphics programming, for example, can sometimes be useful examples of “clever”.

All too often, though, it’s pointless. That’s when it’s bad.

And that’s the real problem, I think — when we are doing useless stuff, or doing things for useless reasons.

But, yes, also: when describing our code is difficult it’s often much easier to change the code to make it easy to describe than it is to describe the unchanged code. And yet, I almost never see “writing documentation for ___ audience” as an architectural or design issue in any of the development projects I’ve been working on.

I wrote some code that generates other code using templates. The core was one line, something like print(subst(read(open(“template.txt”)))). The templates are written in the target language with nested code that gets eval’d by subst(). It was a clever solution but I have a feeling that some other programmer eventually will need to modify it, won’t understand the substitution rules, and will take it out and replace it with something easier to understand, like regexp substitution.