If you are a software developer who feels that the “state of the art” is “not
entirely satisfactory,” you wouldn’t be the first. And you’d be in good
company. In 1984, Donald Knuth, perhaps the greatest benefactor that the realm
of computing has ever known—and no mean writer—introduced a concept he
called “literate programming,” saying,

Let us change our traditional attitude to the construction of programs: Instead
of imagining that our main task is to instruct a computer what to do, let us
concentrate rather on explaining to human beings what we want a computer to
do.2

Literate programming inverts the traditional relationship between code and
commentary. Instead of code files with scattered remarks, Knuth’s system (which
he called WEB, because the pieces of the program were woven together) made the
document the first-class citizen, with the code interpolated as needed for the
sake of explanation.

Knuth wonders whether literate programming

may be only for the subset of computer scientists who like to write and to
explain what they are doing. My hope is that the ability to make explanations
more natural will cause more programmers to discover the joys of literate
programming, because I believe it’s quite a pleasure to combine verbal and
mathematical skills; but perhaps I’m hoping for too much.

I’ve only been practicing literate programming for a year, so I don’t pretend to
have it all figured out. But in that time, I’ve found several good reasons why
literate programming is a good fit for this project.

Software systems are like sand castles: they’re wonderful, silicon-based
creations that will be gone tomorrow. Maybe you’ll come back and rebuild it.
At some point, you won’t. The only way to keep software alive for generations
is to teach people how to rebuild it.

And in literate programming, it is culturally unacceptable to change the system
without first “teaching the change.” Starting with yourself: you can’t explain
something until you understand it. Literate programming forces you to explain
everything. Literate programming is mostly about writing, and writing is a
thought process.

In other words, any system whose specialty is the amplification of documents,
will have a lot to offer, not only for Shakespeare and related works, but also
for the system itself, if it should choose to use document form.

As things stand, there is a great divide between design and implementation.
This divide limits the depth of collaboration between those who understand the
subject matter and those who understand the technology. This is familiar to
anyone who’s ever been involved in a software project. It’s critical to
communicate to a general audience, or at least to people without technical
knowledge, and literate programming works to bridge that divide.

It’s not clear that Knuth was thinking explicitly about any of these particular
benefits when he first proposed programming as a literary act. But it is clear
today that we need to embrace the practice. All non-trivial programs are
multi-language systems communicating with other multi-language systems. The
dictates of compilers and frameworks are as unconducive to understanding the
full picture of a system now as they were in 1984.

Willshake’s literate programming system is used to help make the system more
understandable. One way that it does that is by focusing on features.

After trying this method for a while, I was convinced that it was a good thing
and set out to write about it. In trying to find anyone else pursuing this
idea, I was lucky to find one person—Kartik Agaram—who explains the
situation very well in his post called “A new way to organize
programs."3 He uses a different technique (which he calls
“directives”) to achieve the same result, but that’s not important. His
explanation of the underlying situation is perfect on every point, and I will
rely on it here.

That said, I’m not sure what to call this concept. I have called it by
variously unfriendly terms, including “additive programming” and “lexical
feature scoping.” As I read his article, Agaram uses the the term “layers” to
mean the composable parts of a system, and “features” to mean a certain kind of
layer.

This is closely related to literate programming. First, it favors a topical
view of the system, by asking that the program be organized in the units that
are meaningful to the person (features), rather than the units that are
meaningful to the computer (bits of code).

we can now organize a program as a series of self-contained layers, each
containing all the code for a specific feature, regardless of where it needs to
go

For our purposes, a “feature” is an atomic unit of deployment. It is a set of
changes that can be added to or removed. A feature is defined by a set of
documents. Adding the documents adds the feature; removing the documents
removes it.

A feature should be the kind of thing that you can propose. So you have to
start from the most basic thing. There are some things that you can’t really
break down any further, or that are so ingrained into the structure that you
can’t isolate them in any meaningful way. The latter are generally called
“cross-cutting concerns.”

Suppose you have a useful product. People use it, and people like it. Then
someone proposes a new feature. You consider it, and you consider it a good
idea. So you write the necessary documents to implement it, and add them to the
program. Does the existence of this new feature change the fact that the
program was good and useful without it? No.

The philosophy of additive building is that adding means adding. That’s it.
From this, some things fall out:

adding doesn’t mean breaking what you have

you can remove what you’ve added, and you’ll get what you had before.

By “add” and “remove” we are talking about lexical scopes, and particularly
files. That is, a feature consists of a set of files. By adding those files to
the project, you get the new feature. By removing those features, you get what
you had before.

There is a special relationship between this method and the practice of literate
programming, since you can bundle multiple outputs into a single document. As
such, you can add the feature simply by placing the document with the rest of
the program, and you can remove it by removing the document.

Does this mean that you can’t change anything outside of the new document when
adding a feature? No. It just means that you can’t make breaking changes. You
might need to add some invisible metadata, or some kind of hook or entry point
that didn’t exist before. Or you might need to restructure how things are put
together to make the necessary modification possible. But such restructuring
should not break what you had before. (Such changes are often called
“refactoring.”) In those cases, “removing” the new feature may mean leaving
those changes in place, since they are benign and may even be generally useful
for the addition of other features. Whether this may include some observable
adjustments to the product is a subjective matter.

This is a way of composing a product from features that are specified in
document form. It makes the “composition” of a program from atomic parts a
first-class concept.

Yet another way to think of this is that you can undo anything that you do. Of
course, there are lots of mechanisms for undoing, not least being a revision
control system. But there’s a difference. Assume that at certain points the
product reaches a stable place. These checkpoints represent usable products.
Then people propose new features, and people work on them, checking in changes.
The older state of the program is available through the revision control system
by going back in time to it. But in additive building, the older version is
always with you, representing the fact that the new feature hasn’t changed the
fact that it was valuable and usable in that state.

Finally, does this mean that you are always adding things? That you can’t just
change something in place? Of course not. If you don’t like the way something
is, change it. Or get rid of it.

Does this mean that every document defines a feature? In other words, can every
document be treated as safely removable?

No, I think not. Indeed, you might use that question as a test for whether the
document represents a feature. A document may be used to add some other
mechanism that is used by a number of other documents. In that case, it’s more
like a “service” (a build service, assuming you’re talking about a build-time
mechanism, which is the kind that documents—as opposed to the programs that
they produce—can use).

The “additive” philosophy also extends to the design of user interfaces. This
is not strictly a consequence of additive programming, but it’s in the same
spirit.

New features should be new features. But this is obvious, right—/of course/
new features are new features, what’s the alternative? The alternative is that
new features tamper with old features, either by removing them, or by altering
them. Why would this be bad? Because you know that the product was working
before. If there was a better way for those controls to work in the earlier
version, then the earlier version should be made better, and then the new
feature considered. This way of thinking also discourages the use of “modes,”
which is a common way to provide lots of features in a small space.4

Doesn’t this limit how many features you can add? Only to the extent that they
are all competing for the same space. But if they are added through “expansion
points,” which grow the overall space without re-packing particular views of it,
there is much more latitude for expansion. Even then, it’s a challenge to keep
adding to something without overloading it. As Rich Hickey puts it, “more stuff
implies more stuff."5 So to the extent
that an additive design encourages additions to be made carefully, that is a
feature.

So, in general, new features should not change the meaning of old features, even
though they may need to be rearranged.

General sense of “a definite plan or scheme” is recorded from 1837. Meaning
“list of pieces at a concert, playbill” first recorded 1805 and retains the
original sense. That of “objects or events suggested by music” is
from 1854. Sense of “broadcasting presentation” is from 1923. Computer sense
(noun and verb) is from 1945. Spelling programme, established in Britain, is
from French in modern use and began to be used early 19c., originally especially
in the “playbill” sense. Program music attested from 1877.

program (v.) Look up program at Dictionary.com
1889, “write program notes;” 1896, “arrange according to program,” from program (n.). Of computers from 1945. From 1963 in the figurative sense of “to train to behave in a predetermined way.” Related: Programmed; programming.

Literature is first and foremost about having ideas important enough to discuss
and write down in some form. So you have to ask, “What is the literature that is
best written down on a computer?” One answer is to make a dynamic simulation of
some idea that you think is important, a simulation that you can play with and
that you can learn from.
(Alan Kay)

Without wanting to be elitist, the thing that will prevent literate programming
from becoming a mainstream method is that it requires thought and
discipline. The mainstream is established by people who want fast results while
using roughly the same methods that everyone else seems to be using, and
literate programming is never going to have that kind of appeal. This doesn’t
take away from its usefulness as an approach.

This article begins with a fairly gentle introduction to Literate Programming,
although

One thing that Literate Programming allows you to do is to focus on the
interesting parts first. A standard principle in writing big poetic epics is to
start not at the beginning, which can be dull, but in medias res, in the middle
of things where it’s exciting. So the Iliad doesn’t start by recounting the
years of inconclusive battles between the Achaeans and the Trojans which
characterized most of the Trojan War. Instead, it starts nine years into it when
Achilles throws a hissy fit and things start to get interesting.6

Willshake is an experiment in literate programming—not because it’s
about literature, but because the program is written for a human audience.

Following is a visualization of
the system. Each circle represents a document that is responsible for
some part of the system. You can open the documents by touching the
circles.

Starting with the project philosophy as
a foundation, the layers are built up (or down, as it were): the programming
system, the platform, the framework, the features, and so on. Everything
that you see in the site is put there by these documents—even this message.

Again, this is an experiment. The documents contain a lot of “thinking
out loud” and a lot of old thinking. The goal is not to make it perfect,
but to maintain a reflective process that supports its own
evolution.