software development and consulting

This is the first in what will likely be a series of blog posts about Clean Architecture. Uncle Bob Martin has written numerous blog posts and given lots of talks about it.

The goal of Clean Architecture is to have the directory structure of your application shout out what your application does rather than what framework was used to present your application or what database is nestled in the depths of your application. Your program is divided into Entities, Use Cases, and Interface Adapters.

Entities encapsulate “Enterprise-wide business rules.” Use Cases encapsulate “Application-specific business rules.” Interfaces and Adapters represent how your Use Cases want to interact with the outside world (e.g. databases, users, printers, etc.).

In Clean Architecture, the Entities cannot know that the Use Cases exist and the Use Cases cannot know anything about the Adapters except for the Interface to them which is defined by the Use Case rather than by the Adapter. The Use Case does not know whether the application is being used from the command-line or from the web or from a remote service calling into it. The Use Case does not know whether the data is being stored in the file system or in a relational database (or conjured from the ether as needed). Nothing in the Adapters can know anything about the Entities.

Simple Example

I have a project that I am just starting. I thought I would use this new project to see how Clean Architecture works for me.

There are large number of talks and videos about Clean Architecture. However, there are not many examples of it despite several years of Stack Overflow questions and blog posts asking for examples.

There are a few simple examples around the web. The most notable is Mark Paluch’s Clean Architecture Example. It is just big enough to get a sense of how things hang together. If you’re willing to put up with Java’s insane directory hierarchies, you can get a pretty good idea of what the application does just by poking around the Use Cases directory.

My First Use Case

My first Use Case is to let the User browse a list of Book Summaries. The User should be able to sort by Title, Author, Publication Date, or Date the Book was acquired. The User should be able to filter the list based upon Genre or Keyword. The Use Case should allow the caller to implement pagination, so the Use Case needs to support returning up to a given number of Book Summaries starting with a specific number.

Some might argue that that is multiple Use Cases glommed together. If that were the case, then I would need some way to pipeline together Use Cases if I’m going to make any sort of reasonably navigable app atop my Use Cases.

But, let’s start with baby steps.

The Simplified Version of my First Use Case

Let’s just say the User wishes to see a list of all of the Book Summaries. The User is fine with seeing all of them at once in whatever order my app sees fit.

There is one use case browse-books which defines the use-case interface browse-books-use-case along with its input structure browse-books-request and its output structure browse-books-response. The use case defines the method browse-books which must be called with a browse-books-use-case instance, a browse-books-request instance, and browse-books-response instance.

(defstruct browse-books-request)

(defstruct browse-books-response
list-of-book-summaries)

(defclass browse-books-use-case()())

(defgeneric browse-books (use-case request response))

In my implementation, the browse-books-response is a simple data structure. One could easily imagine that the browse-books method would return one rather than filling one in that was passed to it. In some variants of Clean Architecture (like the Paluch example cited above), the response model is a class instance upon which a method is called to complete the interaction. But, it would have to be clear from the outset that anyone using this Use Case cannot depend on it being synchronous or asynchronous.

The use case also defines the book-repository interface that it needs.

In the Paluch example, all of the use cases share the same repository interfaces (though Paluch and others have separate repository interfaces for Users and Invoices and Items). In several of Uncle Bob’s videos, he makes the claim (or claims equivalent to the claim) that each use case should define an interface for just the methods it needs to use on a Book repository. In this use case, it would need only the ability to retrieve the list of Books and so I should not have defined find-book-by-isbn here at all, and I should have named this interface browse-books-book-repository.

I wrote browse-books-impl class which extends the browse-book-use-case. It takes an instance of book-repository on construction.

Trouble In Paradise

Already, in this simple interface, I am torn. For this Use Case, I do not need the repository to return me the list of Books. I could, instead, ask the repository to return me the list of Book Summaries. If I do that, my application is just a fig-leaf over the repository.

(defgeneric all-book-summaries (book-repository))

Well, the argument against asking the repository for Book Summaries is that it should not be up to the database to decide how I would like to have my Books summarized. That certainly seems like it should be “business logic” and probably “application specific” business logic at that.

So, fine. I will have the repository return Books and the Use Case will summarize them.

Now, let me extend the Use Case the next little bit forward. What if I want to support pagination? My choices are to push the pagination down to the repository so that I can ask it to give me up to 20 Books starting with the 40th Book. Or, I can let the repository give me all of the books and do the pagination in the Use Case.

(defstruct browse-books-request
start max-results)

Here, I can find no guidance in any of the Clean Architecture videos that I have watched nor in the examples that I have found online. Everyone seems happy with the repositories being able to return one item given that item’s unique identifier or return all of the items.

If the repository is going to return all of the Books, then why wouldn’t my Use Case just return them all and leave the caller to do any pagination that is needed?

This works fine when there are a few dozen books and they are small. It does not scale, and I don’t know how it is supposed to scale without pushing most of the responsibility onto the repository.

(defgeneric all-books-in-range (book-repository start max-results))

Sure, I can push the responsibility onto the repository. But, one of the reasons that Clean Architecture is so structured is to allow easy testing of all of the application logic. The more that I push into the repository, the less that I actually exercise when I run my unit tests with my mock repository (and the more complex my mock repository should probably be).

One possible approach would be to have the all-books method instead be all-isbns. Then, I can retrieve all of the ISBNs and use find-book-by-isbn to get all of the books.

Now, if I want to sort by Author then by Title, I need to:

fetch all of the ISBNs all-isbns,

fetch each ISBN’s title,

sort my ISBNs by title,

fetch each ISBN’s author,

stable-sort my ISBNs by author,

clip to my range,

fetch each book in my range,

summarize each fetched book

Or, I have to write an SQL query, that can do all of that in one database call instead of 2N + R + 1 calls (where N is the number of books in the database and R is the number of books in my range), making my Use Case a fig-leaf again.

The first coding assignment of the Data Management and Visualization class that I am doing on Coursera is just to do some frequency analysis on some of the variables that will be involved in the research you want to do.

Frequency Counts

The assignment was to do frequency counts. If I did tables of raw frequency counts, the tables would be huge. There are 4588 counties in the data set. There are 4075 different values for the total number of votes cast in a county. As such, I bucketed the counts based upon the power of ten of the value. Here is the output for the total number of votes cast:

With the above frequency count, we can see that of the five counties with over a million votes cast, the Democrats got more than a million votes in two of them whilst the Republicans did not get a million votes in any county.

The numbers are pretty close the whole way through, but that still doesn’t mean a great deal. It could be that the fifteen counties where Democrats got fewer than ten votes were counties with ten thousand votes cast. So, I put together a small function then to get the worst counties for a given party:

As you can see from this, there are two counties which show no votes cast. In both of those cases, there are no precincts reporting in the data set. The data set tells the number of precincts in the county along with the number of precincts reporting. These counties with none of the precincts reporting are significant glitches in the data. On the other hand, some counties in the data have hundreds of precincts where all but one reported. I could remove a county from the data if not all of its precincts reported. However, I believe that within a county, single precincts will not differ very much from other precincts which were counted in the data. Further, as I do not have any hope of determining the population density down to the precinct level, I am just going to roll with what I have.

Implementation

One of the things that immediately became apparent is that there are two separate columns in the database labelled "TOTAL VOTES CAST". I wanted to make sure there were no confusion, so I wrote a quick function then to check that both of those columns agree everywhere.

The data here has one row per county. I might have preferred there be one row per county/candidate pair. Regardless, I wrote a short function that takes a party name and all of the columns identifying parties along with the columns identifying how many votes a given party received.

For example, this might get arguments party = "DEM", parties = ("DEM""GOP""LIB""GRN"""""""""""""""""""""""""), and votes = (9169612123455392127NILNILNILNILNILNILNILNILNILNILNILNIL). This function sums up all of the numbers in the votes list where the corresponding entry in the parties list matches the given party.

I made a little data structure to hold the data that I am interested in for each county.

I then created a function which returns a function. The returned function returns an instance of my data structure for the row passed into it. Note: the data set contains rows which roll-up the results for a whole state. For those rows, the FIPS code for the county is zero.

I did that because I originally had all of that functionality in the function which loops over each of the rows in the data set. Now, the function that collects all of these is simpler, but I’m not sure the overall simplicity is much improved.

I created a function to bucket them based on their base-10 logarithm. Of course, this immediately freaked out on the couple of counties for which there is no data in the data set, so I had to take care not to take the logarithm of zero.

I have a couple of nascent projects on my plate which require custom, server-side software. I’ve been trying to use these projects to explore the Clean Architecture concepts using Test-Driven Development (TDD).

I can’t even begin.

According to Clean Architecture, I should start with my application logic independent of whether it will be a command-line tool, a series of tools, a web-application, or what-have-you. So, let me start on the application code.

This is TDD though. So, I need to start with a test. How do I do that?

(ql:quickload:nst)

(nst:def-test-group querying-data ()...)

Wait a minute? I just committed myself to Lisp. I just made a huge business decision, a huge implementation decision, a decision that will shape my life for the next n years, and neither TDD or Clean Architecture had anything to say.

I need building supplies. I can’t go anywhere with my architecture or my development without a programming language.

I want to write my apps in Lisp. One of them, I already have mostly done in Lisp. I am still trepidatious about deploying Lisp.

I use my current web hosting provider to give me a LAMP stack upon which I run WordPress for this blog and some git repositories and a bug-tracking database (that I can’t remember how to log into). All of that, I could move with confidence in a few hours.

What I dread is having to collect the several hundred mail-forwards that I currently have along with the half-dozen IMAP accounts and move them anywhere, let alone to somewhere that I have to manage them myself and deal with SPAM and mail queues and crap.

It seems that for only a few bucks more per month than I’m paying now, I can get Plesk on a VPS that should be big enough for my purposes. Does anyone have any experience with Plesk? Is it going to make email painless for me on Ubuntu? Or, am I going to hate my life? Does someone have a VPS provider they strongly recommend?

Do I really have to write my application in PHP on the chance that I’ll want to deploy it on the web just because email is scary?

During Bush v. Gore, there was a ton of freely available data about that election. I did a quick-and-dirty graph showing that Gore won the most populous counties by wide margins and Bush won most of the rest of the counties by wide margins.

For this course, I am going to revisit that analysis with the 2012 election data and maybe the 2008 election data.

The Hypothesis

In the United States, there is a strong correlation between population density and voting for the Democratic candidate for president.

Related Work

This search turns up severalarticles about a scatterplot by Conor Sen relating the Cook Partisan Voting Index (PVI) plotted against population density based on 2012 data. There are related heat-maps by others from the same data.

That search also turns up a paper by Jowei Chen of the University of Michigan and Jonathan Rodden of Stanford University about why compact voting districts are bad for Democrats. That paper focuses mostly on the shapes of voting districts in Florida and how they have be gerrymandered to make those in population-dense areas very compact while those in less populated areas are tentacled and sprawling and how this results in a higher number of Republican representatives than is warranted by overall population numbers.

A related aspect that shows up in this search is that on specific issues, like transit infrastructure, the congressional voting record is strongly correlated with the population density of the congressperson’s district. This effect is a second-order effect, however. The vote of a congressperson will likely be entwined with what the party as a whole wants as much as (or even more than) their constituents want.

The Tools

I will, of course, being me, use Common Lisp for all of this. I suspect that I will use Fare-CSV for ingesting CSV data. If I have to parse TIGRE data, I will likely rely on some blend of esrap and CL-EWKB or custom geometry code. For plotting, I will likely rely on Vecto but may also try out some of the other libraries like adw-charting or finally get around to making my own multi-backend charting library.

Those are all clearly violated by this code. But, that’s not really the problem. The problem is Why. Nothing about this code tells you why it is here or what is doing.

There is no way to glance at that function and have any idea what’s going on. You have to read it carefully. You have to understand things that aren’t even in this source file to make head nor tail of it. Once you understand the second LET block, you will have nine more lines of code without the least inkling of why there should be nine more lines of code. Anyone care to hazard a guess as to why this function returns T (only) when we’re dragging?

Encapsulation

Two years ago, a colleague and I were tasked with providing docstrings for every function in all of the code we’d written in the last year. We’d done well on providing docstrings to the outward-facing functions, but now we had to do the rest. He started at one end of the directory (in alphabetical order), and I started at the other end. This gave me a good opportunity to look closely at a boat-load of code he’d written that I’d never really delved into before.

He was absolutely religious about encapsulating containers. If he had a hash-table or a p-list or a flat list in a DEFVAR, there was one and only one function that retrieved items from it and at most one function that added items to it. Those functions were one or two lines each (two if they needed a mutex). Those functions were named after what the collection was storing not what mechanism was used to store them.

A lot of times when people talk about the value of encapsulating, they talk about shielding the rest of the code from the implementation details so that if you need to replace how it’s actually implemented on the back end you can do it without breaking any existing code. You are protecting your precious implementation from how people will use it so that you can someday replace the implementation with an even more precious implementation next year (when your language finally gets first-class functions).

I’ve been coding for a good long time now. I’m going to let you in on a little secret. Code almost never gets replaced. When code does get replaced, it almost never continues to adhere to the old API (there was always a semantic leak). If there is a business justification strong enough to let you replace the code, it’s because the old code has become an unmaintainable mess of people subverting the interface or the code as it is didn’t scale and now synchronous things need to happen asynchronously or local things have to happen remotely and hiding that under your old API isn’t going to relieve the bottlenecks.

Trying to insulate your code so that it’s easy to replace is looking down the wrong end of the telescope. The real benefit of encapsulation is that the people who read your code later can be half-asleep and still get everything—your code will scream its meaning. The real benefit of encapsulation is that the person debugging your code can set a break-point in a place that means something—not in the seventeen places the state might have changed but in the only place it could change.

It is still an (ORNULL(CONSINTEGERINTEGER)) but nobody ever has to care. Nobody ever has to try to remember what the integers mean. Sure, you could replace it with a structure or a complex number, but why would you ever bother? Why would you ever look at it again?

The new macros HANDLE-EVENT and IGNORE-EVENT encapsulate the return value of this function into something with meaning.

(defmacro handle-event (()&body body)
`(prog1(values t),@body))

(defmacro ignore-event (()&body body)
`(prog1(values nil),@body))

It might still be too easy to write an event-handler with a path which doesn’t end in one of these two macros, but it is way better than that dangling T was. It looks like it’s really supposed to be there, and it looks like what it means rather than what it is.

The TRANSLATE-WIDGET and KEEP-WIDGET-INSIDE-PARENT functions can benefit greatly with some further helper functions (and analogous functions for top and bottom):

If you find that you are depending on the NULL-ness or positiveness or some other property of some number of state variables to decide which course of action to take, then you should consider making predicates named after your state. In many OO scenarios, you may even want to explicitly track (or calculate) which state you are in at all times.

In more imperative languages, it may even be beneficial to keep a STATE member variable in your class. When doing that, make sure that there is one and only one function which actually mutates the value of that STATE member. This will let you:

Log all state transitions without having to hunt for all of them.

Quickly hunt for all of them if you want to do that

Set a break point on all state changes.

Enforce the validity of transitions (or at least scream loudly when something transitions from STOPPED to PAUSED without having passed through PLAYING first).

If you have to check whether some resource is being used by some instance, don’t ask it which resource it is using, ask it whether it is using the one you want.

;;; Common: Reader is forced to know each player has one socket and;;; that sockets are comparable with #'=(loop :for player :in all-networked-players:until(= socket-with-something-happening(player-socket player)):finally(return player))

;;; Better: All I wanted to know is, "Is this yours?"(loop :for player :in all-networked-players:until(player-using-socket-p player
socket-with-something-happening):finally(return player))

Encapsulation is about protecting the person who has to read your code. It’s not about protecting your code.