Building a Forum with Clojure, Datomic, Angular, and Ansible

27 July 2013

After many long months I’ve finished re-writing
Grateful Place. The site now uses Clojure
as an API server, with Datomic for the database, Angular for the front
end, and Vagrant and Ansible for provisioning and deployment. This
setup is awesome, the best of every world, and I love it.

Below we’ll dive into the code base, covering the most important parts
of each component and how everything works together. We’ll cover:

Clojure API Server

Liberator for Easy API'ing

Going on a Spirit Journey with Friend

Testing a Clojure Web App is More Fun than Testing Rails

Serving files generated by Grunt/Angular

The Uberjar

Datomic

Why Oh Why Did I Do This

The Poopy Code I Wrote to Make Basic Things “Easier”

The Good Code I Ripped Off to do Migrations

Mapification with Cartographer, My Very Own Clojure Library!!!

Angular

Peeking and Secondary Controllers

Directives to the Rescue

Infrastructure

Creating a Local Sandbox with Vagrant

Provisioning with Ansible

Building and Deploying with a Janky Bash Script and Ansible

Development Workflow

Emacs Bookmarks and Keybindings

tmuxinator Config

Actually doing development

All source is available
on github.
This article isn’t meant to be a tutorial. However, if you have any
questions about how things work or about how to work on the codebase,
please leave a comment and I’ll do my best to clarify.

About the Site

Grateful Place is my attempt at creating
the kind of online community that I’d like to belong to. It’s still in
its infancy, but I’d like for it to become a site where people
consciously help lift each other up. One way to do this is by
expressing gratitude on a daily basis, which science says increases
happiness.

Some of the features include watching forum threads, liking posts, and
creating a basic profile. I have a lot more planned, like tags and
“summary” views, and I think the combination of Clojure and Angular
will make it fun and easy for me to continue development :)

If you want to have a look without diving head-first into hippified
waters (what, do you have something against happiness?), you can
use the demo site with username/password
test101/test101. Be warned, though, that that server might have some
bugs.

Now, on to the code!

Clojure API Server

I’m very happy with using Clojure as an API server. The libraries
involved are lightweight and transparent, with no magic, and that
makes it so easy to fully understand what’s going on. That was my
experience with Liberator:

Liberator for Easy API'ing

Liberator provided me with a much better abstraction for handling
logic which I was repeating in all of my controllers. For example, my
create functions all basically looked like this before I moved to
Liberator:

First, the protect macro is used to ensure you’re authorized to
do whatever you’re trying to do. The first argument is a boolean
expression, in this case (:id auth), which just checks whether
you’re logged in. If the boolean expression is true, run everything
that follows. Otherwise return an error status and error messages
(see implementation).

Check whether params is valid using the specified validation, in
this case (:create validations/post). If it’s valid, run the let
statement, otherwise make the validation errors available in errors
and run (invalid errors).

There are a couple things that I didn’t like about this approach. First,
there was too much distance between the logical branches. For example,
protect is basically an if statement, but the else is hidden.
Also, the
actual code
I wrote in if-valid is a bit long, which makes it difficult to
visually understand how (invalid errors) relates.

Second, this approach required me to introduce more nesting in order
to add more steps or checks in the workflow. This would make it even
harder to understand as I’d mentally have to descend and ascend a
few conditionals in order to understand what’s going on. I’d end up
with something like:

-Decisionone-Decisiononefirstbranch:Decisiontwo-Decisiontwofirstbranch:Decisionthree-Decisionthreefirstbranch...Lotsofcodeherephysicallycreatingdistancebetweenbranches-Decisionthreesecondbranch...Morecodecausingmoredistance-Decisiontwosecondbranch-Decisiononesecondbranch...whatwasthedecisioneven?Ican't remember and now it's hard for me to visuallyassociatethisbranchwithitsparentdecision

So essentially, I’d have to keep an ever-growing decision tree in my
head. The physical representation of the tree, the code, would help to
obscure the logic flow as I added more code.

This allows me to concentrate on one node at a time, instead of having
to keep an increasingly complicated tree structure in my head. For
example, I can physically place the logic for malformed? next to the
code that I want to run if the request is malformed, specified by
handle-malformed.

Liberator has excellent documentation and using it is a big win. It
lets me just plug my own bits of logic into a carefully-coded, useful
HTTP framework. I definitely recommend it.

Going on a Spirit Journey with Friend

Friend still kinda makes my head
hurt. It’s a useful library that gets the job done, but I feel like
using it requires poring over the source code until you attain that
brief flash of enlightenment that allows you to pound out the code you
need for as long as some dark, atavistic, pre-conscious part of your
brain can hold everything together. After that you pray that you won’t
need to change anything because, Jesus, that trip can take a lot out
of you. I don’t know, maybe that’s why peyote was invented.

Anyway, that’s a testament to my own need to learn (and perhaps a need
for slightly clearer documentation) and not to the quality or value of
the library itself. Everything hangs together, working with Ring in a
stateless way, which I really appreciate.

Creating an authenticated session as soon as a user registers
instead of requiring login

It turned out that the first wasn’t all that difficult. I think.
Here’s the code
on github.
It’s also listed below, in the next code block.

The key bit is :login-failure-handler, which simple returns a map
for Ring. I also have :redirect-on-auth? listed twice. I’m not sure
if this is necessary but every once in awhile I like to do some shaman
programming, throwing chicken bones and listening to the wind in hopes
that everything turns out OK. Things are working and I’m not going to
mess with them.

Creating the authenticated session is a different story. There are a
lot of things going on. In order, they are:

User submits registration form

Request goes through a bunch of Ring middlewares that wrap the
request, adding keyword params and handling json and whatnot

Request hits the middleware created by Friend

The request “hits” the users/attempt-registration Friend workflow

If the registration is valid, return a friend authentication map.
Friend “knows” that this is not meant to be a response sent to the
browser, so the authentication map gets added to the Ring request map
and the resulting map gets sent to the next Ring middleware

The next ring middleware is routes

The users/registration-success-response route matches

users/registration-success-response returns a Ring map, providing
a body. The response is a map like {:id 1234 :username
"flyingmachine"}. This then gets used by Angular.

Here’s all the relevant code. Steps are indicated in brackets, like
[1] or [2] or [3]. Step 1 is omitted as that’s not code, you silly
goose.

;; The ring app, https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/server.clj#L29(defn wrap[to-wrap](-> to-wrap(wrap-session{:cookie-name"gratefulplace-session":store(db-session-store{})})(wrap-restful-format:formats[:json-kw])wrap-exceptionwrap-keyword-paramswrap-nested-paramswrap-params)); The ring app(def app(-> routes;; [6] after a successful registration the routes;; middleware is calledauth;; [3] after request is wrapped, send it to friendwrap;; [2]));; Friend middlware(defn auth[ring-app](friend/authenticatering-app{:credential-fn(partial creds/bcrypt-credential-fncredential-fn):workflows[(workflows/interactive-form:redirect-on-auth?false:login-failure-handler(fn [req]{:body{:errors{:username["invalid username or password"]}}:status401}))users/attempt-registration;; [4]session-store-authorize]:redirect-on-auth?false:login-uri"/login":unauthorized-redirect-uri"/login"}));; [4] Friend runs this workflow function. If the workflow function;; returns falsey, then friend tries the next workflow function. In;; this case, when a user submits a registration form then the `when`;; boolean expression is true and the function will not return falsey.;; If the registration is successful it will return an authentication;; map and continue to step 5. If the registration is unsuccessful it;; will return a Ring response map, which is basically a map that has;; the keys :body or :status.;; https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/controllers/users.clj#L20(defn attempt-registration[req](let [{:keys[urirequest-methodparamssession]}req](when (and (= uri"/users")(= request-method:post))(if-validparams(:createvalidations/user)errors;; [5] Here's where we return the authentication map, which;; Friend appends to the request map, sending the result to the;; next middleware(cemerick.friend.workflows/make-auth(mapify-tx-result(ts/create-userparams)record){:cemerick.friend/redirect-on-auth?false})(invaliderrors)))));; [7] The compojure route, https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/middleware/routes.clj#L67(authroutePOST"/users"users/registration-success-response);; [8] the final step in our journey(defn registration-success-response[paramsauth]"If the request gets this far, it means that user registration was successful."(if auth{:bodyauth}))

I’m both proud and appalled that I wrote all that code.

Testing a Clojure Web App is More Fun than Testing Rails

For testing I decided to try out
Midje. Midje is easy to get used
to, and @marick has articulated a clear and compelling philosophy for
it.

But before we get into some code let me explain the heading, “testing
a Clojure web app is more fun than testing Rails.” This has to do with
Clojure itself and not with any testing library.

There’s no real magic in any of the code I wrote. Everything is just a
function. You give it an input and it returns an output. You give your
application a Ring request and it goes through all the layers and
returns a Ring response. You don’t have to do any crazy setup hijinks
or create special environments like you do in Rails - especially like
you have to do when testing controllers. This makes testing so much
easier and more fun.

So, that said, I feel like there’s not much remarkable with my tests.
There’s a lot of room for improvement.

I ended up creating a lot of
helper functions
to DRY up my
controller tests,
and those might prove helpful to someone else. I also ended up
writing a crazy-ass macro for creating functions with default
positional arguments:

So I needed some way to get the Clojure app to actually serve up these
files. I also needed to be able to serve the files when they’re
packaged as resources in the final uberjar. This turned out to be
really easy:

We’re just iterating over each possible path for the front end files
and creating both a file route and a resource route for them. This is
a lazy way to do things, resulting in a few unnecessary routes. In the
future, it would be nice to make the app “know” whether to use the
single resource route, html-app, or whether it needs to use the file
routes, ../html-app/app and ../html-app/.tmp.

The Uberjar

As I started to deploy the forum I found that I needed and easy way to
run database-related tasks. Here’s what I came up with:

So you can run java -jar gp2.jar server and get a server running, or
reload the database or run migrations. I could also have used lein
on the server, and I’ll probably do that eventually. For now I’m just
creating uberjars and copying them over.

Holy cow, the Clojure section is over! Let’s talk about Datomic now!

Datomic

Why Oh Why Did I Do This

When I set about re-writing the site it felt risky to use Datomic
because a) I didn’t know how to use it and b) it didn’t seem like it
would add much value over postgres or mysql for my tiny side project.

But those were also compelling reasons to go with it: a) it’s exciting
to learn a completely new way of working with databases, designed by
some really freaking smart people who know which side of the bread is
buttered and b) it’s just a tiny side project and I can do whatever I
want.

Ultimately I’m happy with the decision. I’ve learned a lot by
researching Datomic (see
“Datomic for Five-Year-Olds”)
and using it has afforded the same simple, lightweight experience as
using Clojure.

You won’t find any mind-blowing code here – I’m still trying to
learn how to use Datomic well – but hopefully you’ll find it
useful or interesting.

The Poopy Code I Wrote to Make Things “Easier”

I wrote a number of wrapper functions in the misleadingly-name
gratefulplace.db.query namespace:

(ns gratefulplace.db.query(:require[datomic.api:asd])(:usegratefulplace.config));; This is dynamic so I can re-bind it for tests(def ^:dynamic*db-uri*(config:datomic:db-uri))(defn conn[](d/connect*db-uri*))(defn db[](d/db(conn)));; Don't make me pass in the value of the database that gets boring(def q#(d/q%(db)));; I'll give you an id, you give me a datomic entity or nil(defn ent[id](if-let [exists(ffirst (d/q'[:find?eid:in$?eid:where[?eid]](db)id))](d/entity(db)exists)nil));; Is this an entity?! Tell me!(defmulti ent?class)(defmethod ent?datomic.query.EntityMap[x]x)(defmethod ent?:default[x]false);; I'll give you some conditions, you'll give me an entity id(defn eid[&conditions](let [conditions(map #(concat ['?c]%)conditions)](-> {:find['?c]:whereconditions}qffirst)));; I want one thing please(defn one[&conditions](if-let [id(apply eidconditions)](entid)));; I want all the things please(defn all[common-attribute&conditions](let [conditions(concat [['?ccommon-attribute]](map #(concat ['?c]%)conditions))](map #(ent(first %))(q{:find['?c]:whereconditions}))));; Passing the connection all the time is boring(def t#(d/transact(conn)%))(defn resolve-tempid[tempidstempid](d/resolve-tempid(db)tempidstempid));; I make a lot of mistakes so please make it easy for me to retract them(defn retract-entity[eid](t[[:db.fn/retractEntityeid]]))

Some of these functions simply reduce the code I write by a tiny bit,
for example by allowing me to not pass a connection or database value
into every single database-related function, which would make no sense
for me as I only have one database.

Others, like one and all provide me with an “easier” way of
performing common queries but at the expense of sometimes writing
queries in roundabout ways or taking away some of my flexibility.

For example, in the all function I’m limited to only one data
source. The result is that I sometimes have to use the datomic.api
functions in places where I’d prefer not to, and the codebase doesn’t
quite feel cohesive. One example of this is the query function in
the watches controller:

I have to call datomic.api/q directly because I want to pass in
?userid.

I’m not sure whether I should drop these functions entirely and just
use the datomic api or whether I should continue tweaking them to meet
my needs.

The Good Code I Ripped Off to do Migrations

The
gratefulplace.db.manage namespace
has some code I stole and modified from Day of Datomic. It’s a really
cool, simple way of ensuring that migrations get run. The basic idea
is that you keep track of schema names which have been installed, then
install any schemas that haven’t been installed. It’s a simple,
logical approach and the code that implements it is pretty neat, as
you would expect from Stu Halloway.

Mapification with Cartographer, My Very Own Clojure Library!!!

Cartographer is the
result of my attempt to easily do some processing and pull in
relationships when converting a Datomic entity to a map. I think the
README explains it all so you can learn more about it there. Here are
some of the maprules used in GP2:

Angular

I’ve been learning Angular since last November and I love it. Using
it, I feel like I finally have the right tools for creating web apps.

Peeking and Secondary Controllers

I wanted to implement the idea of “peeking” at things on the forum.
For example, if you click on a user link you’ll just view a summary of
his info in the right column instead of completely leaving the page
you’re on.

The idea is that, while reading a thread, you might find a response
interesting. You want to know a little more about the author but don’t
want to lose your place. So you “peek” at him, which shows you some
info and preserves your place in the thread. It was just something fun
I wanted to try.

However, as far as I know Angular doesn’t make this very easy for you.
The approach I took was to have a
Foundation controller
which places the
Support
service on the scope. Since all other controllers are nested under
Foundation, they’ll have access to $scope.support.

The purpose of Support is define a way to show views in the right
column and make data accessible to the view. For example, the author
directive has the following:

So, ultimately, what’s happenins is that when you call
Support.peek.show("user", data), it sets some variables so that the
view associated with the “user” peek is shown. That view then accesses
the data you passed to Support.peek.show with, e.g.,
support.peek.data.username.

I know this isn’t a super-detailed explanation of what’s going on, but
I hope some investigation of the code will answer any questions you
might have.

Directives to the Rescue

Angular directives are as powerful as everyone says they are, and I
think I’m finally utilizing them well. You can see all my
directives on github.

Infrastructure

Because GP2 uses Datomic Free, I couldn’t deploy to Heroku. This meant
having to actually handle provisioning a server myself and deploying
without following a tutorial. In the end things are working well. The
site’s residing on a [Digital Ocean][https://www.digitalocean.com/]
server, which has been very easy to work with.

Creating a Local Sandbox with Vagrant

Creating a local sandbox lets you make all your provisioning mistakes
more quickly. If you’re creating a new provisioning script of tweaking
your existing one, you should do it in a virtual machine.

Vagrant makes this process as easy as an
old shoe. Once you’ve installed virtualbox and vagrant all you have to
do is run vagrant up from the infrastructure directory and you’ll
have a virtual machine ready to go. The Vagrant web site has excellent
tutorials so check it out if you want to learn more.

Provisioning with Ansible

Ansible’s
supposed to be super simple compared to Puppet and Chef. I found it
easy to learn. It’s also simple enough to easily modify scripts and
powerful enough to do exactly what I want it to, which is provision a
server with Java and Datomic and deploy my app to it. You can check
out my setup in infrastructure/ansible. If you’re using Datomic free
please do use it as a starting point.

provision.yml has just about everything you need to get a server up
and running, with the exception of uploading SSH keys. deploy.yml is
used by the janky bash script below to upload an uberjar, run
migrations, and restart the server.

Building and Deploying with a Janky Bash Script and Ansible

Here’s my janky Bash scripts which first build the app and then
deploys it with Ansible:

Workflow

OMG this article is almost over! Listen, I know you don’t need to know
this and it makes no difference to you but I am out here in the North
Carolina heat sweating my ass off trying to finish this article so I
can get on with my day. So it’s pretty exciting that we’re almost
done.

Anyway - here are workflow improvements I developed over the course of
this project. You might also want to check out this
My Clojure Workflow, Reloaded.

Emacs Bookmarks, Snippets, and Keybindings

I created a
bookmark
to open my server/src/gratefulplace/server.clj file with just a few
keystrokes instead of having to navigate to it. I recommend doing this
for any project which you’ll be toiling over for months on end!

So, once you have server.clj open and you’ve run nrepl-jack-in you
can hit C-c C-v to start the server. Also check out the
nrepl keybindings
for some great workflow helpers.

tmuxinator config

In order to do development you need to have Datomic and Grunt running.
Instead of having to open up a bunch of terminal tabs and handle all
that manually every time I want to start working, I use tmuxinator so
that I can get my environment set up in one comand. Here’s my config:

# ~/.tmuxinator/nicu.yml# you can make as many tabs as you wish...project_name:gp2project_root:~/projects/web_sites/gp2rvm:1.9.3tabs:-angular_server:git pull && cd html-app && grunt server-datomic:datomic-shell:

Actually Doing Development

So, in order to get to the point where you can actually start writing
code and seeing the results, do the following:

Install datomic and set up your own datomic alias

Run mux gp2 to start tmux with your tmuxinator conf

Open emacs

Hit C-x r l to open your list of bookmarks and choose the
bookmark for server.clj

Run M-x nrepl-jack-in in emacs

Hit C-c C-v to start the jetty server

The End

That’s it! I hope you’ve found this article useful. I’m going to go
have a life for a little while now. Haha, just kidding! I’m going to
spend the next two hours hitting refresh on my reddit submission!