Cleaning Up store

The kvs:store/1 function has gotten embarassingly long.
Initially this may have been justifiable to keep explanation
simple, but that time has passed.

The function contains both the pattern matching to
disambiguate message types, and also the implementation of how to
respond to each type of message. A quick simplification is to move each
response into its own function.

The pattern-matching portion is simplified to:

store(Store=#kvs_store{data=Data,pending_reads=Reads,pending_writes=Writes})->receive{Sender,retrieve,Client,Key}->message_retrieve(Store,Sender,Client,Key);%% and so on..

Adding Tests

Refactoring without tests is driving without road signs.
After changing a substantial piece of functionality it
is time-intensive to check that existing functionality
remains intact. Where time intensive is a euphemism
for humans do it badly and inevitably screw up.

Erlang provides a number of testing facilities2, the simplest
of which is eunit, which we'll be using to test our project.
EUnit is comparable to [phpunit] or [junit], except (perhaps) requiring a bit less boilerplate code.

Our initial eunit tests will be rather simple,
but we can enhance them as we add further functionality
down the line.

5>c(kvs).{ok,kvs}6>c(kvs_server).{ok,kvs_server}8>c(kvs_tests).{ok,kvs_tests}9>kvs_tests:test().kvs_tests: not_running_test...*failed*::error:function_clause in function lists:'-filter/2-lc$^0/1-0-'/2 called as '-filter/2-lc$^0/1-0-'({error,{no_such_group,kvs}}, #Fun<kvs.1.52371442>) in call from kvs:stop/0 in call from kvs_tests:not_running_test/0======================================================= Failed: 1. Skipped: 0. Passed: 3.error10>

Ahh, the kvs_tests:not_running_test is still having an issue, but everything else
is passing. We've preserved the quirks and the functionality from before the refactor:
a hallmark of a successful refactor.

From Message Passing to gen_server

The final stage of this refactor is to transition from our message-passing based implementation
to one using an OTP gen_server. gen_server stands for generic server and is
exactly what it sounds like. gen_servers are functionally equivalent to the message-passing
based approach we've used thus far, but they have a few important advantages:

they eliminate boiler-plate code,

they play well with the rest of the Erlang OTP utilities like supervisors, hierarchy trees, etc,

they are familiar to other Erlang programmers so it is faster to understand a program's overall structure.

Most substantial Erlang projects either start out or end with at least one gen_server.

As we go through this rewrite, note that we'll keep the client interface constant, so that the
hypothetical user is sheltered from any changes, and so that we can continue running our
existing tests. As a bit of a spoiler, after a great deal of turmoil the combined size of kvs.erl and kvs_server.erl
will decrease from 200 to 189 lines of code.

Now let's look at our first change: updating the implementation of the client interface
for starting kvs servers in kvs.erl.

Next we need to update the implementation of the client
interface for updating and retrieving values. Two things
worth noticing are:

we get the send & block on receive construct for free from gen_server/call,

we no longer need to explicitly pass along the sender's process id, as we'll get that for free as well.

%% old get implementationget(Key,Timeout)->Pid=pg2:get_closest_pid(kvs),Pid!{self(),get,Key},receive{Pid,got,Value}->{ok,Value};{error,Error}->{error,Error}afterTimeout->{error,timeout}end.%% new get implementationget(Key,Timeout)->Pid=pg2:get_closest_pid(kvs),gen_server:call(Pid,{get,Key},Timeout).

And next the same changes for updating values.

%% old set implementationset(Key,Val,Timeout)->Pid=pg2:get_closest_pid(kvs),Pid!{self(),set,Key,Val},receive{Pid,received,{set,Key,Val}}->{ok,updated};{error,Error}->{error,Error}afterTimeout->{error,timeout}end.%% new set implementationset(Key,Val,Timeout)->Pid=pg2:get_closest_pid(kvs),gen_server:call(Pid,{set,Key,Val},Timeout).

Now that we've completed the changes to kvs.erl
it's time to move onto kvs_server.erl. It would be rather
dull to look at all of the changes required to convert into a gen_server
(there is a changeset of everything which may be
instructive). Rather, we'll take a high-level approach and note some of the interesting details.

In the old implementation we relied upon the store/1 function to dispatch incoming
messages to the correct function, but the gen_server performs the dispatch for us,
instead asking us to implement the handle_call/3 function.

%% excerpt from old implementationstore(Store)->receive{Sender,get,Key}->message_get(Store,Sender,Key);%% some other pattern matchesstop->okend.%% excerpt from new implementation%% @doc handle explicit stop requestshandle_call(stop,Sender,State)->{stop,stopped,stopped,State};handle_call({get,Key},Sndr,State=#kvs_store{pending_reads=Read})->%% implementation...

Above we are using handle_call, but there is a similar function named
handle_cast which is also used in the new kvs_server.erl implementation.
The difference between the two is that handle_call is a synchronous operation from the client's perspective,
and handle_cast is an asynchronous operation from the client's perspective.

Note that handle_call needs only be a synchronous operation from the client's perspective,
and that it is possible to have the server store the client request in its state, receive other incoming requests, and then return
a response to the first request (this implementation makes extensive use of this pattern).

There are many other aspects of the gen_server implementation we could dive into, but
we could get stuck in this investigation for a very long time. Rather, from our perspective the
goal was to restructure the code so that we can make surgical algorithmic changes as the
series progresses.

Phew. Now that we've shuffled all of that code we've gotten out of the business of working
against the OTP, and are now finally starting to work with it. Several entries from now after we've worked
further on the core implementation, we'll return to this line of work and take advantage
of some other Erlang goodness (in particular supervisors),
but for the time being it is good to be ready to get on with some algorithms.

Note that the naming of the functions as message_??? is entirely arbitrary.↩

The Erlang test_server is perhaps the most profoundly
flexible and most profoundly confusing piece of testing software ever
written. I have tried to grok its glory at several times, but still
find it challenging to use at best.↩

Hi folks. I'm Will, known as @lethain on Twitter.
I write about software and management topics,
and love email at lethain[at]gmail.
Get email from me by subscribing to
my weekly newsletter.