Erlang TDD hands on project – WorkerNet part 3

The third part will unveil the job layer, the part of the Worker Net “stack” that handles the jobs of the network. Just as before, it is easy to express the functionality through stories.

“I want to be able to describe a job as

A set of Files [#wn_file{}]

The possible resource types needed (disjunction)

A list of commands for running the job

A timeout the job is given while running (in seconds)

A job id – primary key for the job“

“I want to be able to register a job in the job layer, through any node.”

“Several jobs should be queued on the same resource type(s), being processed one and one in order as they where queued”

“A job must be possible to cancel before it starts running, through any node.”

“The output generated by a job should be possible to see in realtime, through any node and stored to logfiles.”

“Once a job is done, the result should be stored in the file layer, together with the logs.”

“I want to be able to delete a job once it’s done, through any node.”

“I want to be able to list all jobs in the system, through any node.”

These tests require a lot more setup than the old ones, as the job layer must be able to transfer, add and delete files. The job layer must also be able to list the available resources. Also, what remains is to design the functionality of the layers involved, I have prepared such a design which can be seen in the picture below

The text which is in bold denote certain parts of interest that need fleshing out or be considered some extra times. The parts which require some more consideration are

The requirements for the resource process

How is the job process going to aquire the Pid of all the resource processes?

How is the negotiation going to be implemented? (timeouts etc)

How will the job be executed?

Once these parts are answered, wer’e almost done! And since we are doing TDD here, we are expecting answers in the form of tests. As first thing, add a new line to the Makefile for testing the job_layer

This first test is quite basic and shows how a job has files that may be located on any node, a job id (JID) and a list of resource types (disjunction). To make this test pass, we need to start implementing the record type. So the new entry in worker_net.hrl is

Next, the actual logic module needs to be implemented, it seems like gen_server fits the bill quite nicely here for the job_layer. For each part, it is good practice to make it as simple as possible, “you ain’t gonna need it” (YAGNI) is a good thing to remember. Without further ado, the implementation that passes the first test, of denoting and registering a job through any node. Take note that two new modules had to be introduced

wn_job_keeper.erl the job process started by the job_layer, sends the pid to the appropriate resource processes. Started after registration.

wn_resource_process.erl the resource process started whenever a new resource is registered

As the design requires a resource process to be started for each newly registered process, we need to test the new functionality. The side effect of registering a new resource must be checked. So first out, the modifications to the wn_resource record

The function resource_processes_are_alive/2 was added to each test in appropriate places where a resource is registered. Once this modification was made, the change was imposed on the wn_resource_layer.erl module. The changes are shown with the %% Change comment

we can try making a test design for the job execution, and adding the timeout field for the record, but that will be handled in the next post – otherwise this post will get gigantic and never be posted as I’ve had a lot of other stuff to do the previous weeks.